The birth of Big Data — and how Hadoop happened

By the early 2000s, the web had grown faster than any single machine could keep up with. Indexing it meant storing and processing volumes of data that broke the usual playbook — too big for one server, too messy for a traditional database. "Big Data" became the name for that problem: data whose volume, velocity and variety outgrow conventional tools.

Two papers that changed everything

In 2003 Google published a paper on the Google File System (GFS) — a way to store enormous files reliably across thousands of cheap, failure-prone machines. A year later, in 2004, came MapReduce: a programming model that split a huge computation into small pieces, ran them in parallel across the cluster, and combined the results. Together they described how to turn a warehouse of commodity computers into one giant, fault-tolerant data engine.

From a web crawler to a toy elephant

Doug Cutting and Mike Cafarella were building Nutch, an open-source web search engine, and hit exactly the wall Google had described. They re-implemented GFS and MapReduce as open source so Nutch could scale. In 2006 that storage-and-compute layer was spun out into its own project — and Cutting named it after his young son's yellow stuffed elephant: Hadoop.

The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere. — Doug Cutting, on naming Hadoop

Why it mattered

Yahoo! adopted Hadoop early and ran it on thousands of nodes, proving it at scale. Two ideas were the breakthrough: move the computation to where the data already lives (instead of shipping terabytes across the network), and assume hardware will fail — replicate data and re-run failed tasks automatically. That made Big Data affordable on ordinary machines, and an entire ecosystem (Hive, HBase, Pig, later Spark) grew on top.