Hello Internet, revisited
Rereading my first blog post from 2012, I don't think I got anything wrong. What gets me is how much the world around it has changed.
In December 2012 I wrote my first blog post, Hello Internet. I was in Oxford, a few months into running a small analytics team at NaturalMotion, and about ten years into working with data. Rereading it now, I don't think I got anything wrong. What gets me is how much the world around it has changed.
Where the world was
The tools that define data work today were brand new or didn't exist.
- Hadoop had only reached version 1.0 a year earlier, in December 2011.
- The IPython notebook, the ancestor of Jupyter, shipped the same month.
- In October 2012, Harvard Business Review called data scientist "the sexiest job of the 21st century".
- A week before my post, Amazon announced Redshift, a data warehouse you could rent by the hour. It wasn't generally available until February 2013.
Most companies didn't have a data team. Most that did were still working out what one was for.
What I was actually doing
My hardest problem that year was app interaction data: a lot of it, sitting in files rather than a relational database. Every tap, session and purchase from NaturalMotion's games, and the questions everyone wanted answered were somewhere inside it.
The kit was Python, MapReduce and Hadoop, running on a single Linux machine under my desk. I built it over a weekend from parts I bought at the local computer fair, paid for out of my own salary. If you wanted to work with data at that scale in 2012, that was often how you did it: you built the thing yourself, and then you wrote the jobs to run on it.
The waves since
What strikes me most is that each wave of change didn't make data work easier so much as move the hard part somewhere else.
| Era | How you analysed behaviour at scale | Where the hard part was |
|---|---|---|
| 2012 | Files, MapReduce jobs and whatever hardware you could get | Storing and processing the data at all |
| Cloud warehouses | SQL came back, on rented machines that scaled | Getting the data in, and paying for it |
| The modern data stack | Pipelines, modelled tables and dashboards for everyone | Agreeing what a metric means |
| Product analytics and experimentation | Self-serve funnels, cohorts and A/B tests | Running experiments that tell you the truth |
| LLM analysts | Ask a question in plain English and get an answer | Knowing whether the answer is right |
The team changed with each wave too. In 2012 the data person was a developer, statistician and sysadmin at once. Then came analysts, data engineers, analytics engineers and data scientists as separate jobs. Now some of those jobs are starting to be done by agents.
What didn't change
The parts that mattered in 2012 still matter. The scientific method. Being able to analyse data properly. Thinking hard about how real people behave and why.
What has changed is speed. In 2012 a good question took me days to answer: write the job, run it overnight, check the output, try again. Now the speed of getting an answer is the speed of typing the question. That makes the question the bottleneck, and the people who'll do best are the ones who still know how to ask a good one and how to tell when the answer is wrong.
If I could tell one thing to a new data scientist starting today, it's that. The tools will keep changing. The thinking won't.
Related: The Data Team Test, 13 years on.