The Data Team Test, 13 years on

In 2000 Joel Spolsky published The Joel Test: twelve yes/no questions that told you, in about three minutes, whether a software team was any good. Do you use source control? Can you make a build in one step? Do you fix bugs before writing new code? No maturity model, no consultants. Just twelve questions you could ask in an interview.

In February 2013 I was leading data analytics at NaturalMotion, and I wanted the same thing for data teams. So I wrote one. It sat in my blog repo, marked unpublished, for thirteen years.

The 2013 list

Do you use source control?
Do you have a separate development and production environment?
Do you have access to raw data in a warehouse or other offline store that doesn't impact the live system?
Do you have a way of recording and marking dirty data?
Do you record bugs and fix them before writing new code?
Do you have a process to schedule and prioritise analysis?
Do you spend time with product owners and domain experts?
Do programmers and analysts have quiet working conditions?
Do you use the best tools money can buy?
Do you have testers?
Do you have a mechanism to communicate findings and make recommendations?
Do you and other teams use data-driven development techniques?

What held up, and what didn't

The questions about people held up best. Spending time with product owners (7), having a way to turn findings into recommendations (11) and getting other teams to work from data (12) are still where most data teams win or lose.

The questions about infrastructure aged fastest. In 2013, having a warehouse that didn't take down the live game (3) was a real achievement. Today a cloud warehouse is a credit card away, and "the best tools money can buy" (9) mostly means picking fewer of them. Separate dev and prod (2) became table stakes.

Two questions were quietly about something bigger. "Recording and marking dirty data" (4) was really asking whether anyone owns data quality. "Do you have testers?" (10) was asking whether anyone checks the numbers before the CEO sees them. Both matter more now, not less, because the thing reading your data is increasingly a model that won't notice when it's wrong.

The Data Team Test, 2026

Same rules as Joel's. Yes or no, no partial credit.

Is every metric that matters defined once, in code, with an owner? If "active user" means three different things in three dashboards, nothing else on this list matters.
Can someone outside the data team answer a basic question without filing a ticket? And if you've put an AI assistant in front of your data, does it give the same answer an analyst would?
Do you test your data like code? Tests on pipelines, freshness checks and agreed contracts with the teams that produce the data.
Do you find out data is broken before your stakeholders do?
Does every experiment have its success metric and decision rule written down before it launches?
Do you measure long-term holdouts, not just A/B test wins? A string of winning tests can add up to a product that's no better.
Do data people sit in the product squads, not a service desk?
Do you have evals for every model and prompt that ships? If you can't say how good it is today, you can't say whether the next change made it worse.
Can you trace an AI answer back to the data it used?
Do agents get the same access controls and audit trail as people?
Can you list the decisions your data changed last quarter? Not dashboards shipped. Decisions changed.
Can a new hire ship a useful analysis in their first week?

Scoring is the same as Joel's: twelve is great, eleven is fine, and ten or below means something structural is wrong, usually in questions 1, 4 or 11.

Part two: the Product-Engineering Team Test

The data test asks whether a team can trust its numbers. This one asks whether a product team can turn what it learns into something shipped. Same rules: yes or no, twelve questions.

Shipping

Can any engineer get a change to production the same day, safely?
Can you turn a feature off without a release? Flags and kill switches, not hotfixes.
Does every launch have a success measure agreed before it ships?

Knowing what's happening

Would you find out about a broken flow from your monitoring before your customers tell you?
Does the team watch real people use the product at least once a month? Sessions, interviews, support tickets read by engineers.
Can you say what changed in your key metric last week and why?

Working as one team

Do engineers, design and product share one goal, not three backlogs?
Are the people closest to the problem trusted to decide how to solve it?
Do you clean up before it hurts, rather than after? Tech debt has a budget, not a crisis.

Building with AI

Does every AI feature have evals that run before each change ships?
Do you know what your AI feature costs per user, and is it worth it?
Does the team use AI in its own work by default, with a person reviewing what ships?

Twelve is excellent, eleven is fine, ten or below and the gaps tell you where to start.

What changed in thirteen years

In 2013 the hard part was getting the data. Now the hard part is trusting it, and making sure the people and the agents using it get the same answer. The 2013 list was about building a data team. The 2026 lists are about building a company where data is something everyone, and everything, can use without a data person in the room. More on how that happened in Hello Internet, revisited.

FAQ

What is the Joel Test? A twelve-question yes/no checklist for software teams, published by Joel Spolsky in 2000. It's still used in interviews to judge an engineering team quickly.

What is a Joel Test for data teams? The same idea applied to data and analytics teams: a short yes/no checklist covering data quality, experimentation, ownership and impact. The 2026 version above adds AI-era questions about evals, traceability and agent access.

How do I use the Data Team Test? Answer each question yes or no as a team, honestly. Fix the lowest-numbered "no" first: metric definitions and data quality come before experimentation and AI.