Most AI projects fail one layer below the model

The AI Playbook for the UK Government, published by the Government Digital Service on 10 February 2025, sets out ten principles for civil servants deploying artificial intelligence. Know what AI is and what its limitations are. Use it lawfully, ethically and responsibly. Know how to use it securely. Have meaningful human control at the right stages. Understand how to manage the full life cycle. Use the right tool for the job. Be open and collaborative. Work with commercial colleagues from the start. Have the skills and expertise needed. Use these alongside your organisation’s own policies and assurance.

Read them again looking for the one about data. There isn’t one.

This is not a criticism of the playbook, which is a careful document and does address data in its procurement and governance sections. It is an observation about where the profession’s attention sits. Ten principles were chosen as the things a public servant must hold in their head, and the state of the records was not among them, in guidance written for organisations whose records are the entire reason the technology might be useful to them.

Somebody already named the missing layer

The most precise British description of this problem was written three years before the current wave and was about health research rather than artificial intelligence.

Professor Ben Goldacre’s review Better, broader, safer, published on 7 April 2022, made an argument that reads now as though it were written about model deployment. Data curation, it insisted, is a complex, standalone, high status technical challenge in its own right. Not a preliminary to the interesting work. The work.

The review’s finding about how that work is currently done is the expensive part. Curation happens differently, to variable and unseen standards, duplicatively in every team, data centre and project. Analysts told the review of deep frustration at the time it consumes and at the fact that none of it transfers between settings.

Every organisation that has run more than one AI project recognises this description whether or not it recognises the words. The second project does not start where the first finished. It starts where the first started, because the first project’s understanding of which patient identifier is authoritative, or which of four customer tables is maintained, lives in a notebook belonging to someone who has moved on.

That is what an infrastructure problem looks like. It is not that the work is hard. It is that the work is repeated.

The pipeline has to be code, and the reason has changed

Goldacre’s proposed minimum standard was Reproducible Analytical Pathways: shared, reviewable, reusable, well documented code for curation and analysis, explicitly to avoid duplication and to avoid unverifiable analyses that nobody can inspect.

That was a good idea for statistics. For AI systems it is closer to a precondition, because of a property of these systems that the analytics era did not have to deal with.

A statistical output is produced once and can be checked once. A model in production produces outputs continuously, against data that is arriving continuously, and the question you will eventually be asked is not whether the answer is right but why the answer changed. Answering that requires knowing exactly which version of which transformation produced the records the model saw in March, and rerunning it. If the curation was done in a spreadsheet, or by a data scientist filtering rows in a session that no longer exists, the question is unanswerable. Not difficult. Unanswerable.

Organisations discover this when a regulator, an auditor or a customer asks. By then the pipeline is in production and rebuilding it as code is a project nobody has budget for.

Access is infrastructure, and Britain has a working model of it

The part that most enterprise programmes leave entirely undesigned is the route by which somebody is allowed to use the data.

Public research has a defined answer. The Office for National Statistics’ Secure Research Service is a trusted research environment holding accreditation as a processor under the Digital Economy Act 2017, granted by the UK Statistics Authority. Researchers become accredited, work only from approved safe settings, find data through a metadata catalogue that states restrictions, and have their outputs checked before release.

The organising idea is the Five Safes framework, published in its current government form on 30 January 2025: safe people, safe projects, safe settings, safe data and safe outputs. It is worth understanding why that decomposition is powerful rather than merely tidy. It separates five questions that organisations usually collapse into one. Is this person trustworthy, is this purpose legitimate, is the environment controlled, has the data been reduced to what the purpose needs, and does the result leak anything. Collapsing them produces the two failure modes every large company has: a blanket refusal because one of the five is uncertain, or a blanket approval because four of the five are fine.

Almost no private organisation has an equivalent. Internal data access is negotiated case by case, by email, between a requesting team and whoever currently feels responsible, with no standing criteria and no clock. The cost of that shows up as delay, and delay is why AI programmes miss their dates far more often than model performance is.

The specific thing worth stealing is not the government’s process. It is the existence of a published route with named steps, so that a team can find out in an afternoon what they would have to satisfy, rather than discovering it over eleven weeks.

The two things organisations buy instead

Faced with this, procurement reliably produces one of two purchases, and neither is the missing layer.

The first is a vector store, bought because retrieval augmented generation was the pattern in the diagram. A vector index is a retrieval mechanism over documents you have already decided to trust. It has no opinion about whether the policy document it just returned was superseded in 2023, because supersession is a fact about your document management, not about embeddings. Organisations that put an unmaintained document library behind a chat interface have built a very fast way to distribute stale guidance.

The second is a governance platform, bought because an audit asked who owns what. These tools are genuinely useful once an organisation has decided who owns what, and completely inert before. A catalogue populated by an automated crawler tells you which tables exist and what their columns are called. It cannot tell you which of the three tables called customer is the one finance reconciles to, because that is institutional knowledge and nobody has been asked to write it down.

Goldacre’s review reached the same conclusion from the health research direction and drew the sharper structural inference: build a small number of shared secure analytics platforms rather than letting every project stand up its own. The enterprise translation is unpopular and correct. Most large British organisations have more data platforms than they have people who understand any one of them, because each business unit funded its own, and consolidation is the least glamorous programme a chief data officer can propose.

What to build, in order

The sequence matters more than the components, because organisations reliably build them backwards.

First, an inventory of what exists. Not a governance catalogue full of aspirational metadata, an honest list of the systems that hold records, who runs each one, and what state it is in. Most organisations do not have this and every subsequent step is guesswork without it.

Second, ownership of definitions. Somebody has to be accountable for what each significant field means, with the authority to change the system that produces it rather than only the reports that consume it. Analytics teams are usually handed the responsibility and denied the authority, which produces documentation of the problem rather than a fix.

Third, an access route with a stated elapsed time. Not a faster approval. A published one, so a programme can plan around it.

Fourth, curation as code, versioned, reviewed and rerunnable. This is where the compounding starts, because it is the first step whose output the next project inherits.

Only then models. A model is the cheapest and most replaceable component in this list, and it is the only one most boards have discussed.

The procurement failure this produces

The playbook’s own advice on buying AI is the part of it most worth reading, and it points at the same layer. It tells teams drafting requirements to start with the problem statement, to set out their data strategy and requirements, to focus on data quality, bias and limitations, to consider data format and provide suppliers with dummy data where possible, and to consider strategies to avoid vendor lock-in.

Every item on that list is a statement about your own estate rather than about the supplier’s product. An organisation that cannot describe its data cannot write those requirements, so it writes requirements about the model instead, and then buys an outcome it has no way of specifying. The supplier, reasonably, prices the integration risk into the contract or discovers it during delivery, at which point the programme is a change control conversation.

This is also where the cost that nobody forecast arrives. Curation, access engineering and pipeline maintenance are permanent operating costs, not project costs, and they do not appear in a business case built around a licence. The running arithmetic of that is worked through in intelligent automation.

The uncomfortable implication

If the argument above is right, most organisations’ AI roadmaps are sequenced wrongly, and the correction is unattractive because it looks like a delay.

There is a version of this that is not a delay, and it is the only recommendation here worth acting on immediately. Pick the single dataset that the next three intended use cases all depend on, and do the whole stack properly for that one: inventory it, name an owner for its definitions, publish an access route, and rebuild its preparation as reviewable code. One dataset, done to a standard, is worth more than a governance programme covering everything, because it produces an artefact the next team inherits rather than a policy the next team ignores.

The reason to start there rather than with the model is that the failure mode below the model is silent. A model that is wrong about the world usually looks confident, and confident wrongness reaching a customer at scale is the specific risk this layer exists to contain. The records problem underneath it is examined at data quality is the bottleneck, not the model. Model coverage sits in AI and automation, and everything this desk publishes on collection and lineage is filed under data and analytics.