The solution is the last thing you should pick

The largest single economic benefit forecast for a programme now costed at up to £9.3 billion was five minutes.

Five minutes saved per police officer per shift, compared with the arrangements the Emergency Services Network is meant to replace. That is the largest forecast economic benefit in the case, and the National Audit Office recorded three things about it in May 2019. The Home Office had not revised the police efficiency assumptions it made in its 2015 business case. Those assumptions were yet to be accepted by the police. And greater adoption of mobile technology within policing since 2015 meant the impact on productivity might by then be less than predicted.

The NAO went further about the wider costing, noting that the Home Office’s figures used assumptions it had not had time to test with users.

This is not a story about a department that failed to talk to police officers. The Home Office talks to police officers constantly. It is a story about a sentence describing what a constable’s shift would be like, written in 2015, used to justify billions of pounds of commitment, and never checked with a constable while the solution around it hardened into three main contracts.

The scale of what rests on that assumption is worth stating, and so is the limit of it. Benefits from the network are forecast at £1.5 billion in the period to 2037, made up of £756 million of financial savings and £718 million of economic benefits. The five minute saving is the largest single item on the economic side, which is the smaller of those two halves. So this is not the biggest number in the whole case. It is the biggest number in the half of the case built on claims about how people work, and it has been held constant across a decade in which the devices in that constable’s pocket changed completely.

Talking to users is not the discipline. Sequencing is

Every organisation running a transformation does user research of some kind. Most do a great deal of it. The reason the research so often changes nothing is structural rather than attitudinal, and it has a date attached.

A procurement timetable requires a specification. A specification requires knowing what is being bought. So the product, or at minimum the shape of the product, gets named at the point when the least is known, because that is when the commercial process demands a document. Everything discovered afterwards arrives at a system whose boundaries are already contractual.

After that point, research does not produce decisions. It produces a backlog of grievances, a change request, or a training plan.

Which gives a test more useful than any maturity model. For each piece of user research on the plan, name the decision it is capable of changing, and name the date after which it cannot change that decision. If the research lands after the date, it is documentation. That is not worthless, but it should be budgeted as documentation and nobody should expect it to alter the design.

A pilot can be designed so that it cannot tell you anything

The other place this goes wrong is subtler, because it happens inside an activity that looks exactly like the right thing to do.

HMRC launched a pilot for Making Tax Digital for Self Assessment in April 2017. The NAO reported what happened to it. Despite the range of potential scenarios available to test, the scale of testing and piloting was limited, in part because HMRC set restrictive requirements, including that participants must have an accounting year aligned to the tax year. More than a thousand taxpayers signed up. HMRC needed to remove most of them because they were ineligible. When the pilot closed to new entrants in 2022, there were 15 participants.

Fifteen people, for a change that will eventually reach several million self-employed people and landlords. And the eligibility rule that produced that number was not arbitrary: aligning the accounting year to the tax year makes the software simpler. It selected for the taxpayers whose circumstances the design already handled.

That is the general failure. Eligibility criteria for a pilot are usually written by the people who built the thing, and they exclude precisely the cases the design has not solved. The pilot then reports success, accurately, about a population chosen for being easy. The NAO also recorded that HMRC had still not resolved design issues including how to allow multiple agents to represent taxpayers, how to handle jointly owned property, and how to deal with changes in taxpayers’ circumstances, which is a fair description of the population the pilot had ruled out.

Three questions worth more than a research plan

Who is excluded from the sample, and by which rule? Not who is in it. Every research population has an exclusion criterion, it is nearly always written for operational convenience, and it nearly always removes the hard cases. The excluded set is the honest description of what the pilot cannot tell you.

Which number in the business case describes somebody’s day, and has anyone shown that number to somebody whose day it is? In the emergency services programme that number was five minutes. In most programmes it is a percentage reduction in handling time, and it usually originated in a workshop rather than an observation. The test is not whether the number is plausible. It is whether the people it describes have agreed to it, because they are the ones who have to produce it.

And what would have to be true for the chosen solution to be the wrong one, and who is watching for that? A programme that cannot answer this has not chosen a solution. It has adopted one.

The sequencing rule is not a design fashion

The argument for understanding the problem before naming the product is usually made on quality grounds, which is why it loses to procurement timetables. It is better made on cost grounds.

The Emergency Services Network is now forecast to cost £3.1 billion more than the 2015 business case, an increase of 49%, and its break-even point against simply continuing with the old system has moved to July 2029, seven years later than forecast. Some of that is technical. A material part of it is that the definition of what was being bought kept moving after the contracts were signed, which is what happens when the specification precedes the understanding.

Picking the solution last is not slower. It moves the discovery of what the thing has to do from after the contract to before it, where changing your mind is free. The related question of what a partner choice does to the price of changing your mind is taken up in speedboat or oil tanker, and the rest of this desk’s reporting on large programmes is filed under digital transformation.