Counting the open data portals still being updated

On 13 August 2026 the national open data catalogue for the United Kingdom held 59,096 datasets and 1,480 registered publishing organisations. Of those organisations, 1,175 had ever published anything. Three hundred and five had registered and published nothing at all.

Those are the easy numbers. The hard question is the one in the title, and the catalogue is remarkably badly equipped to answer it.

The freshness field does not measure freshness

Every dataset on data.gov.uk carries a metadata_modified timestamp, and it is the obvious thing to count. Do that and 28,867 datasets were modified in the past year, with 24,276 of those in the past ninety days. Which would mean that 84% of a year’s maintenance happened in one quarter.

It did not. Querying the CKAN API in narrow date windows shows what actually happened. In a single two-day window last week, 6,305 datasets were marked as modified, of which 4,008 belonged to the British Geological Survey and 1,196 to the Greater London Authority. In a fifteen-day window six weeks earlier, 11,981 were marked, including 2,166 from Natural England and 1,632 from the Office for National Statistics.

Nobody updated four thousand geological datasets in forty-eight hours. A harvester ran. data.gov.uk is largely a federated catalogue, ingesting records from departmental and local portals, and its modification timestamp records when the ingestion machinery last touched a record rather than when a human published new figures. The resource-level dates carry the same contamination: 1,507 individual file records in our sample were created on 6 August 2026 alone.

So the single field that a user would reasonably rely on to judge whether a dataset is maintained tells them, instead, when a script last ran. That is the most consequential finding here and everything below is an attempt to work around it.

What can be counted honestly

One field does not get rewritten. metadata_created records when a dataset record first appeared and stays put.

New records added per year run: 2,347 in 2010, 2,810 in 2011, 503 in 2012, 4,279 in 2013, 1,928 in 2014, 4,824 in 2015, then a peak of 9,717 in 2016. After that, 2,751, 2,091, 4,487, 2,481, 1,555 and 1,593 through to 2022, before 5,687 in 2023, 5,400 in 2024, 3,456 in 2025 and 3,187 in the first seven and a half months of 2026.

That is not a story of collapse, which is what the popular account of British open data would predict. It is a story of lumpiness. The spikes correspond to bulk onboarding of spatial and environmental catalogues rather than to policy attention, and the recent years are respectable. Records are still being created at roughly 4,000 a year.

Creation is not maintenance, though. To get at whether the things in the catalogue still work, we sampled it.

The sample

We took 6,000 datasets as sixty evenly spaced blocks of a hundred, drawn from the catalogue ordered by creation date, which spreads the sample across the full history rather than favouring the recent end. Those 6,000 datasets contained 20,086 resource URLs across 478 distinct hosts.

The first finding needed no network access. Eight hundred and forty-four of the 6,000, or 14.1%, had no resource attached whatsoever. Not a broken link. No link. A title, a description, a publisher and nothing to download.

The second finding also needed no network access, and it is worse. One thousand four hundred and forty-four of the resource URLs, 7.2% of the total, point at the UK Government Web Archive rather than at a live system. In 243 datasets, 4.1% of the sample, every single link points there. Those entries are not published data. They are citations of a snapshot of a page that used to hold data, presented in a catalogue whose purpose is to help you obtain it.

Testing the links

For the third finding we made requests. We drew 400 resource URLs at random from the sample, excluding the archive links, and issued one request each, following redirects, on 13 August 2026.

Three hundred and three returned 200, which is 75.8%. Fifty did not establish a connection at all: the host name did not resolve, or the connection timed out. A further thirty returned a server-side or malformed-request error, split thirteen 400s, nine 404s and eight 500s. Fifteen returned 403 or 401, and we are not counting those as dead, because an automated request is exactly what a bot filter is designed to refuse.

So at least 80 of 400, 20%, are unusable. Up to 97, 24.3%, are unusable by an automated client. A single test from one machine on one day is not a definitive audit and it is trivially repeatable by anyone who wants to check it, which is the point of describing the method rather than only the result.

The dead links are a map of machinery of government changes

The interesting part is not the failure rate. It is which hosts failed.

Every one of the eighteen sampled URLs on a ccg.nhs.uk domain failed to connect. Clinical commissioning groups were abolished and replaced by integrated care boards under the Health and Care Act 2022, and when a body ceases to exist its web estate goes with it. Across the full sample there are 620 such URLs still listed in the national catalogue.

The same pattern repeats. One hundred and two URLs point at decc.gov.uk, the Department of Energy and Climate Change, which became part of another department in July 2016. Two hundred and sixty point at hscic.gov.uk, the domain of a health information body that no longer trades under that name. Others in the failure list belong to regional transport units and NHS organisations that have since been merged.

This is a structural defect rather than an operational one. When a public body is reorganised, somebody transfers its statutory functions, its staff and its records. Nobody is responsible for its outbound links in third party catalogues, because that would require an official somewhere to hold a list of every place its data was referenced. So the catalogue quietly accumulates a fossil record of the last fifteen years of Whitehall restructuring, indistinguishable at a glance from working entries.

So who still publishes?

Ranked by volume the answer is consistent and narrow. The Office for National Statistics, the Marine Environmental Data and Information Network, Natural England, the British Geological Survey, the Scottish Government’s spatial data infrastructure, Open Data NI, the Environment Agency and a handful of research institutes hold thousands of datasets each. A few councils, notably Camden, York and Leicester, sit unusually high for their size. Everyone else has dozens or none.

The concentration is not evidence of failure. Environmental, geological and marine data is genuinely voluminous, genuinely reused and genuinely produced by bodies whose statutory job is to produce it. That is what a sustainable open data publisher looks like: one for whom publishing is a by-product of the work rather than an additional duty. The 305 organisations registered with nothing to show are the mirror image, bodies that signed up during a period when signing up was expected.

Meanwhile data.gov.uk itself has been rebranded. The homepage now presents the site as the National Data Library, describing itself as the home of UK public data to inform decisions and build services. That is a new frontage on a catalogue with a fifth of its links broken and a seventh of its entries holding no file.

One field would fix most of this

Nothing here calls for a platform. The catalogue needs one honest field and one cheap job.

The field is a declared update frequency, set by the publisher, separate from any timestamp a harvester can touch. A reader who knows a series is annual can tell a fourteen-month gap from abandonment. Today they cannot, and every judgement they make about staleness is guesswork.

The job is a link checker, run monthly, with the result displayed on the dataset page. It is a weekend of work. Its output would be the first honest description of the catalogue’s condition that any user has ever had, and it would make the 620 orphaned health links visible to the people who could remove them.

Counting rather than asserting is the standing method of this desk, and the wider argument for it sits under data and analytics. Why any of this machinery was built in the first place, and who stopped paying attention to it, is the subject of open data began well, then it stopped.