How is government using big data? The Bureau wants to find out

AI, algorithms, deep learning, big data - barely a week goes by without a new revelation about our increasingly digital future. Computers will cure cancer, make us richer, prevent crime, decide who gets into the country, determine access to services, map our daily movements, take our jobs away and send us to jail. Successive innovations spark celebration and concern.

The UK should be “ready, willing and able” to profit from the economic potential of being a world leader in the artificial intelligence industry, a recent government report emphasises. Academics and civil society, meanwhile, sound warnings over corporate accountability, the intrusiveness of personal data and the ability of legal frameworks to keep pace with technological challenges.

These concerns become particularly acute when it comes to the use of digital technology by the public sector, which is compiling ever larger datasets on citizens as it moves towards an increasingly digitised future. But how this is happening is still a huge unknown. Questions abound about what national and local governments are already doing with data, who they are paying to do the work, and what the potential outcomes could be, especially for society's most vulnerable people.

Attempting to shed some light on those questions is of clear public interest. As such, the Bureau has been looking at this area with a view to launching it as an investigative theme. We started by undertaking a six-month scoping exercise, examining what IT systems the government is buying. In a report out today, ‘Government Data Systems: The Bureau Investigates’, we release our initial findings.

Our report looks at how to use publicly available data to build a picture of companies, services and projects in this area. We trawled through official transparency data, scraped websites, dug into big datasets, and made dozens of freedom of information requests to public authorities. We also interviewed experts, insiders, academics and members of civil society.

Government IT contracting is a flourishing sector. We found more than 1,800 IT companies selling short-term contracts to government, and almost 300 providing more complex services. The Home Office's plans to become a “data-driven organisation” have required work by more than 40 companies in just the last two years, our research revealed.

What it didn’t reveal, however, was precisely what all these companies are working on, or why they were hired in the first place. From start to finish we encountered systemic problems with disclosure, despite the UK's claim to lead the way in open data.

Transparency - and therefore accountability - over the way in which public money is spent remains a grey area. This should ring alarm bells. The state is in the midst of a complex data-driven revolution predicated on saving money through major digital transformation programmes. But its citizens often remain in the dark about what the changes will entail and who is being paid to carry them out.

Sign up to our mailing list and follow @TBIJ on Twitter to keep updated. You can find the summary of our report below, and the full document beneath that. To download the report, use this link.

Government Data Systems: The Bureau Investigates

Executive Summary

• Development of algorithmic and data-driven systems is frequently predicated on austerity - doing more with less.
• The adoption of such systems, the combining of legacy databases and the roll out of “digital by default” services is a major driving force in public policy.
• Information technology contracting for government is a flourishing area: although still dominated by traditional big names, many lesser known companies are also offering a wide variety of services.
• The Home Office’s ambitious plans for new data-driven systems have required the assistance of over 40 companies in just the last two years.
• We traced several types of government purchasing activities for digital systems, finding over 1,800 companies selling simpler or shorter-term consultancy, software and storage services and almost 300 providing more complex requirements.
• Many authorities were unwilling or unable to specify how and why they purchased these services, however, or what their precise specifications were.
• Public authorities - national and local - are supposed to keep transparent and accessible records of the services they purchase (in part to comply with the Public Contracts Regulations 2015). We found that this was rarely the case.
• Government transparency datasets are an inadequate tool for understanding purchases, particularly in the case of highly diverse large companies which offer a multiplicity of services (true of many major government contractors). The UK lags massively behind the US’s granular approach to public spending available through the Federal Procurement Data System, for example.
• Transparency - and therefore accountability - over the way in which public money is spent remains a very grey area in the UK. This is concerning, particularly at a time when the state is driving a complex data-driven revolution predicated on saving money through major digital transformation programmes and legacy system overhauls.