The Fujitsu Scandal: A Lesson in Lazy Procurement, Unaccountable Power, and Inexpert Design for the Age of AI

In a modern world where technology increasingly intertwines with every aspect of business and governance, the Fujitsu scandal serves as a stark reminder of the pitfalls of a flawed procurement process especially in large organisations, a culture in of unaccountable power Whitehall and beyond, and the lack of adequate expertise in managing complex digital systems to robustly challenge their infallibility when the “computer says no”.

The scandal unfolded as it became evident that Fujitsu, a large and reputed contractor, was allegedly deeply involved in a series of missteps and malpractices. This situation arose primarily due to a procurement culture that favored large contractors under the assumption that bigger entities equate to lesser risk versus startups. As we have seen countless times in the past this is not necessarily true. Large organisations may seem at the time to be lower risk because they seem to be more financially more resilient, but their size can mask underlying complexity that can in turn lead to sudden collapse or miscarriages of justice. Organizations like the civil service or public bodies, especially those under pressure, often opt for the seemingly safe route of engaging with established names to save time and effort. This approach, while seemingly pragmatic, overlooks the potential risks and downsides of not thoroughly vetting vendors and their offerings but also of assuming big is always best. Over the years I and others have advocated for other actors in society to get more of a look in, who may be individually small at the scale of the single household or small business or charity, but collectively can bring innovation, fairness and transparency, if given the time to prototype, pilot and prove their approaches. Estonia is often cited as a prime example of how in a small country with relatively tiny but nimble suppliers, they have been able to build smart systems that fail much less than we have seen in government over recent decades. In today’s Britain despite our best efforts, small but smart actors rarely get the chance they deserve. One former senior civil servant said to me that during his time running a major department he never bothered in the end trying to meet startups no matter how amazing or effective their solution was at helping improve customer service, productivity or effectiveness because they were rarely allowed to get through the monolithic and time and money consuming procurement processes in government.

Adding complexity to this scenario is the role of quasi-autonomous non-governmental organizations (quangos). I have been alarmed as a legislator and in observing the behaviour of Whitehall and other public bodies now buckling under an ever increasing workload, often because we now have more data than ever brimming with false positives. With increasing demand from the public for action whether in health, social services, or from business, civil servants and front line staff will often seek to craft or influence laws that reduce their workload (look up Henry the VIII clauses for example), and speed up the conclusion of individual cases, but in clumsy and monolithic ways. Many of these bodies now often have the power to prosecute, a responsibility that comes with its own set of challenges. In the Fujitsu case, the empowerment of the Post Office to prosecute to speed up processes of tackling fraud contributed to an imbalanced environment where overweening power lay in the hands of its executives, far from Parliamentary or any other accountability, with life destroying consequences. If bigger is always best has been proven to be wrong, then in this case the idea they government or the boss knows best and should be given tools to get the job done has also been shown to be a fallacy.

A significant third aspect of this scandal is the evident neglect of common law principles in the design and implementation of Fujitsu’s software systems, reflecting a wider problem with software design generally as well as particularly within the UK, and especially in government systems due to cost and other pressures. Common law principles, which emphasize fairness, transparency, and accountability, are crucial in the realm of software development today: the notion that people or users should be in some presumed innocent before guilty, have mechanisms to genuinely appeal, and have human support advocating for them analogous to a jury of their peers – these are key. Their absence in this case points to a broader issue in the tech industry, where the rush to innovate and deliver often trumps the need for ethical and legal considerations in software design. In government procurement this general culture is exacerbated given the rushed timelines, changing demands of ministers, and extreme cost pressures. The truth is when the computer says no, it is not only not always right, but we need to design the systems to assume it won’t always be right so it can be corrected and so people have recourse to justice and humane or forgiving treatment even as they interact with the modern systems, databases, and forms that represent effectively citizens’ experiences of power today.

Another key factor in the Fujitsu scandal is the lack of expertise in managing and questioning the reality presented by digital twins. Digital twins, or digital replicas of physical systems, are becoming increasingly common in various industries and sectors. These can be digital versions of supply chains in industry where physical goods are tracked going from one stage in a production line and from one company to another on the journey to being sold, or they can also include services provided to clients such as in a government context mapping the state of a citizen’s financial situation, or in the case of the Post Office, the accounts provided by Postmasters based on their sales. When these digital twins are not sufficiently well anchored in verified data, either collected automatically or as validated by third parties (eg humans who do manual inspections and sampling) errors can creep in, which at the scale of a country can mean thousands if not millions of people who may have life critical decisions made about them, accusing them of fraud, non-compliance or other misdemeanors which may or may not be true. Without experts who can forensically analyze and question the data and processes represented by these digital systems, and check for errors, taking into account complaints or appeals by humans, organizations run the risk of being misled or misinformed by the technology they rely on. That is clearly what has happened in the case of Fujitsu, where Postmasters were ignored because nobody was able or willing to question the data coming into and from the systems. It could equally be parents complaining that the system accusing them of abusing or neglecting their children is inaccurate, banks deciding what customers to keep, or those accused by the police of crimes who have been inaccurately linked to them. The issue at stake here is that the computer is not always right, especially if the data it is fed or generates is faulty, and human oversight is always necessary.

The Fujitsu scandal is not just a standalone case; it is a precursor to the challenges we will increasingly face in the age of AI. As we delegate more decision-making and operational processes to artificial intelligence and digital systems, the need for robust procurement processes, adherence to legal and ethical standards in software design, and the availability of experts capable of interrogating digital systems becomes paramount. We need to force procurement processes to enable smaller nimbler players to prototype and then pilot next generation systems over longer periods of time, to replace the current ones that have been procured today as and when they inevitably fail. We need to change the culture of expediency that grants too much power into the hands of the few and rebuild the accountability mechanisms not just of Parliament but also of other tech enabled mechanisms such as prediction markets for projects and policies to hold them to account and give early and ongoing estimates of their likely success. We need to embed common law principles into software and not just make laws to mop up after they have destroyed countless lives, since most problems happen well before they find their way into court these days. And we need to raise up and empower forensic investigators who advocate for the rights of those who cannot defend themselves as AI begins to take over every system citizens have to deal with, before it is too late.

In conclusion, the Fujitsu scandal is a cautionary tale that underscores the need for a more balanced, thoughtful, and well-informed approach to technology procurement and implementation. As we move further into the digital age, it is essential that we learn from these mistakes and implement strategies that prioritize transparency, accountability, and expertise in their dealings with technology and its providers. With a British media often obsessed with personalities, with the politicians and executives who seemingly presided over these failures held up as the culprits, we can often miss the real issues, and neglect to really investigate them especially once the news cycle moves on. The truth is, our system and culture of government and corporate decision making has failed here at every level – it is not just about the particular individuals concerned in this sorry tale, but about the systems and the culture that have enabled them – which urgently need reform.