Between 1999 and 2015, over 900 sub-postmasters in the United Kingdom were prosecuted for stealing money from the local Post Offices they were tasked with operating. The main witness against them was a software application called Horizon developed by Fujitsu that was introduced by the British Post Office in 1999.
Beginning in the late 1990s, the Post Office introduced a new IT system called Horizon, developed by Japan's Fujitsu, to replace the traditional paper-based accounting methods. However, this system soon proved to be deeply flawed, regularly displaying financial discrepancies and showing that money, often amounting to thousands of pounds, had mysteriously gone missing from the Post Office accounts. Actually, all this had to seen with an attempt of the British Government to save his local IT industry (ICL, see comment from raffraffraff)
Despite the sub-postmasters' attempts to reconcile the errors, the Post Office took a relentless stance. Over the course of two decades, from 1999 to 2015, the government-owned organization pursued legal action against thousands of affected sub-postmasters. Out of these, an astounding 700 were convicted of criminal offenses, with some even serving time in prison.
The impact on the lives of these innocent individuals and their families was devastating. Livelihoods were destroyed, reputations tarnished, and savings wiped out. Some sub-postmasters, unable to bear the shame and distress, tragically took their own lives.
The story of Jo Hamilton, a post office manager in southern England, exemplifies the harrowing experiences of these victims. In 2003, her Horizon computer system displayed a shortfall of £2,000, which quickly doubled to £36,000 by the time the Post Office took her to court in 2007, charging her with theft and false accounting. Facing overwhelming pressure, Hamilton had no choice but to plead guilty to false accounting, despite the charges being unfounded.
This scandal has been the subject of extensive legal battles and media coverage for years, but it was only recently, with the broadcast of a TV drama highlighting the brutal human toll, that public awareness and outrage have reached a fever pitch. The UK government has acknowledged this as one of the gravest miscarriages of justice in the nation's history, leaving a profound and lasting impact on all those affected. Actually, Fujitsu’s bug were known since 1999!
However, the UK's political and legal establishment still faces crucial questions that demand answers. Who was aware of the systemic issues within the Horizon IT system, and when did they become aware? Which individuals, if any, should be held criminally responsible for the devastation inflicted upon these innocent sub-postmasters and their families? And to what extent should Fujitsu, the Japanese technology giant worth $30 billion that developed the flawed Horizon system, be held accountable and required to provide compensation to the victims.
The Post Office has acknowledged the gravity of the situation, stating that it has already paid over £138 million ($176 million) in compensation to the affected sub-postmasters.
And now comes the point: this software didn’t have any artificial intelligence line of code. What would happen when they have it? I find interesting the words by blogger Ben Evans:
Fujitsu was not building machine learning or LLMs - this was 1970s technology. This was an institutional failure inside Fujitsu and inside the Post Office, and in a court system failing to test the evidence properly. And, to be clear, the failure was not that there were bugs, but in refusing to acknowledge the bugs. Either way, to take the language that people now use to worry about AI: a computer, running indeterminate software that was hard to diagnose or understand, made ‘decisions’ that ruined people’s lives - it ‘decided’ that money was missing. The staff at the Post Office just went along with those decisions.
The rise of advanced artificial intelligence, particularly generative AI, has transformed the pace at which various processes can be completed. The very same speed that makes generative AI so powerful can also mean that negative consequences can manifest much faster. In the case of the UK Post Office scandal, one would have expected the British prosecutors to start questioning why they were prosecuting a new sub-postmaster every week, even before the implementation of the Horizon IT system.
The need for human involvement is rapidly diminishing for some AI applications, but many concerned about AI’s speed are starting to prompt some pushback to this lack of participation. The proliferation of apps and automated systems designed primarily for the system's benefit rather than the users is becoming widespread, with discretion mainly sidelined.
The proliferation of apps and automated systems that seem to be designed primarily for the benefit of the system itself, rather than the users, is becoming increasingly widespread. In these cases, user discretion and control appear to be largely sidelined, as the systems operate with increasing independence and efficiency.
This trend raises valid concerns about the appropriate balance between AI capabilities and human involvement. While the speed and efficiency of AI-driven processes can be advantageous in many scenarios, there is a growing recognition that this rapid advancement may come at the cost of user agency, oversight, and the ability to meaningfully engage with the systems that govern so many aspects of our lives.
Ya afloran nuevos empleos como "Supervisores de IAs". Y también el reto de diseñar mecanismos de incentivos bien compensados que eviten tanto una excesiva complacencia como una aplicación tergiversada de la crítica a los resultados de la IA. En el camino, aparecerán damnificados como estos...