The Post Office Fujitsu Horizon scandal
December 30, 2023 3:03 PM   Subscribe

Mr Bates vs. The Post Office (video preview) is a dramatised account of the Post Office Fujitsu Horizon scandal [previously]. Full background of the case is available in 'The Great Post Office Trial' a series of BBC Radio podcasts from 2020. At the end of 2023 not a single postmaster has been given full financial compensation, making the compensation scheme a second scandal in itself. So few claims have been processed that the compensation pot has been reduced by half. Meanwhile as the Post Office Horizon inquiry concludes, lawyers say enough evidence has emerged for police to consider prosecuting former Post Office executives, which may include Paula Vennells CBE, the former CEO of Post Office Ltd.
posted by Lanark (41 comments total) 17 users marked this as a favorite
 
The tl,dr; around the scandal for those who haven't clicked through (apologies if I glossed over some parts):

The UK privatized parts of the postal service and offered a way for local business to install a terminal letting them sell postage and provide other aspects of the postal services. The software was so poorly written and the process so opaque to all parties that people with these machines started running negative balances, some to the tune of tens or hundreds of thousands of dollars - out of desperation, some people took out loans to balance the books in these faulty systems. The Post Office Company sued (and jailed) many people for fraud before it finally came to light that the this was bad software and inept corporate practices, not rampant systematic (and distributed) fraud being committed. It's a sad and depressing tale around how poor software development for even mundane things like postage stamps can have catastrophic consequences - at least one person ended their life over the debts and lawsuit and many more have criminal convictions based on evidence produced by unquestioned outsourced software.
posted by mrzarquon at 3:24 PM on December 30, 2023 [14 favorites]


I read the old thread about the previous post and can't help going back to the same thinking I've had all the way through. There's something in the culture of IT that makes this more likely.

A certain amount of insiderness and obscurantism plus engineer mindset which means people often blame the users, and seldom the computer or the work practices.
posted by treblekicker at 3:47 PM on December 30, 2023


+ previously@2020
posted by away for regrooving at 3:57 PM on December 30, 2023


There's something in the culture of IT that makes this more likely.

"Move fast and break things." That's the summation of it. The push to move forward no matter who gets hurt in the process.

Which is why society needs to respond with "And if you break people in the process, then we break you."
posted by NoxAeternum at 4:00 PM on December 30, 2023 [11 favorites]


Yeah, I've been re-reading the old thread to refresh my memory before commenting here and I completely agree with this comment in that thread, from Nelson (so if upvoting, credit the person who wrote it):
I'm annoyed at how all the reporting and discussion of this event talks about "the software sent them to jail". No. A court sent them to jail. A judicial system and a bunch of people and employers acting in bad faith sent their employees to jail. A buggy software system written by bad software engineers was a contributor, sure, but it takes a system to send someone to jail.

Pinning it on the software is a way to absolve personal responsibility. The folks who use and oversee the software are responsible. So are the programmers, although per usual I bet they never get held accountable.
This is absolutely right. There were multiple serious systemic failures that contributed to this and the coding errors, while not acceptable, were probably the least surprising part of the failure. It is foreseeable, perhaps even inevitable, that any complex IT project is going to have errors.

But the behavior of Post Office management, the Crown Prosecution Service, the UK court system, and other parties that had or ought to have had a duty to include human judgement as a critical part of their decision-making is far less understandable to me.
posted by Nerd of the North at 4:11 PM on December 30, 2023 [14 favorites]


It’s also not unheard of for the actual programmers to ask for test cases, time to write test cases, alerts to throw for failures, and have them removed or ignored by management.

Most famously one of the shuttle disasters, yes? But I suspect that the higher ups often want a bad system to use as cover.
posted by clew at 4:42 PM on December 30, 2023 [1 favorite]


There's something in the culture of IT
and
"Move fast and break things." That's the summation of it.

The outfit that made Horizon, Fujitsu (nee ICL), was institutional computing of the old guard (think e.g. IBM or Unisys in the USA). The architects of today's most modern computing discouragements very likely saw themselves as embodying the opposite of that kind of (perceived) stodgy, slow-moving, mainframe-dinosaur kind of corporate culture. I think "Move fast and break things" is an expression coined partly to distinguish hoodie-clad Web-era working philosophy from the ways the starched-collar Fujitsus of the world liked to do business.

You could hypothesise that there's "something in the water" of working in computers that makes people make certain kinds of mistakes no matter what they wear, but it would be useful to identify the reasons that this should be so. "A pox on both their houses" is plenty valid, though, regardless.
posted by Chef Flamboyardee at 4:49 PM on December 30, 2023 [7 favorites]


While it's tempting to throw the software engineers under the bus, large IT engagements with consulting firms are almost inherently broken, particularly in the heavyweight models used back in the day (I'm still in IT but prefer to work "in-house" nowadays.)

Characteristics include:
* Big promises made in back rooms, almost always some degree of minor corruption in the forms of trips or something.
* The client has, or develops, their own internal powerbase of stakeholders, project managers, etc.
* Heavy "waterfall" methodology, involving development of detailed "requirements gathering"
* Engineers working ridiculous overtime to deliver code that fulfils the requirements, because the bid underestimated the actual effort needed by at least 50%.
* Due to pressure and fatigue, lots of design compromises are made. Requirements are met in the bare-minimum context.
* Generally with waterfall it's the "corner cases" (e.g. intersection of edge, or special, cases) that get ya. Even the best analysts will only identify some of them. The others don't get discovered (if they get discovered at all) until the implementation phase. Depending on time and scope pressures, they may get backstopped in some basic fashion, or they may get swept under the rug.
* The big consulting firms generally prefer to staff with a lot of young professionals just out of college, as they work cheap and don't have families so they can be flown into a city and worked for crazy hours. They are not likely to be great at identifying the traps inherent in the corner cases above.
* Once the project is done, the client generally lacks the technical capability to maintain the codebase; partly because they haven't cultivated the necessary skillset, and partly because the codebase is crap. This means that either the client is stuck with an immutable mess, or the consultant retains a foothold with the client permanently, for that sweet recurring revenue.

From my observation, these heavyweight projects ranged from "abject failure" to "massively over time/budget." Completion of promised functionality on time, and in a maintainable state, was always wildly unlikely.

It's possible to deliver projects in a timely fashion, generally using iterative/agile methodologies. Hopefully the big consulting firms are using those now and sucking less? I'd be interested to hear from anybody that has more recent knowledge on that front.
posted by microscone at 5:56 PM on December 30, 2023 [8 favorites]


Yeah, “move fast and break things” is a misdirection. This is huge, staid enterprise IT with its most common form of diligence theater leading to a system which on paper satisfies all of the requirements but doesn’t, well, actually work. This was the most common outcome from the large consulting companies many years before Facebook even existed, and those companies’ revenue is largely derived from their ability to repeatedly convince other large organizations that they’re not like that.

What made it so bad here is that it combined with the UK legal authorities gross negligence trusting large companies when they make claims about their IT systems rather than expecting truth. They did the same thing with ATM “phantom withdrawals”, accepting credulously banks’ claims that their customers must have been lying about transactions or sharing PINs. The IT angle is really just that some people have far too simple a model of computer accuracy, and I think it’s only part of the larger issue of having special treatment for large, wealthy organizations: nobody should be able to assert infallibility without having a skeptical examination of the entire system.
posted by adamsc at 6:09 PM on December 30, 2023 [4 favorites]


As bad as the software was, a great deal of the blame lies on the Post Office, which (I think almost uniquely?) could initiate private prosecutions on its own. They were the ostensible victim, investigator, and prosecutor. They prosecuted almost a thousand of their own employees! It's just astounding.
posted by BungaDunga at 6:16 PM on December 30, 2023 [10 favorites]


It seems to me that the software failure is a substantially different component of this scandal. It's a pretty normal bunch of business operations failures. The dishonesty of the executives, the judicial system failures and political failures are the really genuinely scandalous part.
posted by srboisvert at 6:23 PM on December 30, 2023 [4 favorites]


There was an earlier UK scandal involving phantom withdrawals from ATMs. In the fine tradition of British justice it was decided that the computer is by definition always right and innocent people who complained were falsely convicted of fraud, with many losing their jobs.
posted by monotreme at 6:31 PM on December 30, 2023 [3 favorites]


I bet there are a bunch of insiders who raised questions about this and were told to shut up or were fired.

Find those people, gather their accounts, and prosecute the people who tried to shut them up and prosecute the superiors of those people all the way up.

And that should include people inside the prosecution establishment who tried to raise questions.
posted by jamjam at 7:00 PM on December 30, 2023 [3 favorites]


This gets into the realm of how you can't give AI responsibility to make decisions of consequence because you can't hold an AI accountable. It's just the people who listen to the AI you hold accountable.

That's a thing that originates well outside of this horrible series of lives ruined. But it originates from before it, and if it were more widely known way back then maybe none of this would have happened.

I do think anyone who ruined someone's life based on this should at the very least be removed from power for abrogating their responsibility to a non-human.
posted by hippybear at 7:16 PM on December 30, 2023


Yeah, this wasn't so much move fast and break things as move slow and find little people to grind to a smooth paste for lubricating the skids.

IT is in general horrible, but IT consultancy is where horrible festers, and the more expensive the project and the higher the associated executive bonuses, the worse it gets.
posted by flabdablet at 10:02 PM on December 30, 2023




On the institutionalized complacency of British Justice . . . In 1977, the Birmingham Six brought a civil suit against the West Midlands Police for damages; holding that they had been assaulted while in custody. The case was struck out on Appeal on the principal of Issue Estoppel. The fact that British justice chooses to dress up their law with jargon derived from Medieval French as they dress up their judges in the fashion of 1720 indicates how out of touch with modern realities the whole system can be. Lord Diplock estoppled the legitimacy of their claim because it "would otherwise bring the administration of justice into disrepute among right-thinking people". The point is that once the law has decided something, you can't question its correctness. It's probably true that "right-thinking people" in those days were prepared to believe all sorts of demonising nonsense about all the Irish, so Diplock was weasily correct in his application of Estoppel in that case.
posted by BobTheScientist at 2:47 AM on December 31, 2023 [3 favorites]


They were the ostensible victim, investigator, and prosecutor. They prosecuted almost a thousand of their own employees! It's just astounding.

That’s what gets me. If it was one case that would be something but if it quickly reaches into the hundreds the week after the new software was rolled out someone in the justice system should have noted something wasn’t right.
posted by jmauro at 3:06 AM on December 31, 2023 [1 favorite]


"Move fast and break things." That's the summation of it.

That's definitely not true in general of projects like this, and having worked in this area I'd be rather surprised if it was the case here. During a career as an independent IT consultant I was peripherally involved in several UK Government projects (including another one by Fujitsu, which also failed), and "moving fast" is about the last thing you could level at them. "Breaking things", yes, but running hugely over time and budget is quite normal, and even expected.

There are obviously some large projects that work out, but from what I've seen there are several contributory factors when these projects fail, such as...
  • The big consulting firms make persuasive pitches. Where government money is concerned they're like sharks scenting blood in the water, and it's amazing what you can do with a good Powerpoint deck.
  • They send in a team, making the team as junior (i.e. cheap) as they can get away with, while charging as much as they can. Heck, late in one project I saw well-thumbed copies of "VB6 for Dummies" on more than one desk, which did not inspire confidence. There was often little or no application of good design or testing practices, especially testing, so all kinds of bugs got through to production.
  • The developers often don't have detailed knowledge of the requirements and the business domain, just being told to code isolated pieces of code.
  • And perhaps worst of all, the project managers on the government side often know sweet FA about IT. Requirements would change from week to week, sometimes in ways that were completely mad [see below]. They couldn't judge whether the project was going well, just taking the word of the consulting firm, and it wasn't until things were obviously screwed that they got independent advice.
(The best requirements change was 6 months into a project, when the project manager from the government side innocently asked if we could change from a Microsoft stack (Windows, VB.NET, SQL Server, IIS) to Linux+J2EE+Oracle. "Would that be a big change?" he asked, and wondered why we went outside and had a scream)

It's just luck that most failed government IT projects don't impact people's lives in the way this one did.
posted by 43rdAnd9th at 8:46 AM on December 31, 2023 [5 favorites]


The Post Office Company sued (and jailed) many people for fraud before it finally came to light that the this was bad software and inept corporate practices, not rampant systematic (and distributed) fraud being committed.

Just a slight addition to this...before it finally came to light to the public that etc.

I have no doubt that numbers of people at various levels in the PO knew full well that the software was at fault at the time they were prosecuting innocent people, sending them to prison, and driving them to bankruptcy and suicide.
posted by reynir at 1:21 PM on December 31, 2023 [2 favorites]


To add a "yes, and" to all of this: a massive part of the problem here is that - under UK law - computers are presumed to be "operating correctly". That means, legally speaking, if it came out of a computer it is presumed to be true and correct unless evidence can be produced to the contrary.

Yes, this is a bad idea.
posted by parm at 5:34 AM on January 2 [2 favorites]


When the first criminal cases were appealed following the civil case presided by Fraser J, the appeals won on two grounds.

"i) Ground 1: the reliability of Horizon data was essential to the prosecution and, in the light of all the evidence including Fraser J’s findings in the High Court, it was not possible for the trial process to be fair;
ii) Ground 2: the evidence, together with Fraser J’s findings, shows that it was an affront to the public conscience for the appellants to face prosecution"

The second point is essentially that the Post Office abused their position by prosecuting despite knowing that a fair trial was not possible, and as a result brought the system of law and justice itself into disrepute.

More than 700 sub-postmasters were convicted during the relevant period and most of those convictions still stand. Anyone whose conviction rested essentially on Horizon data would have their convictions overturned on appeal if they were willing and able to apply either to the Crown Court (if they were convicted at a magistrates court) or to the Criminal Cases Review Commission. I hope the publicity from the ITV drama will encourage more people to come forward and clear their names.
posted by plonkee at 8:11 AM on January 2 [1 favorite]


Post Office victims' compensation pot cut by half
Professor Chris Hodges, chair of Horizon Compensation Advisory Board, said in a letter to the government an unwillingness to appeal was due to a "deep distrust of authority", evidence being lost or destroyed and issues with compensation if a Post Office manager is not granted a retrial...

Last week, Prof Hodges said the convictions were "unsafe not only because they relied on the Horizon computer evidence, but also because of egregious systemic Post Office behaviour in interviews and pursuing prosecutions".

"This led to guilty pleas and false confessions, driven by legal advice to victims to minimise sentences, and by the psychological pressure of dealing with an institution systematically disregarding the truth and fairness," he said.

His board therefore decided the only viable approach was for all Post Office-driven convictions over the Horizon period to be overturned, so that the victims of the scandal could be[sic] receive compensation.
(to be clear, that conclusion hasn't been implemented, victims need to individually appeal their sentences to get them overturned)
posted by BungaDunga at 10:52 AM on January 2


Criminal investigation for potential fraud launched, Metropolitan police interview two people under caution.
posted by Lanark at 4:24 PM on January 5


Went to the Bay and found the ITV 4-part series about this that's on television right now and making all the waves that previous attempts to get the public's attention had failed to achieve.

I'm well outside of the UK and don't know where I might see it otherwise, but I'm glad I watched it. It's hella damning, really. A full-on example of why you should never trust a computer system to say whether anyone is guilty of anything.

I highly recommend. There's a torrent with a zillion seeds, so check it out if you're interested.
posted by hippybear at 6:55 PM on January 9 [1 favorite]


A couple of interesting presentations on the technicalities:
The Horizon IT Scandal: From Computerphile a couple of years back. Talks about what the five ("ACID") essential design criteria for creating a transaction processing system - and why Horizon was shown to have failed on each. What stands out to me here is that - for all its scale - Horizon was not being asked to do anything remotely novel or terribly hard: sums, really.
Delivering the Fail - explains the various laws broken by the Post Office and Fujitsu. The point that the Post Office needed to try to maintain public and business confidence in its useless system.
posted by rongorongo at 3:45 AM on January 10 [2 favorites]


Rishi Sunak announces plan to pass law quashing Horizon scandal convictions

It really does seem like the publicity from the TV show forced his hand. Never mind the public inquiries, books, podcasts, news reporting- if it's on TV, then it matters.
posted by BungaDunga at 6:06 AM on January 10 [1 favorite]


In case anyone thinks this has just blown up recently: The Computer Weekly articles about the scandal since 2009    posted by Lanark at 9:28 AM on January 10 [12 favorites]


I've been hearing about this for years, but didn't realize the coverage had been that entirely blanketing for nearly 15 years. Honestly, until this most recent blow-up in the media, I thought I'd read that the problem had been recognized and the situations were being resolved. It's really shocking that it even now STILL is only beginning to be resolved.

I hope the legislation that gets passed makes everyone as whole as they can be made. Watching that ITV series was really infuriating.
posted by hippybear at 12:46 PM on January 10


Also, Lanark, you creating that linked list of articles -- I know how much work that is, and thank you.
posted by hippybear at 12:47 PM on January 10


In case anyone thinks this has just blown up recently: The Computer Weekly articles about the scandal since 2009
Kudos to writer Gwyneth Hughes and the other behind Mr Bates vs The Post Office, however. It takes considerable skill to take a story well known in the esoteric worlds of Software Engineering and, accounting and Law and give it the emotional punch that a story of false accusation and lives ruined deserves. I suspect there are many people who don’t understand the technical intricacies but who can immediately sympathise with those caught by flaws in supposedly infallible IT systems.
posted by rongorongo at 3:14 PM on January 10


Private Eye have made their report into the Horizon scandal free to access.

I hope we'll see some actual accountability of those in power who presided over all this. Some jail sentences for them might be a tiny step in the right direction for the UK justice system.

Thanks for the link to the Delivering the Fail talk rongorongo. It's a damning report, drawn mostly from Fujitsu and the Post Office's own reports.
posted by amcewen at 3:25 AM on January 11 [2 favorites]


“Nearly a Thousand People Were Convicted of Stealing Over Decades. It Was a Computer Glitch.” Max Colchester and Joanna Sugden, The Wall Street Journal, 10 January 2024
posted by ob1quixote at 9:05 PM on January 11


You know, I don't want to get all James Somerton on the authors of that Wall Street Journal article, but there is nothing contained within it that isn't in the documentary Mr. Bates vs. The Post Office: The Real Story. Like literally tiny details are identical. Is there any original reporting in the piece?
posted by hippybear at 4:51 PM on January 12


If you thought the discourse had plumbed the depths, it turns out a) that 40% of those prosecuted are POC b) that PO management used racist codes / classification in their investigations [they're very sorry that they failed to disclose that to the inquiry]. I'm reserving particular side-eye for pols and pundits who are only now joining the pile-on being shocked, dismayed and finger-pointing IRL because they saw Toby Jones being British on the telly.
posted by BobTheScientist at 2:10 AM on January 13 [4 favorites]


Let’s not forget the detail of the Post Office incorrectly claiming the compensation they paid out as tax deductible - this inflated their apparent profit which was used a justification for paying large bonuses to their chief executive. This puts them about £100 million in the red and may make them insolvent.
posted by rongorongo at 5:11 AM on January 13 [2 favorites]


It’s been very gratifying to see the post office scandal finally blowing up. I first heard about it from Marina Hyde, who’s been on it for years. But I wanted to highlight someone who was way ahead of everyone, Rebecca Thomson, who in 2009 broke the story in the aforementioned Computer Weekly. She was interviewed about it two years ago in the Times [archive link], on of many, many national news organizations that didn’t pick up on the story. Among the many incredible details in her interview, one I hadn’t seen elsewhere is that after she left journalism for a job at an accounting firm, the Post Office used their connection to the firm to silence her, years and years after she’d stopped reporting on it.

Also, while I’m at it, for people who have just heard about this story, it’s easy to think that the revelations are new. In the broad details, the facts of the case have been publicly known since 2012, it just took until now for the media to give this story the blanket coverage it should’ve gotten in 2012, or indeed 2009 when Thomson and Computer Weekly broke the news. The Computer Weekly’s timeline is very thorough.
posted by Kattullus at 12:33 AM on January 14 [4 favorites]


The Post Office incorrectly claiming the compensation they paid out as tax deductible
When you add on the likely HMRC penalties of up to 30% for this, the Post Office could be paying more in tax than the £138 M they have so far paid to the Postmasters.
Though given the Post Office is owned by the government, the tab will be picked up by UK taxpayers either way.
posted by Lanark at 2:55 AM on January 14 [1 favorite]


Is Post Office Limited part of the UK government? Wikipedia suggests it's a private corporation spun off of the Royal Mail.
posted by hippybear at 7:25 AM on January 14


The Post Office is a limited company owned entirely by the government, no doubt they had plans to sell it off at some point (as they did with Royal Mail) but that hasn't happened yet.
posted by Lanark at 7:55 AM on January 14


Also the individual postmasters are all franchises, so those are effectively separate small businesses. I think that's where a lot of the "them" and "us" attitude comes from in this case.
posted by Lanark at 5:49 AM on January 17


« Older 2024 Anti-Trans Legislative Risk Map   |   The Battle of Bamber Bridge Newer »


This thread has been archived and is closed to new comments