The Return key isn't working
April 13, 2019 10:28 PM   Subscribe

With tax day (in the USA) just around the corner, time for a dive into the 'tech' of the Internal Revenue Service. LGR posts on How the 60-Year-Old IRS Computer System Failed on Tax Day 2018. (IBM System/360 previously)
posted by KirkpatrickMac (22 comments total) 14 users marked this as a favorite
Sigh. Is there a transcript available? (As text, not as images of text superimposed on the film?)
posted by meaty shoe puppet at 3:10 AM on April 14 [1 favorite]

As someone who supported scanning and indexing of paper returns, all I can say is "Ha HA!"
posted by mikelieman at 3:26 AM on April 14 [3 favorites]

I used TurboTax again this year. It was even more of a pain than last year, and failed in different ways. I completed its hoops, and submitted my returns. It told me they were sent. Days later, and with time running out, I discovered that they weren't sent, so I tried to submit them again. TT wanted me to pay again for my state return. I entered the confirmation code they'd given me when I paid before. TT rejected it repeatedly. Finally, for no apparent reason, it decided I was paid up, and I was able to submit my returns again. This time, I got confirmation. Intuit sucks.
posted by Kirth Gerson at 5:05 AM on April 14 [4 favorites]

Is there a transcript available?
Select the “. . .” next to Save under the video. A transcript is in there.

There's not much of a story. It's mostly trying to make a lot of the fact that the code to maintain the US Individual Master File is written in COBOL and IBM 360 Assembly Language:
And as of the making of this video, the IRS still relies on those old programs written in COBOL and IBM assembly languages. Remember the Individual Master File, the giant database of all individual taxpayers? Well, the IMF is still relied upon to reference all that taxpayer data, meaning that even though the computer hardware itself has been upgraded, they still have to emulate the computer systems from the Kennedy Administration. Not only that, but every time the US tax code changes, the old programs have to be updated, and this has caused a massive headache for the IRS. According to a Government Accountability Office report in 2016, some 20 million lines of code are still used that date back to the creation of the IMF in the 1960s. And this ancient programming is only going to result in greater challenges as time marches on, something the government has been aware of for decades.
But in reality:
And this brings us to the outage of April 17, 2018. According to a Treasury Inspector General report later that year, a known firmware bug caused a Tier 1 high-availability storage array to fail at around 3 AM on Tax Day. This was an 18-month-old piece of hardware installed to support the Individual Master File. That morning it detected a deadlock condition after a warmstart due to cache overflow, causing 59 systems in total to fail. Since almost all other IRS services and systems ingest data from the IMF mainframe, they too failed, and e-File systems were offline for 11 hours during Tax Day. It was later determined that buggy firmware on their IBM hardware was to blame, something that IBM had known about since June of 2017. And IBM released a firmware update fixing the bug that November, five months before Tax Day. But after a December 2017 meeting between them and Unisys, the agency’s storage contractor, the IRS decided not to update on the advice of Unisys, since the older firmware was thought to be more stable in supporting the all-important Individual Master File.
So not actually about 20 million lines of code from the Kennedy Administration, but more about a subcontractor not implementing a driver patch in hardware from 2016.

Can you imagine rewriting the US tax system in $MODERN_LANGUAGE? There would be decades of weenying about discussing what framework to use, or tabs vs spaces in the code indenting. Contrast that with Grace and Jean who defined COBOL, running on computers with roughly the same amount of memory as your car keys.
posted by scruss at 5:49 AM on April 14 [22 favorites]

Intuit sucks

Not as much as H&R Block, who this year tried to tell me that I somehow owed just under $1300. I had to go over to TurboTax and enter everything again to get the proper figure. /tangent
posted by Mr. Bad Example at 6:27 AM on April 14

It's mostly trying to make a lot of the fact that the code to maintain the US Individual Master File is written in COBOL and IBM 360 Assembly Language

And, yet, those who would be all LOLCANYOUBELIEVEHOWANTIQUETHEGOVERMENTSYSTEMSARE??? probably wouldn't be cool with spending tax dollars to bring that code into the 21st century. Rather, they'd use the antique code (regardless of whether the outage was actually a contractor fuckup) as a reason to kill-off the IRS completely.
posted by Thorzdad at 7:08 AM on April 14 [2 favorites]

I'd rather we spend the tax dollars on that, instead of bombing innocents abroad.
posted by xedrik at 7:22 AM on April 14 [5 favorites]

The tendency to fetishize newness in software is total madness when you're talking about systems that are meant to be rock-solid. That ancient cobol code has 50 years worth of debugging edge cases built into it. It is literally impossible to write new java or whatever to replicate that stability.

I can't think of a good simile, so I'll go with a bad (but hopefully close-enough) one instead: rewriting working systems from a half-century ago in modern languages is equivalent to rewriting the entire legal code in lolspeke.
posted by Reclusive Novelist Thomas Pynchon at 7:24 AM on April 14 [17 favorites]

There's a PDF from TIGTA (IRS's watchdog). It explains what happened in great detail. The takeaway for me was less "let's rewrite everything in Rust" and more "production environments are hard, contractors should escalate incidents appropriately and document decisions made about the viability of firmware updates"
posted by RobotVoodooPower at 7:32 AM on April 14 [8 favorites]

Also: imagine the meetings
posted by thelonius at 7:33 AM on April 14 [8 favorites]

Here in the NL where I live IT projects from the tax department are challenging and uncool too. Maybe not as much assembly code. But a lot of other uncool technology like Websphere, business rule engines, and probably Cobol too.
Here in tthe NL our government has a bigger role and subsidies and tax breaks are a major part of how politics tries to achieve desirable change.
That means that our tax code and subsidies/allowances (toeslagen) are as complicated as the social goals that the government tries to achieve. Which is: quite complicated.
The slogan of the Dutch tax department is "we can't make it more fun, but we can make it easier". Which, when I think about it, is a great strategic vision.
And to be honest: I can go to the website of the tax department, login and go through their webapplication that asks all the relevant questions and follow up questions. And almost all of the data is prefilled in: mortgages, savings, income. Filing my taxes is 5 mins of work.
So for a government agency with lumbering, expensive and sometimes failing IT projects I guess they're doing quite well considering.
posted by jouke at 9:32 AM on April 14 [8 favorites]

My IRS tax story is from an acquaintance who worked for a company tasked with improving how they processed tax returns. He discovered that they opened the return, set the check aside, then entered the return into the system.

The important part was that they set the check aside. As long as you owed money, you calculated your tax correctly, and included a check which cashes, you were good to go. No one was actually checking that the amount on the check matched what was owed. You could owe $1000 and include a check for $10, and as long as your tax return was correct no one was the wiser.

They fixed the process.
posted by eye of newt at 9:35 AM on April 14 [2 favorites]

Am I the only one who wants to see some of the IRS sourcecode just for interest's sake? I've always wanted to see non-trivial COBOL/assembly.
posted by Alensin at 9:56 AM on April 14 [4 favorites]

It is literally impossible to write new java or whatever to replicate that stability.

The IRS was engaged in a project to translate the assembler code to Java, and the IRS even applied for patents on technology. The article is from last year; I'm not sure if that project is moving forward.
posted by RobotVoodooPower at 11:23 AM on April 14 [2 favorites]

yes, and for the first ten years after it’s put into service it will be less reliable than the mid-20th-century code.
posted by Reclusive Novelist Thomas Pynchon at 11:27 AM on April 14 [1 favorite]

In a speech to the National Press Club last April, Koskinen mentioned Wang and his colleague Mark Yu “who developed a method for translating the programming language used in our legacy tax processing applications into the JAVA language.”

Why, after 25 years, there are still people who think that Java is spelled with CAPSLOCK, is beyond me. It never was! NEVER. NOT AN ACRONYM.
posted by thelonius at 11:28 AM on April 14 [2 favorites]

Well, COBOL has all caps, but JAVA has one less letter, so it's got to be more efficient. Not as good as dynamic programming, mind you, but pretty good.
posted by RobotVoodooPower at 12:10 PM on April 14

The language used is irrelevant. It is the function that is important. Although I suspect that if it is like many of the old systems that I have worked on, the only way to to find out what the functions really are is to decode the Cobol or PL/1 or Fortran or Assembler.
posted by Burn_IT at 1:06 PM on April 14 [1 favorite]

I'm pretty sure getting TurboTax to properly recognize my filing-state-taxes-in-two-states situation required entering COBOL code directly onto the form.

(Assuming, of course, that I entered everything correctly. If I messed up the PA return, I expect to be disappeared sometime within the month.)
posted by delfin at 3:24 PM on April 14

Worth pointing out that the tool referenced is for removing the assembler dependency rather than the COBOL dependency. Removing the assembler dependency has some real benefits in that currently all that code has to run on a 360 emulation so getting that into Java would provide some real cost and flexibility benefits.

The benefits of getting away from COBOL are much less clear. It works, it's well tested and while there aren't masses of practitioners anymore it's relatively straightforward to train an experienced dev in COBOL.
posted by southof40 at 7:43 PM on April 14 [3 favorites]

relatively straightforward to train an experienced dev in COBOL.

There's even a foosball table in the break room!
posted by thelonius at 5:38 AM on April 15 [4 favorites]

Removing the assembler dependency has some real benefits in that currently all that code has to run on a 360 emulation so getting that into Java would provide some real cost and flexibility benefits.

Wouldn't that be swapping one VM for another, though? It would also mean that the new assembler→java code wouldn't run on a 360 emulation, so you'd have to rewrite all the COBOL in java too. Then test it. And pay for all that, somehow.
posted by scruss at 10:25 AM on April 15 [1 favorite]

« Older I'll dance at your quinceanera, wedding. . ....   |   ‘If your coffee needs doctoring, it must be broken... Newer »

You are not currently logged in. Log in or create a new account to post comments.