The daemon that no one comprehends.
November 4, 2021 6:10 PM Subscribe
COBOL: The code that controls your money. "In fact, these days, when the phone rings in the house Thomas retired to — in a small town outside of Toronto — it will occasionally be someone from the bank. Hey, they’ll say, can you, uh, help… update your code? Maybe add some new features to it? Because, as it turns out, the bank no longer employs anyone who understands COBOL as well as Thomas does, who can dive in and tweak it to perform a new task. Nearly all the COBOL veterans, the punch-card jockeys who built the bank’s crucial systems way back when, who know COBOL inside and out — they’ve retired. They’ve left the building, just like Thomas. And few young coders have any interest in learning a dusty, 50-year-old computer language. They’re much more excited by buzzier new fields, like Toronto’s booming artificial intelligence scene. "
"The Bank of New York Mellon in 2012 found it had 112,500 individual COBOL programs, constituting almost 350 million lines; that is probably typical for most big financial institutions. When your boss hands you your paycheck, odds are it was calculated using COBOL. If you invest, your stock trades run on it too. So does health care: Insurance companies in the U.S. use “adjudication engines’” — software that figures out what a doctor or drug company will get paid for a service — which were written in COBOL. "
-----
"At one point, his team wanted to build a way for food stamp recipients to book a meeting with a government official. The old California systems already had a section that would accept a request like that. But in the field where you’d input “when are you free to meet?” the older system only let you type 40 characters — and it wouldn’t let you use hyphens, so you couldn’t use a short form of language, like “M-W,” to show you were free Monday through Wednesday.
What a pain, Guarino thought. So he met with the person who managed that old software system. “Unfortunately, yes, those are real constraints,” the guy told him. And it was a COBOL problem; it had been written decades ago. “So what can you do? Can you make the field bigger or whatever?” Guarino asked. “And he was just like, straight up — no! There’s nothing we can do.” "
---
"Being “stable” and old, though, can create a paradox — a curse of success. Because when code runs nicely without anyone needing to check up on it, eventually people drift away. They stop looking at it, stop inspecting it. And that means they stop understanding how, precisely, it works.
Certainly, they know that it works. Hey, it’s functioning every day, processing millions of transactions in a snap! But nobody quite knows why or how. COBOL has become an inscrutable mystery, a daemon that performs its tasks dutifully, but in a manner no one quite comprehends."
"The Bank of New York Mellon in 2012 found it had 112,500 individual COBOL programs, constituting almost 350 million lines; that is probably typical for most big financial institutions. When your boss hands you your paycheck, odds are it was calculated using COBOL. If you invest, your stock trades run on it too. So does health care: Insurance companies in the U.S. use “adjudication engines’” — software that figures out what a doctor or drug company will get paid for a service — which were written in COBOL. "
-----
"At one point, his team wanted to build a way for food stamp recipients to book a meeting with a government official. The old California systems already had a section that would accept a request like that. But in the field where you’d input “when are you free to meet?” the older system only let you type 40 characters — and it wouldn’t let you use hyphens, so you couldn’t use a short form of language, like “M-W,” to show you were free Monday through Wednesday.
What a pain, Guarino thought. So he met with the person who managed that old software system. “Unfortunately, yes, those are real constraints,” the guy told him. And it was a COBOL problem; it had been written decades ago. “So what can you do? Can you make the field bigger or whatever?” Guarino asked. “And he was just like, straight up — no! There’s nothing we can do.” "
---
"Being “stable” and old, though, can create a paradox — a curse of success. Because when code runs nicely without anyone needing to check up on it, eventually people drift away. They stop looking at it, stop inspecting it. And that means they stop understanding how, precisely, it works.
Certainly, they know that it works. Hey, it’s functioning every day, processing millions of transactions in a snap! But nobody quite knows why or how. COBOL has become an inscrutable mystery, a daemon that performs its tasks dutifully, but in a manner no one quite comprehends."
Re-write all of it in Intercal. Or Whitespace.
posted by Jessica Savitch's Coke Spoon at 6:30 PM on November 4, 2021 [9 favorites]
posted by Jessica Savitch's Coke Spoon at 6:30 PM on November 4, 2021 [9 favorites]
Great post---and it's a fascinating history of a tool. This kind of thing reminds me forcefully of this para from a 2016 article on maintenance and technology:
We can think of labour that goes into maintenance and repair as the work of the maintainers, those individuals whose work keeps ordinary existence going rather than introducing novel things. Brief reflection demonstrates that the vast majority of human labour, from laundry and trash removal to janitorial work and food preparation, is of this type: upkeep...posted by Fiasco da Gama at 6:35 PM on November 4, 2021 [9 favorites]
Why not write an AI to learn how to program in COBOL?
posted by porpoise at 6:41 PM on November 4, 2021 [8 favorites]
posted by porpoise at 6:41 PM on November 4, 2021 [8 favorites]
Is Weird Al a standard measure of time these days?
That's part of the problem with COBOL- all your intervals are defined as Weirds Al, and there's no way to change it. They never thought they'd need more specific timeframes than that.
posted by TheWhiteSkull at 6:41 PM on November 4, 2021 [70 favorites]
That's part of the problem with COBOL- all your intervals are defined as Weirds Al, and there's no way to change it. They never thought they'd need more specific timeframes than that.
posted by TheWhiteSkull at 6:41 PM on November 4, 2021 [70 favorites]
Why not write an AI to learn how to program in COBOL?
That would be a weird AI.
posted by saturday_morning at 6:42 PM on November 4, 2021 [103 favorites]
That would be a weird AI.
posted by saturday_morning at 6:42 PM on November 4, 2021 [103 favorites]
Weird AL-GOL is just Pascal, isn't it?
posted by credulous at 6:53 PM on November 4, 2021 [16 favorites]
posted by credulous at 6:53 PM on November 4, 2021 [16 favorites]
Let me guess... the code works well enough that they've eliminated most of their developer base by attrition. Nobody has tried to hire an entry level developer in 25 years, and it was thirty five years since they did a competitive analysis of salaries. So, instead of maintaining an ecosystem, they'll pay well into $xxx/hr to drag their hoary consultants out of retirement for a few weeks a year to keep the house of cards from collapsing
posted by wotsac at 6:55 PM on November 4, 2021 [43 favorites]
posted by wotsac at 6:55 PM on November 4, 2021 [43 favorites]
> Weird AL-GOL is just Pascal, isn't it?
It's an inconvenient truth.
posted by urbanwhaleshark at 6:56 PM on November 4, 2021 [19 favorites]
It's an inconvenient truth.
posted by urbanwhaleshark at 6:56 PM on November 4, 2021 [19 favorites]
See also "An oral history of bank python"
posted by lalochezia at 7:00 PM on November 4, 2021 [22 favorites]
posted by lalochezia at 7:00 PM on November 4, 2021 [22 favorites]
At my alma mata in the early 90s COBOL was still taught, but not in the computer science labs. There was a building half a mile away that housed a totally separate VAX for the business folks, and was taught by a professor that had no connection to the computer science department. I absolutely hated the language after just a few weeks of working with it.
Not all financial institutions rely on it either, the one I work for has been around since the late 1970s and everything is in C#. The old VAX systems went out the door 15 years ago.
posted by inthe80s at 7:07 PM on November 4, 2021 [3 favorites]
Not all financial institutions rely on it either, the one I work for has been around since the late 1970s and everything is in C#. The old VAX systems went out the door 15 years ago.
posted by inthe80s at 7:07 PM on November 4, 2021 [3 favorites]
> See also "An oral history of bank python"
I really wanted this to be a different story.
posted by urbanwhaleshark at 7:08 PM on November 4, 2021 [29 favorites]
I really wanted this to be a different story.
posted by urbanwhaleshark at 7:08 PM on November 4, 2021 [29 favorites]
So even if you want to change them, it’s too dangerous to try. At the bank Stern worked at, you could lose hair over the stress of tinkering with truly ancient, mission-critical code.This is making me think about animal evolution and development. Even though code is made by thinking humans and DNA is made by blind mutation and selection, you still see some of those same tradeoffs.
“It was a high level of risk to fix things because you could damage something that was already working,” she tells me. So most of the time, instead of intensively rewriting old code, they’d just add small new bits of code, patching things around the edges.
You're on land now and don't need gills? Well, we've got this gill-building routine in here and we don't know what else will break if we get rid of it, so let's maybe let it run for a couple of days and then hack a routine onto the end of it to have it start building part of the ear...
posted by clawsoon at 7:11 PM on November 4, 2021 [34 favorites]
MetaFilter: all your intervals are defined as Weirds Al
posted by ChurchHatesTucker at 7:14 PM on November 4, 2021 [8 favorites]
posted by ChurchHatesTucker at 7:14 PM on November 4, 2021 [8 favorites]
Whoa, lalochezia!
One thing I regret about software as a field is how little time is spent learning from existing systems and judging what they did well, or badly.posted by clew at 7:16 PM on November 4, 2021 [12 favorites]
I thought about learning COBOL for a minute and trying to be part of the next generation of people maintaining these systems. I love doing code archaeology and I was good at my assembly programming classes in university so I thought it might be fun. Buuuut it looked like I'd be making like half what I could get as a generalist coder at a big software company, building a less portable skillset, doing a higher stress job, and getting less respect from my colleagues and coworkers. Why would anyone take such a deal who had other options? If you're going to do such a career limiting move as specializing in a dying language that your economy depends on, it seems to me like you ought to at least get Paid, but clearly these banks hadn't got the memo last time I looked into it.
posted by potrzebie at 7:19 PM on November 4, 2021 [58 favorites]
posted by potrzebie at 7:19 PM on November 4, 2021 [58 favorites]
No it's so much better, the code is probably running on a really high end processor emulating a Mainframe, that does not run the code but emulates a 360/30, no wait, that emulates a 1401. And outputs a 80 column virtual card deck.
posted by sammyo at 7:21 PM on November 4, 2021 [4 favorites]
posted by sammyo at 7:21 PM on November 4, 2021 [4 favorites]
During the y2k crisis I worked for a company called MatriDigm that would take COBOL code, scan it in and extract all the variables and business logic and automatically correct the stored dates by windowing (after the founder tried two other unworkable schemes, widening to four digit years and using a compressed binary year). The idea was to be a general purpose company after y2k that allowed people to alter business logic in the cobol without needing to understand the language. We had at least one ex-IBM fellow on it and a lot of the best and brightest ex-IBM programmers from the boomer generation. For the hard to find COBOL programmers we had to get people back who had moved on to other careers like law, we had to get Russian immigrants who also hadn't worked on COBOL in years, we had two ex-rocket engineers. We had computers in the back room the size of refrigerators with dipswitches on the front to load the initial binary code lines to load the OS from tape. We also had a whole massive server room to run the C++ programs that analyzed everything.
I cannot begin to express how much of a shit show it is in some of these COBOL shops who were clients. In some places the code is solid but basically unmaintained. In others the source code was missing and we had to reverse engineer compiled binary. I can't imagine the situation has improved in 20+ years.
posted by BrotherCaine at 7:22 PM on November 4, 2021 [60 favorites]
I cannot begin to express how much of a shit show it is in some of these COBOL shops who were clients. In some places the code is solid but basically unmaintained. In others the source code was missing and we had to reverse engineer compiled binary. I can't imagine the situation has improved in 20+ years.
posted by BrotherCaine at 7:22 PM on November 4, 2021 [60 favorites]
When I first started working as a coder in the late 90s, the company I was at hired an ex-Soviet COBOL coder because so much of that country's bureaucracy ran on COBOL and ex-pat programmers were able to do a brisk business selling their skills in the run-up to Y2K.
Shortly after Y2K, when I transitioned to a brief period of federal government work, I learned we had two retired coders on staff who were the only people who understood how to properly maintain the UNISYS mainframes that hosted the country's employment insurance programs, and the output line printers that served them.
Ancient tongues have long been the currency of the realm in the IT world, especially for any entity that made an enormous investment in infrastructure way back when that it's not interested in duplicating with an update.
posted by jordantwodelta at 7:24 PM on November 4, 2021 [5 favorites]
Shortly after Y2K, when I transitioned to a brief period of federal government work, I learned we had two retired coders on staff who were the only people who understood how to properly maintain the UNISYS mainframes that hosted the country's employment insurance programs, and the output line printers that served them.
Ancient tongues have long been the currency of the realm in the IT world, especially for any entity that made an enormous investment in infrastructure way back when that it's not interested in duplicating with an update.
posted by jordantwodelta at 7:24 PM on November 4, 2021 [5 favorites]
And few young coders have any interest in learning a dusty, 50-year-old computer language.
Pay me I'll do it 🙋♂️
No seriously, this keeps coming up every few years, but then I look at the job boards each time and employers either want a perfect clone of the old guy that just retired (i.e. decades of experience) or the wage is less than a jr. javascript monkey. I - and many others - DGAF about AI or w/e, but if there is a promise of pay, then just dust off the old manuals leave me in corner for 3-6 months. I'll happily pick up any old legacy thing.
On edit: what potrzebie said
posted by Freelance Demiurge at 7:24 PM on November 4, 2021 [79 favorites]
Pay me I'll do it 🙋♂️
No seriously, this keeps coming up every few years, but then I look at the job boards each time and employers either want a perfect clone of the old guy that just retired (i.e. decades of experience) or the wage is less than a jr. javascript monkey. I - and many others - DGAF about AI or w/e, but if there is a promise of pay, then just dust off the old manuals leave me in corner for 3-6 months. I'll happily pick up any old legacy thing.
On edit: what potrzebie said
posted by Freelance Demiurge at 7:24 PM on November 4, 2021 [79 favorites]
Technical Debt
(PS I myself prefer to measure time in micro-fortnights. OpenVMS represent!)
posted by zaixfeep at 7:33 PM on November 4, 2021 [8 favorites]
(PS I myself prefer to measure time in micro-fortnights. OpenVMS represent!)
posted by zaixfeep at 7:33 PM on November 4, 2021 [8 favorites]
It's even be worse maintaining this stuff than you can imagine: fixes for bugs have, in many cases, been applied by writing in assembly code and then directly patching the binaries (after locating some unused section of program memory in which the patch can fit) rather than recompiling from the original COBOL source (which in many cases no longer exists or is even possible). So you need to understand what sorts of 'surgery' have already been performed by your predecessors during the later paleolithic.
posted by Insert Clever Name Here at 7:37 PM on November 4, 2021 [28 favorites]
posted by Insert Clever Name Here at 7:37 PM on November 4, 2021 [28 favorites]
When I went to college starting in 1989 and majored in programming, I had to learn Pascal, COBOL, Fortran 77, RPG II. I was also supposed to take assembly language but they only offered it at the same time as other courses I needed, and I transferred to 4-year university rather than getting the AA first.
C -- not even C++ -- was an elective. The instructor told us on day one that he wasn't a programmer, wasn't a teacher, but was a "C user." And he barely understood pointers.
Thankfully for my career I'm good at picking things up.
Anyway, our company has some legacy code in Fortran 77. Porting it to C++ is on our "eventually" list. I haven't written a line of Fortran in over 30 years, but it's probably going to land on my plate sometime before that turns into 40 years.
posted by Foosnark at 7:40 PM on November 4, 2021 [7 favorites]
C -- not even C++ -- was an elective. The instructor told us on day one that he wasn't a programmer, wasn't a teacher, but was a "C user." And he barely understood pointers.
Thankfully for my career I'm good at picking things up.
Anyway, our company has some legacy code in Fortran 77. Porting it to C++ is on our "eventually" list. I haven't written a line of Fortran in over 30 years, but it's probably going to land on my plate sometime before that turns into 40 years.
posted by Foosnark at 7:40 PM on November 4, 2021 [7 favorites]
COBOL is a financial transaction language. It has that as a single function. If well written it reads like a newspaper and is readily understandable to almost anyone. It runs the entire economy of the planet. It perhaps deserves a bit more respect...
posted by jim in austin at 7:44 PM on November 4, 2021 [30 favorites]
posted by jim in austin at 7:44 PM on November 4, 2021 [30 favorites]
I keep hearing these anecdotes about "banks" or other regulated/high-rel businesses that have only one or two guys (because, inevitably in the story they are guys) who know anything about Cobol and how the End Is Nigh!
Yeah, no.
Regulated financial sector? The federal regulators are savvy enough to be asking "so you have a pipeline of talent to support this crap, right?", and the answer has to be "yup...look at our legacy, fully trained coders right here". And when I go to my banks software dev centers...LO! WHAT CAN IT BE? There's a crew of Cobol programmers. With budget and new projects. Runbooks for BC/DR scenarios. UNPOSSIBLE, you say. But yes...there they are.
So, yeah, Cobol is still a thing. Totally uncool, not-SillyValley, but pretty much everywhere else.
posted by kjs3 at 7:46 PM on November 4, 2021 [17 favorites]
Yeah, no.
Regulated financial sector? The federal regulators are savvy enough to be asking "so you have a pipeline of talent to support this crap, right?", and the answer has to be "yup...look at our legacy, fully trained coders right here". And when I go to my banks software dev centers...LO! WHAT CAN IT BE? There's a crew of Cobol programmers. With budget and new projects. Runbooks for BC/DR scenarios. UNPOSSIBLE, you say. But yes...there they are.
So, yeah, Cobol is still a thing. Totally uncool, not-SillyValley, but pretty much everywhere else.
posted by kjs3 at 7:46 PM on November 4, 2021 [17 favorites]
Nobody, or at least very few people here have disrespected COBOL. It has its uses. But like every bit of critical infrastructure in the United States, once a year for decades a credulous reporter finds out about how important COBOL is, and writes the same article, without once asking anyone who has spent 15 minutes seriously considering the career change to confirm what the banks and governments say about why they can't hire anyone.
posted by wotsac at 7:51 PM on November 4, 2021 [18 favorites]
posted by wotsac at 7:51 PM on November 4, 2021 [18 favorites]
> if there is a promise of decent pay, just dust off the old manuals leave me in corner for 3-6 months. I'll happily pick up any old legacy thing.
I grew up in Kansas City and DST Systems often advertised they'll hire anyone with a college degree and teach them to write COBOL. They presumably make a business out of charging companies and local governments whose last COBOL programmer retired for annual maintenance contracts, and the occasional feature. But even that company has sold since I left the metro area a decade ago, so IDK if they're still recruiting.
posted by pwnguin at 7:56 PM on November 4, 2021 [2 favorites]
I grew up in Kansas City and DST Systems often advertised they'll hire anyone with a college degree and teach them to write COBOL. They presumably make a business out of charging companies and local governments whose last COBOL programmer retired for annual maintenance contracts, and the occasional feature. But even that company has sold since I left the metro area a decade ago, so IDK if they're still recruiting.
posted by pwnguin at 7:56 PM on November 4, 2021 [2 favorites]
Why not write an AI to learn how to program in COBOL?
That's a hell of a can of worms.
posted by snerson at 8:05 PM on November 4, 2021 [4 favorites]
That's a hell of a can of worms.
posted by snerson at 8:05 PM on November 4, 2021 [4 favorites]
Seven years ago I was a customer of that same company pwnguin mentions and implemented a browser-based system that had a mid-70s COBOL core running on a VM as it’s secret, bulletproof heart. They’d just started rewriting that core in Java. While the COBOL worked great, the Java rewrite would provide exactly one valuable feature: they wouldn’t have to say that their system ran on COBOL, a disclosure that had lost them quite a number of RFPs.
posted by putzface_dickman at 8:15 PM on November 4, 2021 [11 favorites]
posted by putzface_dickman at 8:15 PM on November 4, 2021 [11 favorites]
"Why not write an AI to learn how to program in COBOL?-- posted by porpoise
Well yes, you write the AI program in LISP, which you run in EMACS
posted by symbioid at 8:17 PM on November 4, 2021 [10 favorites]
Well yes, you write the AI program in LISP, which you run in EMACS
posted by symbioid at 8:17 PM on November 4, 2021 [10 favorites]
they'll pay well into $xxx/hr to drag their hoary consultants out of retirement for a few weeks a year to keep the house of cards from collapsing
Holy shit, they're running on Hollerith cards? Not that I'm surprised, mind you.
posted by Halloween Jack at 8:18 PM on November 4, 2021 [2 favorites]
Holy shit, they're running on Hollerith cards? Not that I'm surprised, mind you.
posted by Halloween Jack at 8:18 PM on November 4, 2021 [2 favorites]
COBOL's a little bit too mainstream of a business language for my taste. Now PL/B, there's a real connoisseur's business programming language.
posted by Pyry at 8:21 PM on November 4, 2021 [3 favorites]
posted by Pyry at 8:21 PM on November 4, 2021 [3 favorites]
COBOL is a financial transaction language
Even as a lay person, I bet it's elegant, but can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
posted by Reasonably Everything Happens at 8:59 PM on November 4, 2021 [1 favorite]
Even as a lay person, I bet it's elegant, but can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
posted by Reasonably Everything Happens at 8:59 PM on November 4, 2021 [1 favorite]
can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
Because that costs more this quarter, and they got bonus metrics to hit. Same as last quarter, same as next quarter.
posted by tclark at 9:10 PM on November 4, 2021 [42 favorites]
Because that costs more this quarter, and they got bonus metrics to hit. Same as last quarter, same as next quarter.
posted by tclark at 9:10 PM on November 4, 2021 [42 favorites]
can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
Think of it as a physical plant. You've got a factory that was built up over fifty years. The important machines all run on compressed air. But it's getting harder to find people to maintain the compressors, lay new air hoses, etc. How do you keep the factory turning out widgets at the same pace while replacing all the machines with electric ones?
posted by ChurchHatesTucker at 9:11 PM on November 4, 2021 [14 favorites]
Think of it as a physical plant. You've got a factory that was built up over fifty years. The important machines all run on compressed air. But it's getting harder to find people to maintain the compressors, lay new air hoses, etc. How do you keep the factory turning out widgets at the same pace while replacing all the machines with electric ones?
posted by ChurchHatesTucker at 9:11 PM on November 4, 2021 [14 favorites]
Not all financial institutions rely on it either, the one I work for has been around since the late 1970s and everything is in C#. The old VAX systems went out the door 15 years ago.
One of my previous banks used RPG. I knew they were running on an AS/400, but didn't learn until one day someone happened to have a library listing open for some reason or another what language they were using. One of the neat things about OS/400 is that it makes it easy to use DB2 instead of emulated punch cards and write different parts of your software in whatever language is most appropriate since it's super easy to make calls between languages.
I learned some RPG for a job, but mainly just enough to figure out what a program was doing so I could diagnose issues, make small changes, and parse data files so I could exchange data with other systems. I wasn't paid enough or given the time to learn it well enough to write programs from scratch. I'd certainly be happy to learn more RPG or any COBOL if someone wanted to pay me a good salary to do so. There's not much point in doing it on my own, though, since getting hired as a consultant would require experience on a working code base.
posted by wierdo at 9:15 PM on November 4, 2021 [5 favorites]
One of my previous banks used RPG. I knew they were running on an AS/400, but didn't learn until one day someone happened to have a library listing open for some reason or another what language they were using. One of the neat things about OS/400 is that it makes it easy to use DB2 instead of emulated punch cards and write different parts of your software in whatever language is most appropriate since it's super easy to make calls between languages.
I learned some RPG for a job, but mainly just enough to figure out what a program was doing so I could diagnose issues, make small changes, and parse data files so I could exchange data with other systems. I wasn't paid enough or given the time to learn it well enough to write programs from scratch. I'd certainly be happy to learn more RPG or any COBOL if someone wanted to pay me a good salary to do so. There's not much point in doing it on my own, though, since getting hired as a consultant would require experience on a working code base.
posted by wierdo at 9:15 PM on November 4, 2021 [5 favorites]
My first programming job was at the second largest bank in California, now merged into oblivion. About half the programming staff was female, and almost everyone was excellent. My manager kept program crash dumps (large piles of paper) in one corner of her office so that, when she had a moment, she could investigate and see if she could find common clues. After stacking 25 or so crash dumps over a couple of years, she analyzed, tracked down, and fixed a rare bug which caused system restarts.
The article is correct about salaries. Banks just aren't set up to pay anybody but the highest executives a reasonable salary. After my first year at the bank I was promoted to a "bank officer" which usually was only awarded to branch managers after 20 years of service. But that was the only salary range this bank had for paying someone as much as a computer programmer makes.
posted by blob at 9:26 PM on November 4, 2021 [22 favorites]
The article is correct about salaries. Banks just aren't set up to pay anybody but the highest executives a reasonable salary. After my first year at the bank I was promoted to a "bank officer" which usually was only awarded to branch managers after 20 years of service. But that was the only salary range this bank had for paying someone as much as a computer programmer makes.
posted by blob at 9:26 PM on November 4, 2021 [22 favorites]
can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
It can be shockingly difficult to figure out exactly what a complex software system is trying to do even when you have the source code and people who worked on it
And that’s aside from the scope creep that happens whenever you’re replacing something
posted by flaterik at 9:33 PM on November 4, 2021 [20 favorites]
It can be shockingly difficult to figure out exactly what a complex software system is trying to do even when you have the source code and people who worked on it
And that’s aside from the scope creep that happens whenever you’re replacing something
posted by flaterik at 9:33 PM on November 4, 2021 [20 favorites]
how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
The existing software works, and is probably well-understood at least at the edges, and is surrounded by a whole mass of other systems that have been built around its exact quirks and behavior. It's not something you can rewrite and then drop in over a weekend.
When I was consulting I did a fair bit of work for banks and (more commonly) credit unions. To say that they are usually very protective of the "core system" is an understatement. I mean, that system doesn't just facilitate the bank, it is the bank. It's the no-shit authoritative system that keeps track of how much money is in each account. If it stops working... I guess you can maybe queue up transactions in the various accessory systems for a while (maybe until end-of-day), but in short order you're very seriously screwed as a business.
There are turnkey / "shrink wrapped" software products for bank core systems now, though. But switching to one is not trivial. It ends up being a bit like doing an engine swap on a car while it's running. And in motion. On a highway. In traffic.
One place I visited had a project going to replace their core system. They were in the requirements-gathering phase, and proudly told me that they estimated they were 80% complete on having all the requirements for the replacement system. It had been going on for 5 years.
posted by Kadin2048 at 9:39 PM on November 4, 2021 [32 favorites]
The existing software works, and is probably well-understood at least at the edges, and is surrounded by a whole mass of other systems that have been built around its exact quirks and behavior. It's not something you can rewrite and then drop in over a weekend.
When I was consulting I did a fair bit of work for banks and (more commonly) credit unions. To say that they are usually very protective of the "core system" is an understatement. I mean, that system doesn't just facilitate the bank, it is the bank. It's the no-shit authoritative system that keeps track of how much money is in each account. If it stops working... I guess you can maybe queue up transactions in the various accessory systems for a while (maybe until end-of-day), but in short order you're very seriously screwed as a business.
There are turnkey / "shrink wrapped" software products for bank core systems now, though. But switching to one is not trivial. It ends up being a bit like doing an engine swap on a car while it's running. And in motion. On a highway. In traffic.
One place I visited had a project going to replace their core system. They were in the requirements-gathering phase, and proudly told me that they estimated they were 80% complete on having all the requirements for the replacement system. It had been going on for 5 years.
posted by Kadin2048 at 9:39 PM on November 4, 2021 [32 favorites]
If it's a Turing-complete language (it is) it can be transpiled into another such language. Which can be maintained.
If you have good integration tests.
What's an integration test you say? Ah. Good day then.
I SAID GOOD DAY.
posted by abulafa at 9:40 PM on November 4, 2021 [31 favorites]
If you have good integration tests.
What's an integration test you say? Ah. Good day then.
I SAID GOOD DAY.
posted by abulafa at 9:40 PM on November 4, 2021 [31 favorites]
As a well-seasoned handler of toxic substances (Perl, to be specific) I have, in fact, considered what it would take out of me to become proficient in a different unloved language just to buy the job security. And the answer is that it would take too much out of me for what they pay. You'd think they'd be willing to pay more since money literally doesn't move without COBOL and what is a bank that can't move money, but as other people have pointed out above the banks mostly haven't figured out (or are avoiding having to acknowledge) that it's maybe worth paying actual Senior Programmer salaries instead of just matching what they pay the secretaries who've been there the longest. If I'm going to be underpaid I might as well enjoy the work.
posted by fedward at 9:41 PM on November 4, 2021 [10 favorites]
posted by fedward at 9:41 PM on November 4, 2021 [10 favorites]
can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
It can be shockingly difficult to figure out exactly what a complex software system is trying to do even when you have the source code and people who worked on it
And that’s aside from the scope creep that happens whenever you’re replacing something
Several years ago, the company I work for transitioned between inventory/ordering systems. They were using DOS-based systems well into the 2010s, and they finally made the leap into a new thing with more modern technology about 5 years ago.
And it has never worked right. The UI is entirely opaque, and requires humans to take numbers from one screen and put them manually into another screen for input to work correctly in some instances. We get paperwork printed out that we have to manually sort into numerical order because the system "won't" do it. [Ed.Note - what do computers do really well? sort numbers!]
I could go on about a lot of the stupidness of this system, but it's basically a giant UI fuck-up, where the DOS system was simple enough that even new hires learned it in a few short lessons.
This is a company which has resources to spend on a good transition, but it was not, not at all. All these years in, the database is still incomplete and they have resource and humanpower wasting practices perpetuated by the humans being the slaves to the machines, and not vice versa.
If you think an entrenched global economic engine that is running on generations-old code can manage a smooth transition to something new, let this little microcosm be an example.
posted by hippybear at 9:46 PM on November 4, 2021 [21 favorites]
It can be shockingly difficult to figure out exactly what a complex software system is trying to do even when you have the source code and people who worked on it
And that’s aside from the scope creep that happens whenever you’re replacing something
Several years ago, the company I work for transitioned between inventory/ordering systems. They were using DOS-based systems well into the 2010s, and they finally made the leap into a new thing with more modern technology about 5 years ago.
And it has never worked right. The UI is entirely opaque, and requires humans to take numbers from one screen and put them manually into another screen for input to work correctly in some instances. We get paperwork printed out that we have to manually sort into numerical order because the system "won't" do it. [Ed.Note - what do computers do really well? sort numbers!]
I could go on about a lot of the stupidness of this system, but it's basically a giant UI fuck-up, where the DOS system was simple enough that even new hires learned it in a few short lessons.
This is a company which has resources to spend on a good transition, but it was not, not at all. All these years in, the database is still incomplete and they have resource and humanpower wasting practices perpetuated by the humans being the slaves to the machines, and not vice versa.
If you think an entrenched global economic engine that is running on generations-old code can manage a smooth transition to something new, let this little microcosm be an example.
posted by hippybear at 9:46 PM on November 4, 2021 [21 favorites]
Pretty funny to have sport of PASCAL and COBOL in the same comment section. I once worked on an OS/VS COBOL interactive software engineering workstation. With the compiler written in PASCAL. COBOL programs were run on a virtual machine with an IDE front end on a bitmapped display. It was a year without a summer.
You can get something like that today, from Microfocus. It's based on Visual Studio 2019 (or Eclipse: shudder): https://www.microfocus.com/en-us/products/visual-cobol/overview
They've been continuously in the game since the early 1980s.
posted by the Real Dan at 10:20 PM on November 4, 2021 [5 favorites]
You can get something like that today, from Microfocus. It's based on Visual Studio 2019 (or Eclipse: shudder): https://www.microfocus.com/en-us/products/visual-cobol/overview
They've been continuously in the game since the early 1980s.
posted by the Real Dan at 10:20 PM on November 4, 2021 [5 favorites]
As I understand it, AI does modeling and statistics. It's good at finding good enough solutions to things like fooling human perception and approximations. You can't approximate financial transactions, so you can't approximate the code that performs them.
Anyway that's the answer an AI trained on the data in my uninformed brain would give.
posted by klanawa at 10:21 PM on November 4, 2021 [4 favorites]
Anyway that's the answer an AI trained on the data in my uninformed brain would give.
posted by klanawa at 10:21 PM on November 4, 2021 [4 favorites]
You could have software translate the source code into another language and verify it with unit tests and running both systems in parallel for a long enough time that you were certain they produced the same results, but then you'd be left with a new system where modifications are just as hard and reading the source is like trying to understand minified javascript.
TBH, I'm not sure the banks have the wrong idea. I've seen far too many companies spend far too much money trying to migrate systems and end up having nothing to show for it because it turned out the new system simply didn't work despite years of planning and a literal warehouse full of requirements documents. It's certainly possible to make it happen, often at much lower expense than the failed projects I've seen, but it's still not clear it's actually the best use of resources. It may be better to simply keep a pipeline of talent who can work on the old systems, slowly disentangle the presentation from the data, and stop worrying about things being old.
posted by wierdo at 11:12 PM on November 4, 2021 [13 favorites]
TBH, I'm not sure the banks have the wrong idea. I've seen far too many companies spend far too much money trying to migrate systems and end up having nothing to show for it because it turned out the new system simply didn't work despite years of planning and a literal warehouse full of requirements documents. It's certainly possible to make it happen, often at much lower expense than the failed projects I've seen, but it's still not clear it's actually the best use of resources. It may be better to simply keep a pipeline of talent who can work on the old systems, slowly disentangle the presentation from the data, and stop worrying about things being old.
posted by wierdo at 11:12 PM on November 4, 2021 [13 favorites]
Not to, um, reiterate all of the foregoing but seriously: if it actually mattered, if people in nice suits needed to care, it’d be paying $1M/yr.
…since does not, we can deduce that this expertise does not, in fact, matter. We might like to pretend it matters, our PR people would like us to wring our hands in public about it, but it does not genuinely matter. Or, at minimum, it does not matter to anyone that matters.
…and if you are prepared to pay $1M, I know a JCL guy you can hire, for only a $750K finders fee.
Not joking. $750K fee, give me a call.
If it actually matters, of course.
posted by aramaic at 11:16 PM on November 4, 2021 [21 favorites]
…since does not, we can deduce that this expertise does not, in fact, matter. We might like to pretend it matters, our PR people would like us to wring our hands in public about it, but it does not genuinely matter. Or, at minimum, it does not matter to anyone that matters.
…and if you are prepared to pay $1M, I know a JCL guy you can hire, for only a $750K finders fee.
Not joking. $750K fee, give me a call.
If it actually matters, of course.
posted by aramaic at 11:16 PM on November 4, 2021 [21 favorites]
Even the experts who are retiring don't get paid much compared to other programmers. My father retired in his mid 70s and they kept begging him to come back after 40 years of bank COBOL. He was making more than the vast majority of people in the field, but total compensation was still less than an entry level programmer at a FAANG would make.
COBOL is not some inscrutable thing, anyone who can program could learn it. But banks don't value it enough to actually hire good programmers.
posted by thefoxgod at 11:19 PM on November 4, 2021 [27 favorites]
COBOL is not some inscrutable thing, anyone who can program could learn it. But banks don't value it enough to actually hire good programmers.
posted by thefoxgod at 11:19 PM on November 4, 2021 [27 favorites]
I was tasked with data mining an IDMS system which used screen masks and innumerable comma-delimited text files to harvest and standardize all the data into an Oracle database. The system had been around for so long the core developers had actually all died. The remainder group were long in the tooth and spent their days writing new screen masks against the many files. No standards, no documentation (with a couple of cryptic exceptions), and everyone was in job protection mode. The system had not so much developed, it had mutated. Being in Belgium, there were 'lookup' tables in both French and Flemish (replicate this basic issue multiple times). Two seemingly identical masks with, for example, Name/Telephone/Account Number would have data structures which LOOKED the same but one would have field lengths of 25/12/22 while the other would be 23/12/18. Any 'blank' space was terminated using Hexa-decimal 20 which many will know in the ASCII character set is a non-printable space.
Screen masks were written on the fly to meet with a specific need from a department of the company but were not codified (i.e. screen 00001, screen 00002... screen 10004 etc) or properly documented - I found a line . Over the decades these screens sat there performing essential functions but no-one knew how or why or even where these were. Seeing the writing on the wall for their 'baby' I received zero support so had to go in cold at night time to gain access which they would deliberately prevent during the day with randomized file locking and server reboots if they knew I was working on it. The money I was paid was too good to not do it but I would rather stab my eyes out with a screwdriver than ever have to deal with something like that ever again. Incidentally, I discovered that only 5% of the 10's of thousands of screen masks and hundreds of text files were actually used or needed.
I will not go down that dark wormhole of attacking or defending various development methodologies and 'flavor of the month' software/platforms. The key metric of any application is does it work and does it do the job now and in the future can it be adapted for changing needs. Instead, I will say this... how many of you are end users of systems that you have to use, that you were not consulted on, that do not do the job for which they were intended? Sales people are good at selling concepts and that their software is 'the best' at doing a specific series of tasks but often fail to see the true needs of a company and its end users now and in the future.
posted by IndelibleUnderpants at 11:21 PM on November 4, 2021 [15 favorites]
Screen masks were written on the fly to meet with a specific need from a department of the company but were not codified (i.e. screen 00001, screen 00002... screen 10004 etc) or properly documented - I found a line . Over the decades these screens sat there performing essential functions but no-one knew how or why or even where these were. Seeing the writing on the wall for their 'baby' I received zero support so had to go in cold at night time to gain access which they would deliberately prevent during the day with randomized file locking and server reboots if they knew I was working on it. The money I was paid was too good to not do it but I would rather stab my eyes out with a screwdriver than ever have to deal with something like that ever again. Incidentally, I discovered that only 5% of the 10's of thousands of screen masks and hundreds of text files were actually used or needed.
I will not go down that dark wormhole of attacking or defending various development methodologies and 'flavor of the month' software/platforms. The key metric of any application is does it work and does it do the job now and in the future can it be adapted for changing needs. Instead, I will say this... how many of you are end users of systems that you have to use, that you were not consulted on, that do not do the job for which they were intended? Sales people are good at selling concepts and that their software is 'the best' at doing a specific series of tasks but often fail to see the true needs of a company and its end users now and in the future.
posted by IndelibleUnderpants at 11:21 PM on November 4, 2021 [15 favorites]
If it's a Turing-complete language (it is) it can be transpiled into another such language. Which can be maintained.
This idea contains a tacit assumption that the maintainability of any given codebase depends on the language it's written it to an extent that would make the transpilation exercise worthwhile.
It absolutely does not.
Maintainability depends first and foremost on not needing to cold-analyze a codebase in order to be able to specify its correct behaviour.
Institutions that can't do this have far bigger problems than the specific languages their legacy codebases happen to have been implemented in.
There's an awful lot of technical debt resting on the assumption that the source code is the spec.
posted by flabdablet at 11:31 PM on November 4, 2021 [12 favorites]
This idea contains a tacit assumption that the maintainability of any given codebase depends on the language it's written it to an extent that would make the transpilation exercise worthwhile.
It absolutely does not.
Maintainability depends first and foremost on not needing to cold-analyze a codebase in order to be able to specify its correct behaviour.
Institutions that can't do this have far bigger problems than the specific languages their legacy codebases happen to have been implemented in.
There's an awful lot of technical debt resting on the assumption that the source code is the spec.
posted by flabdablet at 11:31 PM on November 4, 2021 [12 favorites]
A clip to give a good idea of how odious Cobol is to anyone here who isn't familiar with it
Back in university I would have gotten a double-major in physics and comp sci, but to get the comp sci degree you had to take this second-year course on business computing that used Cobol. As you would expect, the course was given by the head of the department. I settled for the single major (having all the requirements for the second major except for CS 201), and found myself surrounded by other physics grads once I started working.
posted by morspin at 12:15 AM on November 5, 2021 [3 favorites]
Back in university I would have gotten a double-major in physics and comp sci, but to get the comp sci degree you had to take this second-year course on business computing that used Cobol. As you would expect, the course was given by the head of the department. I settled for the single major (having all the requirements for the second major except for CS 201), and found myself surrounded by other physics grads once I started working.
posted by morspin at 12:15 AM on November 5, 2021 [3 favorites]
can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
If it's not mission critical maybe, but I've heard stories of billion dollar companies going under after trying to do even a small software replacement. Pay three redundant teams to do the job and pray you get one working system out of it.
posted by BrotherCaine at 12:28 AM on November 5, 2021 [4 favorites]
If it's not mission critical maybe, but I've heard stories of billion dollar companies going under after trying to do even a small software replacement. Pay three redundant teams to do the job and pray you get one working system out of it.
posted by BrotherCaine at 12:28 AM on November 5, 2021 [4 favorites]
can anyone explain how it's not just more cost effective for these very large institutions to cut bait and start with a clean slate?
The short answer is "it's more expensive than this quarter's budget and nobody wants to be on the hook for that" and the long answer starts with "it's really complicated, and they can't stop using the old system while they're building the new one so they'd be running two complete sets of data at some point, and..."
...and that's expensive, of course. Outside of this quarter's budget. Outside of this year's budget. And how much money does it actually save in the long run?
None! If they can keep finding That One Guy who knows COBOL when there's a problem. So every time someone says "hey we really, really, really need to upgrade," someone higher up on the money chain says, "oh yeah? What if COBOL becomes the new retro-trend next year and suddenly the world is drowning in COBOL programmers? All that money would be wasted!" and the proposal is shelved again.
And I suspect that's going to happen until they literally can't get hardware that will run COBOL.
posted by ErisLordFreedom at 12:37 AM on November 5, 2021 [11 favorites]
The short answer is "it's more expensive than this quarter's budget and nobody wants to be on the hook for that" and the long answer starts with "it's really complicated, and they can't stop using the old system while they're building the new one so they'd be running two complete sets of data at some point, and..."
...and that's expensive, of course. Outside of this quarter's budget. Outside of this year's budget. And how much money does it actually save in the long run?
None! If they can keep finding That One Guy who knows COBOL when there's a problem. So every time someone says "hey we really, really, really need to upgrade," someone higher up on the money chain says, "oh yeah? What if COBOL becomes the new retro-trend next year and suddenly the world is drowning in COBOL programmers? All that money would be wasted!" and the proposal is shelved again.
And I suspect that's going to happen until they literally can't get hardware that will run COBOL.
posted by ErisLordFreedom at 12:37 AM on November 5, 2021 [11 favorites]
I am reminded of the anecdote I read somewhere once about airline booking systems, many of which run on systems whose underlying code was written in the sixties. Apparently they are full of comments from programmers, mostly long-since retired or dead, saying "This is a hack, I'll fix it later."
I also once worked with a guy who had built a new web interface that sat on top of a 90s UI that itself sat on top of a green screen CRT system that sat on assembly code. This whole system ran the traffic light system for a major metropolitan city in the UK. He told me that when they were working on improving the web UI, they kept running into weird conditions where little mini gridlocks were springing up all over the city and they couldn't understand why the underlying code wasn't taking this into account and adjusting timings.
Turned out, once they'd blown the dust off a few manuals, that the model that the traffic light system used didn't take account of horizontal space. It just vertically stacked counted cars at intersections. So it didn't realise that there were enough cars waiting at junction A to begin affecting junction B. When the system was first written, the idea that there would ever be enough cars on the road to justify doing that kind of computation had seemed ludicrous, so they never did it. My colleague spent several years of his life fixing that.
posted by Happy Dave at 12:49 AM on November 5, 2021 [21 favorites]
I also once worked with a guy who had built a new web interface that sat on top of a 90s UI that itself sat on top of a green screen CRT system that sat on assembly code. This whole system ran the traffic light system for a major metropolitan city in the UK. He told me that when they were working on improving the web UI, they kept running into weird conditions where little mini gridlocks were springing up all over the city and they couldn't understand why the underlying code wasn't taking this into account and adjusting timings.
Turned out, once they'd blown the dust off a few manuals, that the model that the traffic light system used didn't take account of horizontal space. It just vertically stacked counted cars at intersections. So it didn't realise that there were enough cars waiting at junction A to begin affecting junction B. When the system was first written, the idea that there would ever be enough cars on the road to justify doing that kind of computation had seemed ludicrous, so they never did it. My colleague spent several years of his life fixing that.
posted by Happy Dave at 12:49 AM on November 5, 2021 [21 favorites]
And I suspect that's going to happen until they literally can't get hardware that will run COBOL.
I believe we are past that point already - a lot of these dinosaur systems run on emulators and virtual machines.
posted by each day we work at 12:51 AM on November 5, 2021 [8 favorites]
I believe we are past that point already - a lot of these dinosaur systems run on emulators and virtual machines.
posted by each day we work at 12:51 AM on November 5, 2021 [8 favorites]
Open it up in VS Code and let Github Copilot figure it out — some manager soon
posted by interogative mood at 1:06 AM on November 5, 2021 [1 favorite]
posted by interogative mood at 1:06 AM on November 5, 2021 [1 favorite]
I believe we are past that point already - a lot of these dinosaur systems run on emulators and virtual machines.
IBM says hello. There's a reason their systems have been compiling to bytecode since before I was born. It's precisely so that the exact same shit you have been running since time immemorial will run on a POWER9 machine you buy today, even if the source has been lost or you never had it in the first place.
I mean you could call it emulation, but that's an odd choice of language when it's literally built in to the architecture and the entire software stack and has been for forty years.
posted by wierdo at 1:32 AM on November 5, 2021 [18 favorites]
IBM says hello. There's a reason their systems have been compiling to bytecode since before I was born. It's precisely so that the exact same shit you have been running since time immemorial will run on a POWER9 machine you buy today, even if the source has been lost or you never had it in the first place.
I mean you could call it emulation, but that's an odd choice of language when it's literally built in to the architecture and the entire software stack and has been for forty years.
posted by wierdo at 1:32 AM on November 5, 2021 [18 favorites]
GnuCOBOL
GnuCOBOL (formerly OpenCOBOL) is a free, modern COBOL compiler. GnuCOBOL implements a substantial part of the COBOL 85, COBOL 2002 and COBOL 2014 standards and X/Open COBOL, as well as many extensions included in other COBOL compilers (IBM COBOL, MicroFocus COBOL, ACUCOBOL-GT and others).
GnuCOBOL translates COBOL into C and compiles the translated code using a native C compiler.
Build COBOL programs on various platforms, including GNU/Linux, Unix, Mac OS X, and Microsoft Windows. GnuCOBOL has also been built on HP/UX, z/OS, SPARC, RS6000, AS/400, along with other combinations of machines and operating systems.
Copyright 2001-2020 Free Software Foundation, Inc.
You can download this for free, install it on your computer, and learn as much COBOL as you want. The world will be your oyster (or any mollusc you choose).
posted by Metacircular at 1:33 AM on November 5, 2021 [6 favorites]
GnuCOBOL (formerly OpenCOBOL) is a free, modern COBOL compiler. GnuCOBOL implements a substantial part of the COBOL 85, COBOL 2002 and COBOL 2014 standards and X/Open COBOL, as well as many extensions included in other COBOL compilers (IBM COBOL, MicroFocus COBOL, ACUCOBOL-GT and others).
GnuCOBOL translates COBOL into C and compiles the translated code using a native C compiler.
Build COBOL programs on various platforms, including GNU/Linux, Unix, Mac OS X, and Microsoft Windows. GnuCOBOL has also been built on HP/UX, z/OS, SPARC, RS6000, AS/400, along with other combinations of machines and operating systems.
Copyright 2001-2020 Free Software Foundation, Inc.
You can download this for free, install it on your computer, and learn as much COBOL as you want. The world will be your oyster (or any mollusc you choose).
posted by Metacircular at 1:33 AM on November 5, 2021 [6 favorites]
here's a fun 2018 press release about modernising a system including a million lines of COBOL: https://aws.amazon.com/blogs/apn/automated-refactoring-of-a-u-s-air-force-mainframe-to-aws/
> The system we modernized is an ACAT III Major Defense Acquisition Program and mission-critical system used by more than 18,000 users at over 260 global locations. It provides daily supply chain and equipment support for USAF missions and is the accountable data system for over $30 billion in inventory.
Figure 1 is quietly amusing.
posted by are-coral-made at 1:33 AM on November 5, 2021 [4 favorites]
> The system we modernized is an ACAT III Major Defense Acquisition Program and mission-critical system used by more than 18,000 users at over 260 global locations. It provides daily supply chain and equipment support for USAF missions and is the accountable data system for over $30 billion in inventory.
Figure 1 is quietly amusing.
posted by are-coral-made at 1:33 AM on November 5, 2021 [4 favorites]
COVID presumably means that the future of no longer being able to get retired people to work on old systems is coming a bit faster than was expected.
posted by Nancy Lebovitz at 2:07 AM on November 5, 2021 [3 favorites]
posted by Nancy Lebovitz at 2:07 AM on November 5, 2021 [3 favorites]
Why no AI can do it? Y U NO blank page REDO FROM START?
The metaphor: it's not just learning to speak French or starting over a new life in France (without knowing how to function there), it's being indistinguishable from people who've lived whole lives there.
posted by k3ninho at 2:27 AM on November 5, 2021 [4 favorites]
The metaphor: it's not just learning to speak French or starting over a new life in France (without knowing how to function there), it's being indistinguishable from people who've lived whole lives there.
posted by k3ninho at 2:27 AM on November 5, 2021 [4 favorites]
As I understand it, there's a lot of silly money floating around, and part of what's wrong is that it doesn't just want returns, it wants *fast* returns. Possibly even fast returns that sound *interesting*.
My fantasy project would be spend the money to train the next generation of COBOL programmers then hold the banks for ransom.
posted by Nancy Lebovitz at 2:41 AM on November 5, 2021 [7 favorites]
My fantasy project would be spend the money to train the next generation of COBOL programmers then hold the banks for ransom.
posted by Nancy Lebovitz at 2:41 AM on November 5, 2021 [7 favorites]
We’re experiencing supply chains crash because of reasons that seem to echo a lot of what’s going on here. People retiring. Companies not investing in young hires. Countries shutting borders to foreign workers. Worsening working conditions because “AI will soon do this job anyway!”. Old factories producing essential products shutting down. Etc.
Business leaders have been praising each other for lean just-in-time production and cost cutting for years. Which works …until it doesn’t.
So what happens when there is no Thomas to pull out of retirement to write a patch or a new feature? The article mentions how software written in COBOL software is reliable and that it’s often other pieces of software that fail. Well, the web interface can be fixed. There’s someone there who can do that.
It’s not like Critical Transaction Company A can fail for a few weeks while Other Finance Company B takes over the load. Company B may have invested in modern infrastructure and people, but if it relies on Company A to run flawlessly, what then?
Could this problem be much, much bigger than “every few years there’s an article on COBOL”?
posted by romanb at 2:51 AM on November 5, 2021 [9 favorites]
Business leaders have been praising each other for lean just-in-time production and cost cutting for years. Which works …until it doesn’t.
So what happens when there is no Thomas to pull out of retirement to write a patch or a new feature? The article mentions how software written in COBOL software is reliable and that it’s often other pieces of software that fail. Well, the web interface can be fixed. There’s someone there who can do that.
It’s not like Critical Transaction Company A can fail for a few weeks while Other Finance Company B takes over the load. Company B may have invested in modern infrastructure and people, but if it relies on Company A to run flawlessly, what then?
Could this problem be much, much bigger than “every few years there’s an article on COBOL”?
posted by romanb at 2:51 AM on November 5, 2021 [9 favorites]
"COBOL is a coding language older than Weird Al Yankovic."
That's wrong. Baby Alfred was born on October 29th, 1959 and presumably conceived eight-nine months or so earlier. COBOL was conceived in September of 1959, but the first draft wasn't completed until November 13th, 1959.
posted by Kattullus at 3:02 AM on November 5, 2021 [19 favorites]
That's wrong. Baby Alfred was born on October 29th, 1959 and presumably conceived eight-nine months or so earlier. COBOL was conceived in September of 1959, but the first draft wasn't completed until November 13th, 1959.
posted by Kattullus at 3:02 AM on November 5, 2021 [19 favorites]
It feels like many poeple are failing to understand that the cobol problem is not a language problem, but a platform problem. The language is extremely simple, designed for non technical people to be able to contribute code; anyone can pick it up.
For a decade I worked at migrating cobol applications to open systems; we migrated cobol code to cobol code and they still were very complex projects, that took the whole customer organization to work part time on it for a year.
The thing is that the parts of mainframe platforms are extremely coupled, so you can't just pick an open alternative for each part and run it. Possibly not even 10% of the input/output streams passed the first parallel tests. So then you have to find the big compatibility issues like non-standard sql behaviour of db2, remapped data structures where big endian vs. little endian is an issue or tune hundreds of compiler flags to bring the behaviour of the migrated code to that of the shop you're working with; and then you go down to the nuanced issues, like binary manipulation, implicit interactions, etc. At that point you get a system that is up to 10 times slower than the original one, so you have to customize it to take advantage of the new platform to bring it up to speed and the more, with a final product that's faster and extremely less costly to maintain.
And yet, you still have cobol applications, still need cobol developers. There's no rewriting cobol, at least not in big organizations. New applications are written for the web and over time the relative size of the core banking applications will be smaller and smaller, until the whole mainframe platform is phased out.
posted by valdesm at 3:10 AM on November 5, 2021 [18 favorites]
For a decade I worked at migrating cobol applications to open systems; we migrated cobol code to cobol code and they still were very complex projects, that took the whole customer organization to work part time on it for a year.
The thing is that the parts of mainframe platforms are extremely coupled, so you can't just pick an open alternative for each part and run it. Possibly not even 10% of the input/output streams passed the first parallel tests. So then you have to find the big compatibility issues like non-standard sql behaviour of db2, remapped data structures where big endian vs. little endian is an issue or tune hundreds of compiler flags to bring the behaviour of the migrated code to that of the shop you're working with; and then you go down to the nuanced issues, like binary manipulation, implicit interactions, etc. At that point you get a system that is up to 10 times slower than the original one, so you have to customize it to take advantage of the new platform to bring it up to speed and the more, with a final product that's faster and extremely less costly to maintain.
And yet, you still have cobol applications, still need cobol developers. There's no rewriting cobol, at least not in big organizations. New applications are written for the web and over time the relative size of the core banking applications will be smaller and smaller, until the whole mainframe platform is phased out.
posted by valdesm at 3:10 AM on November 5, 2021 [18 favorites]
Exactly that. Familiarity with the language itself is the least part of what makes old COBOL warhorses valuable; their stock in trade is familiarity with the systems implemented in that language.
posted by flabdablet at 3:12 AM on November 5, 2021 [12 favorites]
posted by flabdablet at 3:12 AM on November 5, 2021 [12 favorites]
"Baby Alfred was born on October 29th, 1959"
The iron law of pedantry is that a pedantic correction will inevitably contain an error.
Having actually bothered to look at his Wikipedia page, "Weird Al" Yankovic was born on October 23rd, 1959. Still, my point holds.
posted by Kattullus at 3:21 AM on November 5, 2021 [7 favorites]
The iron law of pedantry is that a pedantic correction will inevitably contain an error.
Having actually bothered to look at his Wikipedia page, "Weird Al" Yankovic was born on October 23rd, 1959. Still, my point holds.
posted by Kattullus at 3:21 AM on November 5, 2021 [7 favorites]
CoBOL -- CoMMON BUSINESS ORIENTED LANGUAGE
You can credit or blame Grace Hopper for it's existence; she was a computer scientist in the US Navy, not sure her rank when she was given this responsibility but she was a rear Admiral by the time she retired.
I learned it in 1989, did an internship at Pennzoil downtown Houston, they were straight-up with me, it was an internship, just a bit of blather to put on a resume. I did not have any kind of degree -- any outfit would have hired me if I'd had even a degree in basket weaving and they'd have taught me to code; I already knew how to code (I was no ace but I had the rudiments) but no matter. Hard truth -- any degree from any four year school and they'd have been courting me, but without a degree my resume went directly into the trash.
HR's biggest job is gatekeeper; I knew I would not get hired through that, that I'd have to do all of the heavy lifting, cold calling (day after day, week after week, turned month after month, but knowing that no matter how long it would turn up) asking for the computer room, mostly they just hung up on me but I was playing the long game.
I finally got hired by First Interstate Bank (took right at a year) but not to bang out CoBOL but rather to learn assembly language, one short step up from writing in goddamned machine language. I gave all I had and it surely was not enough but they didn't run me off; they shut down their entire data shop in Houston, went from four data centers to two. Took at least a year to find that next gig, writing CoBOL for a state regulatory commission. It was there that I *finally* got some database experience, and if you've got some work history including some database coding also, a degree hardly matters anymore.
I stayed with them right at five years and then went out and saved the world from Y2K. I realize everybody thinks that was a big joke but it damn sure was not a joke, big or small -- the reason the world didn't crash and burn was because of the tremendous amount of work put in by programmers from all over the world -- I worked with a LOT of Philipino's, a few Samoan guys, lots of Indians also. Both the Indians and Philpino's were about equally split, men and women. Speaking of which -- the reason that programmers here are getting stiffed is because our sweet govt shoving jobs off-shore. Guarantee -- you wave all these people goodbye and *plenty* of people would rub CoBOL books all over their heads.
Mostly when I was working Y2K did I work on really, really old code. Completely unstructured, completely undocumented, *filled* with GO TO statements, which will get your fingers cut off if you try to sleaze that crap into production today. I worked CoBOL but also DYL260, and DYL280 -- DYL260 in particular was absolutely hieroglyphic, DYL280 a few years newer thus considerably more readable, *much* easier to make sense of. We only had one page of documentation for DYL260, the only way to see how it works is to feed a file in, run it, then look at your file, see what has changed -- truly, black box programming. Because I'm stubborn as fuck I became the go to guy for DYL260. One system we thought we were completely clean in, and then I saw something that looked suspiciously like a date field, and it *was* a date field, and we had to go through any program that was touched by data output by that particular file.
That really sucked.
So -- why aren't these people willing to pay programmers a decent (by which I mean obscenely, grotesquely high) they aren't paying programmers a decent wage because they don't have to, so much of it is off-shore now.
Another question kept on coming up in this thread -- Why not just dump this big steaming pile of dogshit and build bright shiny new systems? For one, if it isn't broke, why play silly games with it. Two: Every piece of business rules is in this code. You're going to just change that? No, your not. For one, big iron is amazingly fast, you punch in your data, hit enter, and Bang there is your data. Not even a seconds pause. That state regulatory agency I worked at, someone from Oracle or who knows where told some idiot in management that it's by god easy-peasy to get pretty, flashy, fun software, and it would even prevent cavities in your teeth, and you'd never, ever get another hemorrhoid, and it's just totally blazing fast, to boot.
Hardy har.
So every business rule is codified in these millions upon millions of lines of code. I have no idea how many billions of dollars that's worth, or trillions, but it's absolutely got to be substantial.
~~~~~
I was speaking with an old friend last night, we learned CoBOL together, we both worked downtown Houston for a while. He's a great guy, a trusted friend. We get to reminiscing The Old Days; I was never a great programmer but I got good at it. Same was I was as a carpenter -- I was never great but I was pretty damned good. And that stubborn part of me, that kind of forces my hand, I get angry and then I get *really* angry, and determined, and I say many comical bad words but there's no comedy in my heart as I bark and jump up and down and bang the keys.
I miss it, sometimes. There is an editor -- ISPF (Interactive System Productivity Facility) -- which is used to write programs, and to write JCL (don't ask), it's a really powerful tool and I got real good at it, and I miss it. Easy to miss something if you got proficient in it. For whatever reason(s) I got REAL good at working with tables (arrays if you're not onto CoBOL) -- I got to where I could really easily hold a dimensional table if one needed written, somehow my mind can follow that piece down.
~~~~~
Anyways. CoBOL isn't going anywhere anytime soon. Nor are all of the systems built with assembly language, and running on big iron. The dept I supported in that bank I worked it, it had massive systems entirely devoted to "Float" processing, a huge profit center for any bank; basically, how long can we keep our grubby, grabby hands on your money once it's in our hands IE how long can we float along on another persons money.
I'm sure that there's been lots of changes, it's been thirty years and change since I worked in that bank, and coming on 20 years since banging the keys on any CoBOL.
posted by dancestoblue at 4:22 AM on November 5, 2021 [34 favorites]
You can credit or blame Grace Hopper for it's existence; she was a computer scientist in the US Navy, not sure her rank when she was given this responsibility but she was a rear Admiral by the time she retired.
I learned it in 1989, did an internship at Pennzoil downtown Houston, they were straight-up with me, it was an internship, just a bit of blather to put on a resume. I did not have any kind of degree -- any outfit would have hired me if I'd had even a degree in basket weaving and they'd have taught me to code; I already knew how to code (I was no ace but I had the rudiments) but no matter. Hard truth -- any degree from any four year school and they'd have been courting me, but without a degree my resume went directly into the trash.
HR's biggest job is gatekeeper; I knew I would not get hired through that, that I'd have to do all of the heavy lifting, cold calling (day after day, week after week, turned month after month, but knowing that no matter how long it would turn up) asking for the computer room, mostly they just hung up on me but I was playing the long game.
I finally got hired by First Interstate Bank (took right at a year) but not to bang out CoBOL but rather to learn assembly language, one short step up from writing in goddamned machine language. I gave all I had and it surely was not enough but they didn't run me off; they shut down their entire data shop in Houston, went from four data centers to two. Took at least a year to find that next gig, writing CoBOL for a state regulatory commission. It was there that I *finally* got some database experience, and if you've got some work history including some database coding also, a degree hardly matters anymore.
I stayed with them right at five years and then went out and saved the world from Y2K. I realize everybody thinks that was a big joke but it damn sure was not a joke, big or small -- the reason the world didn't crash and burn was because of the tremendous amount of work put in by programmers from all over the world -- I worked with a LOT of Philipino's, a few Samoan guys, lots of Indians also. Both the Indians and Philpino's were about equally split, men and women. Speaking of which -- the reason that programmers here are getting stiffed is because our sweet govt shoving jobs off-shore. Guarantee -- you wave all these people goodbye and *plenty* of people would rub CoBOL books all over their heads.
Mostly when I was working Y2K did I work on really, really old code. Completely unstructured, completely undocumented, *filled* with GO TO statements, which will get your fingers cut off if you try to sleaze that crap into production today. I worked CoBOL but also DYL260, and DYL280 -- DYL260 in particular was absolutely hieroglyphic, DYL280 a few years newer thus considerably more readable, *much* easier to make sense of. We only had one page of documentation for DYL260, the only way to see how it works is to feed a file in, run it, then look at your file, see what has changed -- truly, black box programming. Because I'm stubborn as fuck I became the go to guy for DYL260. One system we thought we were completely clean in, and then I saw something that looked suspiciously like a date field, and it *was* a date field, and we had to go through any program that was touched by data output by that particular file.
That really sucked.
So -- why aren't these people willing to pay programmers a decent (by which I mean obscenely, grotesquely high) they aren't paying programmers a decent wage because they don't have to, so much of it is off-shore now.
Another question kept on coming up in this thread -- Why not just dump this big steaming pile of dogshit and build bright shiny new systems? For one, if it isn't broke, why play silly games with it. Two: Every piece of business rules is in this code. You're going to just change that? No, your not. For one, big iron is amazingly fast, you punch in your data, hit enter, and Bang there is your data. Not even a seconds pause. That state regulatory agency I worked at, someone from Oracle or who knows where told some idiot in management that it's by god easy-peasy to get pretty, flashy, fun software, and it would even prevent cavities in your teeth, and you'd never, ever get another hemorrhoid, and it's just totally blazing fast, to boot.
Hardy har.
So every business rule is codified in these millions upon millions of lines of code. I have no idea how many billions of dollars that's worth, or trillions, but it's absolutely got to be substantial.
~~~~~
I was speaking with an old friend last night, we learned CoBOL together, we both worked downtown Houston for a while. He's a great guy, a trusted friend. We get to reminiscing The Old Days; I was never a great programmer but I got good at it. Same was I was as a carpenter -- I was never great but I was pretty damned good. And that stubborn part of me, that kind of forces my hand, I get angry and then I get *really* angry, and determined, and I say many comical bad words but there's no comedy in my heart as I bark and jump up and down and bang the keys.
I miss it, sometimes. There is an editor -- ISPF (Interactive System Productivity Facility) -- which is used to write programs, and to write JCL (don't ask), it's a really powerful tool and I got real good at it, and I miss it. Easy to miss something if you got proficient in it. For whatever reason(s) I got REAL good at working with tables (arrays if you're not onto CoBOL) -- I got to where I could really easily hold a dimensional table if one needed written, somehow my mind can follow that piece down.
~~~~~
Anyways. CoBOL isn't going anywhere anytime soon. Nor are all of the systems built with assembly language, and running on big iron. The dept I supported in that bank I worked it, it had massive systems entirely devoted to "Float" processing, a huge profit center for any bank; basically, how long can we keep our grubby, grabby hands on your money once it's in our hands IE how long can we float along on another persons money.
I'm sure that there's been lots of changes, it's been thirty years and change since I worked in that bank, and coming on 20 years since banging the keys on any CoBOL.
posted by dancestoblue at 4:22 AM on November 5, 2021 [34 favorites]
Why not write an AI to learn how to program in COBOL
"Here I am, brain the size of a planet, and they've got me doing this."
Probably the most popular higher-ed student information system has a (sluggish) heart of COBOL, and even though all the front ends are written in Java or whatever, each customer needs to pay an annual fee for a software layer that's just a shim to keep the COBOL running.
posted by wenestvedt at 4:24 AM on November 5, 2021 [5 favorites]
"Here I am, brain the size of a planet, and they've got me doing this."
Probably the most popular higher-ed student information system has a (sluggish) heart of COBOL, and even though all the front ends are written in Java or whatever, each customer needs to pay an annual fee for a software layer that's just a shim to keep the COBOL running.
posted by wenestvedt at 4:24 AM on November 5, 2021 [5 favorites]
Is Weird Al a standard measure of time these days?
posted by goHermGO at 9:30 PM on November 4
Actually, COBOL is the Weird Al of programming languages.
posted by ZenMasterThis at 5:26 AM on November 5, 2021
posted by goHermGO at 9:30 PM on November 4
Actually, COBOL is the Weird Al of programming languages.
posted by ZenMasterThis at 5:26 AM on November 5, 2021
Having actually bothered to look at his Wikipedia page
So you've got Baby Alfred facts just top of your head??
posted by trig at 5:36 AM on November 5, 2021 [6 favorites]
So you've got Baby Alfred facts just top of your head??
posted by trig at 5:36 AM on November 5, 2021 [6 favorites]
I think ErisLordFreedom's comment is right on the money. There isn't really "cutting bait" in the IT world. At best you can migrate some things, but you're always going to be on the hook for that old dusty key thing that just can't move.
More on my AI comment: it's not a good idea because it starts from the wrong assumption (that we have a dearth of people who don't know COBOL). As stated earlier in the thread, it's the systems experience/expertise that makes a programmer good. But even if you did want to create an AI that wrote COBOL, (a) what code bases are you going to train it on? More to the point - whose code bases? (b) corrections to its output would have to be managed by a team of people who know COBOL and the ecosystem the program sits in anyway. Actually, they'd have to know the COBOL/ecosystem side AND how to create and update an AI. It's a weird combination of skillsets that would either come dearly in salaries, or you'd have to train them on one half or other of the job, which would also come dearly in salaries AND project time.
this comment brought to you by my dislike of AIs. I'd like to speak to a representative!
posted by snerson at 5:51 AM on November 5, 2021 [9 favorites]
More on my AI comment: it's not a good idea because it starts from the wrong assumption (that we have a dearth of people who don't know COBOL). As stated earlier in the thread, it's the systems experience/expertise that makes a programmer good. But even if you did want to create an AI that wrote COBOL, (a) what code bases are you going to train it on? More to the point - whose code bases? (b) corrections to its output would have to be managed by a team of people who know COBOL and the ecosystem the program sits in anyway. Actually, they'd have to know the COBOL/ecosystem side AND how to create and update an AI. It's a weird combination of skillsets that would either come dearly in salaries, or you'd have to train them on one half or other of the job, which would also come dearly in salaries AND project time.
this comment brought to you by my dislike of AIs. I'd like to speak to a representative!
posted by snerson at 5:51 AM on November 5, 2021 [9 favorites]
See also "An oral history of bank python"
Writer: So then we came up with Samuel L. Jackson's most famous movie quote.
Jackson: They wanted me to say "I'm tired of these motherfucking snakes in my motherfucking safety deposit boxes!" Can you believe some writer came up with that crap?
Writer: It's the best thing I've ever done in my entire career. It'll be written on my gravestone.
posted by NotMyselfRightNow at 5:53 AM on November 5, 2021 [4 favorites]
Writer: So then we came up with Samuel L. Jackson's most famous movie quote.
Jackson: They wanted me to say "I'm tired of these motherfucking snakes in my motherfucking safety deposit boxes!" Can you believe some writer came up with that crap?
Writer: It's the best thing I've ever done in my entire career. It'll be written on my gravestone.
posted by NotMyselfRightNow at 5:53 AM on November 5, 2021 [4 favorites]
Why not write an AI to learn how to program in COBOL?
Well... many years ago I worked on a modernization project (i.e. re-write the system using modern platform (.NET) and make it web-based). The incredibly overpriced 3-letter consultancy spent a year gathering requirements to win the bid.
Then, they built their team - 50% were sub-contractors (myself included in that group), as they did not have internal people with the correct skills.
I arrived, wearing my "solution architect" hat - and started digging through the 5-horizontal feet of requirements binders to begin building a cohesive architecture.
Several weeks in, I got to the section on "Business Rules" - basically this system was to approve student loan applications/grants/bursaries. Each year, the rules around who would get approved, for how much changed - this system was in-place since the 1970's - I expected at least one, possibly two full binders of documentation on what the existing rules were.
What did I get?
One sentence... "Go see Hugh".
Well - I scoured the systems for anything related to "Hugh" - was it a file-share? A server? The source-code repository?
No. Hugh was a COBOL programmer. Unfortunately, between the start of the requirements-gathering phase, the documentation and winning the bid, Hugh had retired...
Well - we put on our best thinking-caps... We wrote a parser, we ingested the COBOL business rules files directly - and then we generated our .NET assemblies. (While also storing the rules in a meta-language)
This proved to be a good choice... Why? Well, one of the requirements for the new system was to be able to take an existing application and run it for approval through time... Backwards. Therefore, we built a business rules meta-language, and the rules for each year generated the assemblies. Of course - as the loan/grant/bursary applications changed as well from year-to-year, the data-layer also had to be a meta-layer.
Exceedingly complex - but - we had working beta system 8-months after starting, and launched according to schedule.
posted by rozcakj at 6:21 AM on November 5, 2021 [18 favorites]
Well... many years ago I worked on a modernization project (i.e. re-write the system using modern platform (.NET) and make it web-based). The incredibly overpriced 3-letter consultancy spent a year gathering requirements to win the bid.
Then, they built their team - 50% were sub-contractors (myself included in that group), as they did not have internal people with the correct skills.
I arrived, wearing my "solution architect" hat - and started digging through the 5-horizontal feet of requirements binders to begin building a cohesive architecture.
Several weeks in, I got to the section on "Business Rules" - basically this system was to approve student loan applications/grants/bursaries. Each year, the rules around who would get approved, for how much changed - this system was in-place since the 1970's - I expected at least one, possibly two full binders of documentation on what the existing rules were.
What did I get?
One sentence... "Go see Hugh".
Well - I scoured the systems for anything related to "Hugh" - was it a file-share? A server? The source-code repository?
No. Hugh was a COBOL programmer. Unfortunately, between the start of the requirements-gathering phase, the documentation and winning the bid, Hugh had retired...
Well - we put on our best thinking-caps... We wrote a parser, we ingested the COBOL business rules files directly - and then we generated our .NET assemblies. (While also storing the rules in a meta-language)
This proved to be a good choice... Why? Well, one of the requirements for the new system was to be able to take an existing application and run it for approval through time... Backwards. Therefore, we built a business rules meta-language, and the rules for each year generated the assemblies. Of course - as the loan/grant/bursary applications changed as well from year-to-year, the data-layer also had to be a meta-layer.
Exceedingly complex - but - we had working beta system 8-months after starting, and launched according to schedule.
posted by rozcakj at 6:21 AM on November 5, 2021 [18 favorites]
it seems to me like you ought to at least get Paid, but clearly these banks hadn't got the memo last time I looked into it
In yet a different career direction, I became a specialist in a certain Enterprise-technology - heavily used in the Canadian banking sector - and one of my sub-specialities was migrations to new versions.
Every two years or so, I would have recruiters frantically calling me to get me to work for one single bank (on contract of course), because it was starting an upgrade project - it was a very prestigious opportunity, I should be lucky they would even deign to call me...
Except - that they were always consistently offering a third of the going market hourly rate for these highly specialized services. From reading between the lines of their various "job postings", it seems to me that a project which should have take between 6-12 months, was dragging for 5-7 years...
Banks are cheap.
posted by rozcakj at 6:28 AM on November 5, 2021 [15 favorites]
In yet a different career direction, I became a specialist in a certain Enterprise-technology - heavily used in the Canadian banking sector - and one of my sub-specialities was migrations to new versions.
Every two years or so, I would have recruiters frantically calling me to get me to work for one single bank (on contract of course), because it was starting an upgrade project - it was a very prestigious opportunity, I should be lucky they would even deign to call me...
Except - that they were always consistently offering a third of the going market hourly rate for these highly specialized services. From reading between the lines of their various "job postings", it seems to me that a project which should have take between 6-12 months, was dragging for 5-7 years...
Banks are cheap.
posted by rozcakj at 6:28 AM on November 5, 2021 [15 favorites]
I believe we are past that point already - a lot of these dinosaur systems run on emulators and virtual machines.
In 1996, I got the chance to tour a regional Canadian stock exchange data-center - I was so excited, first-time I would be in a real data-center - I was so looking forward to seeing the "big iron". I was told that it took-up an entire floor of an office tower. (haha, times have changed).
We arrived - they opened the door... It was vast - all "clean-room-white", with raised floor-tiles, so incredibly bright - and... so freaking empty... There were two rack towers in the center of the vast empty expanse.
"Oh yeah, with today's new hardware we were able to virtualize and consolidate everything"
1996...
posted by rozcakj at 6:44 AM on November 5, 2021 [5 favorites]
In 1996, I got the chance to tour a regional Canadian stock exchange data-center - I was so excited, first-time I would be in a real data-center - I was so looking forward to seeing the "big iron". I was told that it took-up an entire floor of an office tower. (haha, times have changed).
We arrived - they opened the door... It was vast - all "clean-room-white", with raised floor-tiles, so incredibly bright - and... so freaking empty... There were two rack towers in the center of the vast empty expanse.
"Oh yeah, with today's new hardware we were able to virtualize and consolidate everything"
1996...
posted by rozcakj at 6:44 AM on November 5, 2021 [5 favorites]
I once met someone at a party who did COBOL consulting and since I enjoy a certain amount of hobby retro-programming, ever since then I've had it mentally on my "if I quit academia" list (ime every academic has one, even if it's not a very serious list and they don't admit its existence). So I'm a bit sad to learn that it generally doesn't pay well. The person I was talking to did describe it as well-compensated, but she was also mainly consulting for government banks I believe, so maybe that's a different story. (Also possible that someone who has been doing this for an entire career doesn't know pay scales for current software development..)
posted by advil at 6:58 AM on November 5, 2021 [3 favorites]
posted by advil at 6:58 AM on November 5, 2021 [3 favorites]
here's a fun 2018 press release about modernising a system including a million lines of COBOL
COBOL has a million lines of code just in it's data division. :)
All those numbers listed are super small for a modern system (18k users, 500k transactions), but the hardware support numbers sound correct, maybe even a bit low. My group did a COBOL transition before 2018, our hardware costs went from $40m a year for bare metal to currently about $200k a month in the MS Cloud for 20X the number of transactions. The transition finance numbers are so crazy, any bank that sticks with COBOL doesn't understand basic project financing.
posted by The_Vegetables at 7:28 AM on November 5, 2021 [10 favorites]
COBOL has a million lines of code just in it's data division. :)
All those numbers listed are super small for a modern system (18k users, 500k transactions), but the hardware support numbers sound correct, maybe even a bit low. My group did a COBOL transition before 2018, our hardware costs went from $40m a year for bare metal to currently about $200k a month in the MS Cloud for 20X the number of transactions. The transition finance numbers are so crazy, any bank that sticks with COBOL doesn't understand basic project financing.
posted by The_Vegetables at 7:28 AM on November 5, 2021 [10 favorites]
doesn't understand basic project financing.
Never underestimate the ability of execs to be utterly clueless about the foundations of their own industry, in exchange for an ability to either be a crazed lunatic who doesn't think for more than five seconds at a go, or else a mewling coward who cannot make a decision in less than six years.
posted by aramaic at 7:44 AM on November 5, 2021 [10 favorites]
Never underestimate the ability of execs to be utterly clueless about the foundations of their own industry, in exchange for an ability to either be a crazed lunatic who doesn't think for more than five seconds at a go, or else a mewling coward who cannot make a decision in less than six years.
posted by aramaic at 7:44 AM on November 5, 2021 [10 favorites]
My sister works for a big bank; my brother and I are nerds who have cursed COBOL within her hearing. So last year she was excited to describe how the bank found some junior staff, offered them a little extra pay, and sent them all through a class to be COBOL programmers. That way they would be in-house (so no frantic hiring) and already up to speed on the system when they were needed.
I had to admit, that bank was spending some now to save more later, instead of just thinking about the current quarter's numbers.
posted by wenestvedt at 7:59 AM on November 5, 2021 [9 favorites]
I had to admit, that bank was spending some now to save more later, instead of just thinking about the current quarter's numbers.
posted by wenestvedt at 7:59 AM on November 5, 2021 [9 favorites]
Okay, so we're coming up on the point where banks will really fail because of old code. The retired people mostly won't be available. What happens then? I think doomsday(?) is within 20 years.
posted by Nancy Lebovitz at 8:00 AM on November 5, 2021 [1 favorite]
posted by Nancy Lebovitz at 8:00 AM on November 5, 2021 [1 favorite]
trig: So you've got Baby Alfred facts just top of your head??
As much as I’d like to claim encyclopedic knowledge of Weird Al, I heard mention of his birthday on the radio, and thought it had been last Friday. Either I misremembered what day it was or it was a repeat.
posted by Kattullus at 8:12 AM on November 5, 2021 [2 favorites]
As much as I’d like to claim encyclopedic knowledge of Weird Al, I heard mention of his birthday on the radio, and thought it had been last Friday. Either I misremembered what day it was or it was a repeat.
posted by Kattullus at 8:12 AM on November 5, 2021 [2 favorites]
When I was in college in the mid 90s I took COBOL classes. Did okay in them. But I also did not have a car so I couldn't take internships in Pittsburgh ( 90 minutes away ) and as such had to take one in the English department, where I learned PHP and HTML and the rest is history.
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 8:23 AM on November 5, 2021 [2 favorites]
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 8:23 AM on November 5, 2021 [2 favorites]
There's one other piece of the "why haven't they migrated to something supported already" question, and that is that there are two types of banks. [0]
I work in the observability space, which is an unnecessarily fancy way of saying that I help organizations figure out how to monitor and improve their large complex systems. I've gotten to work with a lot of banks -- many of whose names you would recognize -- and invariably after the first 30 minute phone call I can tell which type of bank they will be.
One type values deep and creative thinking. They are tech forward, often building systems that rival or exceed the much more public ones in Silicon Valley companies. They push the envelope, constantly trying to improve. They think hard about what could go wrong, find ways to mitigate those scenarios, then think about what could go wrong with the new system. They reward people for finding problems and have blameless retrospectives when something goes wrong by mistake, because they are primarily interested in solving the problem. This type of bank hires "A players".
The other type of bank... well, they hire "C players" and dumb those folks down as much as possible. They rely on rules and tradition instead of thinking, and their biggest concern is always, always having someone else they can blame for anything. Swapping out a system that will fail in the future is a risk. After all, it works now and it's been there for decades so you can hardly blame anyone for keeping it. A project to replace it might have some bumpy patches and cause disruption and the person who started the project will be the one who gets the blame. And if you're a C player, it's harder to find a job because you don't really have any useful skills, so better to just stay comfortably where you are.
The first type of bank has already replaced their COBOL long ago, or has it carefully isolated and well understood so that when it becomes necessary they can finish that process seamlessly. The second type of bank wants to pay someone $30k per year to not have to be blamed for ever changing anything.
(Yes, I have in fact changed who I bank with as a result of working this job. No, sadly I won't name names because I do like my job and want to continue having it.)
_____
[0]: Not just banks -- this applies pretty much universally from what I can tell. But we're talking specifically about banks here.
posted by fader at 9:29 AM on November 5, 2021 [14 favorites]
I work in the observability space, which is an unnecessarily fancy way of saying that I help organizations figure out how to monitor and improve their large complex systems. I've gotten to work with a lot of banks -- many of whose names you would recognize -- and invariably after the first 30 minute phone call I can tell which type of bank they will be.
One type values deep and creative thinking. They are tech forward, often building systems that rival or exceed the much more public ones in Silicon Valley companies. They push the envelope, constantly trying to improve. They think hard about what could go wrong, find ways to mitigate those scenarios, then think about what could go wrong with the new system. They reward people for finding problems and have blameless retrospectives when something goes wrong by mistake, because they are primarily interested in solving the problem. This type of bank hires "A players".
The other type of bank... well, they hire "C players" and dumb those folks down as much as possible. They rely on rules and tradition instead of thinking, and their biggest concern is always, always having someone else they can blame for anything. Swapping out a system that will fail in the future is a risk. After all, it works now and it's been there for decades so you can hardly blame anyone for keeping it. A project to replace it might have some bumpy patches and cause disruption and the person who started the project will be the one who gets the blame. And if you're a C player, it's harder to find a job because you don't really have any useful skills, so better to just stay comfortably where you are.
The first type of bank has already replaced their COBOL long ago, or has it carefully isolated and well understood so that when it becomes necessary they can finish that process seamlessly. The second type of bank wants to pay someone $30k per year to not have to be blamed for ever changing anything.
(Yes, I have in fact changed who I bank with as a result of working this job. No, sadly I won't name names because I do like my job and want to continue having it.)
_____
[0]: Not just banks -- this applies pretty much universally from what I can tell. But we're talking specifically about banks here.
posted by fader at 9:29 AM on November 5, 2021 [14 favorites]
Baby Alfred was born on October 29th, 1959 ... COBOL ... first draft ... November 13th, 1959.
As of (and not counting) today, Alfred Matthew Yankovic is 22,653 days old. CoBOL is 22,638 days old.
So, CoBOL is currently 0.99933783604821 Weirds Al, for those of you keeping track.
posted by ChurchHatesTucker at 10:06 AM on November 5, 2021 [9 favorites]
As of (and not counting) today, Alfred Matthew Yankovic is 22,653 days old. CoBOL is 22,638 days old.
So, CoBOL is currently 0.99933783604821 Weirds Al, for those of you keeping track.
posted by ChurchHatesTucker at 10:06 AM on November 5, 2021 [9 favorites]
I had no idea the secret cabal of bank software developers was right here on Metafilter.
posted by clawsoon at 10:15 AM on November 5, 2021 [8 favorites]
posted by clawsoon at 10:15 AM on November 5, 2021 [8 favorites]
Somewhere, in a cool computer room in the sky, Admiral Hopper is having a good laugh.
posted by tommasz at 10:36 AM on November 5, 2021 [4 favorites]
posted by tommasz at 10:36 AM on November 5, 2021 [4 favorites]
In conclusion, "Weird" Al Yankovic should be inducted into the Rock and Roll Hall of Fame.
posted by Superilla at 11:09 AM on November 5, 2021 [4 favorites]
posted by Superilla at 11:09 AM on November 5, 2021 [4 favorites]
This analysis led to our decision to use a COBOL-to-Java code automated refactoring solution. This option would take a low-risk, incremental approach and apply a blended agile/traditional methodology and tools to ensure rapid, high-quality software delivery.
Oh. Oh no.
posted by Kadin2048 at 11:15 AM on November 5, 2021 [15 favorites]
Oh. Oh no.
posted by Kadin2048 at 11:15 AM on November 5, 2021 [15 favorites]
Programming posts are one of my favorites on Metafilter due to all the stories that come out. Thank you storybored for the post and for the community for sharing. :)
posted by curious nu at 12:00 PM on November 5, 2021 [9 favorites]
posted by curious nu at 12:00 PM on November 5, 2021 [9 favorites]
There's something romantic about how you unpeel the modern layers of code to find deep down an ancient beating heart.
It's also kind of endearing to older software types like me - your pearl of brainy work hidden away inside the oyster, who knows for how many more decades. Business system apps hang around and they run the day-to-day. Not like the flashy consumer stuff that comes and goes.
I coded in a custom-built version of Pascal in the mid-80s. I was part of the team that built the file system for a telecom switch. That code has now been translated into C. It ran on a custom OS. The OS now runs as a task in several layers of emulation. But it still runs!
I believe it is being superseded by new cloud-based telecom products, but those who buy the standalone server get the full nostalgic bundle. The company's telecom gear is processing millions of phone calls a year, running software I wrote as a nervous greenhorn out of university forty years ago. I get warm fuzzies thinking about it.
posted by storybored at 12:03 PM on November 5, 2021 [10 favorites]
It's also kind of endearing to older software types like me - your pearl of brainy work hidden away inside the oyster, who knows for how many more decades. Business system apps hang around and they run the day-to-day. Not like the flashy consumer stuff that comes and goes.
I coded in a custom-built version of Pascal in the mid-80s. I was part of the team that built the file system for a telecom switch. That code has now been translated into C. It ran on a custom OS. The OS now runs as a task in several layers of emulation. But it still runs!
I believe it is being superseded by new cloud-based telecom products, but those who buy the standalone server get the full nostalgic bundle. The company's telecom gear is processing millions of phone calls a year, running software I wrote as a nervous greenhorn out of university forty years ago. I get warm fuzzies thinking about it.
posted by storybored at 12:03 PM on November 5, 2021 [10 favorites]
verify it with unit tests and ...
BUT WHO WILL WRITE THE TESTS?
Yeah it's a joke. Don't er... mock me ;)
posted by klanawa at 12:05 PM on November 5, 2021 [5 favorites]
BUT WHO WILL WRITE THE TESTS?
Yeah it's a joke. Don't er... mock me ;)
posted by klanawa at 12:05 PM on November 5, 2021 [5 favorites]
I wonder if part of the problem in our supply chain is that so many of these old COBOL programmers retired or died during covid.
posted by interogative mood at 12:13 PM on November 5, 2021 [1 favorite]
posted by interogative mood at 12:13 PM on November 5, 2021 [1 favorite]
BUT WHO WILL WRITE THE TESTS?
Uh...
:-/
... We generated those from our "meta-language" as well... After all, each "business rule" had a known input range, which would provide an known/expected (approved/denied) output...
posted by rozcakj at 12:40 PM on November 5, 2021 [3 favorites]
Uh...
:-/
... We generated those from our "meta-language" as well... After all, each "business rule" had a known input range, which would provide an known/expected (approved/denied) output...
posted by rozcakj at 12:40 PM on November 5, 2021 [3 favorites]
Let me guess... the code works well enough that they've eliminated most of their developer base by attrition. Nobody has tried to hire an entry level developer in 25 years, and it was thirty five years since they did a competitive analysis of salaries. So, instead of maintaining an ecosystem, they'll pay well into $xxx/hr to drag their hoary consultants out of retirement for a few weeks a year to keep the house of cards from collapsing
Wow sounds like you read the story
posted by sock poppet at 12:41 PM on November 5, 2021 [1 favorite]
Wow sounds like you read the story
posted by sock poppet at 12:41 PM on November 5, 2021 [1 favorite]
...to keep the house of cards from collapsing
more at "to add a plastic atrium to a house of stone, without cracking any of the stones."?
posted by storybored at 1:26 PM on November 5, 2021
more at "to add a plastic atrium to a house of stone, without cracking any of the stones."?
posted by storybored at 1:26 PM on November 5, 2021
So every business rule is codified in these millions upon millions of lines of code. I have no idea how many billions of dollars that's worth, or trillions, but it's absolutely got to be substantial.
That's the core of the problem right there. You're going to have to reverse engineer the business logic that has been codified in the system by accretion over half a century. That's not a coding problem it's a specification problem and it could take years to document all the use-cases that have been built up over time. Coding is easy, knowing what to code is the hard part.
posted by octothorpe at 1:52 PM on November 5, 2021 [13 favorites]
That's the core of the problem right there. You're going to have to reverse engineer the business logic that has been codified in the system by accretion over half a century. That's not a coding problem it's a specification problem and it could take years to document all the use-cases that have been built up over time. Coding is easy, knowing what to code is the hard part.
posted by octothorpe at 1:52 PM on November 5, 2021 [13 favorites]
this comment brought to you by my dislike of AIs. I'd like to speak to a representative!
Greetings from the temple to Roko's basilisk. I am currently responsible for the care and feeding of a young virtual assistant you have definitely heard of.
AI, for all its wonderful magic, is best deployed where the cost of being wrong one percent of the time is low, and being right is subjective. So many of the problems assigned to AI is ultimately about society's leaders deploying AI. Recommender systems going wrong might seem harmless, but it really depends on the domain -- the consequences of preemptively recording shows you might like is qualitatively different than recommending parole.
I recognize that Github's AI tool has captured some imagination, but it's intended to supplement programmers rather than fully replace them, and in any case has virtually no COBOL to train on. In the general case,
ML writing code could be fine, depending on how the results are used. Like, if I used ML to generate code to fuzz test a complier, nobody would object, even though the code is likely inscruitable. If I used the ML to generate code that flags loan applicants for extra scrutiny, medium sized problem. But if someone uses AI to rewrite all the COBOL that runs a bank, which everyone's afraid to touch because there is no test suite or quality controls, well, let's hope the office burns down before anyone discovers what a mistake that was.
There's also the question of alternatives: what is the alternative to not using this? Multiple times in my career I have beaten a fully staffed AIOps team to solving a problem using regular expressions and RAS, before they've even finalized an implementation plan. In contrast, self-driving cars is generally regarded as better than humans, but public skepticism over the tech will likely mean more dead commuters than otherwise would result. In the case of bank COBOL the alternative is paying humans to do the task. Average executive pay plays a factor in willingness to pay engineers, so probably the long term solution is yet another regional bank merging with a larger bank that already solved the problem. As discussed in the last banking thread, the US is littered with small regional banks as a policy choice, and it will take a few lifetimes for industry to converge on the new equilibrium since that policy was reversed.
posted by pwnguin at 2:07 PM on November 5, 2021 [2 favorites]
Greetings from the temple to Roko's basilisk. I am currently responsible for the care and feeding of a young virtual assistant you have definitely heard of.
AI, for all its wonderful magic, is best deployed where the cost of being wrong one percent of the time is low, and being right is subjective. So many of the problems assigned to AI is ultimately about society's leaders deploying AI. Recommender systems going wrong might seem harmless, but it really depends on the domain -- the consequences of preemptively recording shows you might like is qualitatively different than recommending parole.
I recognize that Github's AI tool has captured some imagination, but it's intended to supplement programmers rather than fully replace them, and in any case has virtually no COBOL to train on. In the general case,
ML writing code could be fine, depending on how the results are used. Like, if I used ML to generate code to fuzz test a complier, nobody would object, even though the code is likely inscruitable. If I used the ML to generate code that flags loan applicants for extra scrutiny, medium sized problem. But if someone uses AI to rewrite all the COBOL that runs a bank, which everyone's afraid to touch because there is no test suite or quality controls, well, let's hope the office burns down before anyone discovers what a mistake that was.
There's also the question of alternatives: what is the alternative to not using this? Multiple times in my career I have beaten a fully staffed AIOps team to solving a problem using regular expressions and RAS, before they've even finalized an implementation plan. In contrast, self-driving cars is generally regarded as better than humans, but public skepticism over the tech will likely mean more dead commuters than otherwise would result. In the case of bank COBOL the alternative is paying humans to do the task. Average executive pay plays a factor in willingness to pay engineers, so probably the long term solution is yet another regional bank merging with a larger bank that already solved the problem. As discussed in the last banking thread, the US is littered with small regional banks as a policy choice, and it will take a few lifetimes for industry to converge on the new equilibrium since that policy was reversed.
posted by pwnguin at 2:07 PM on November 5, 2021 [2 favorites]
The tales of ancient programmers desperately needed to maintain COBOL are an example of capitalism killing itself.
There's no dark art to COBOL, anyone who can code can learn it. I did, I was the very last person at my college who had a degree plan that required both assembly and COBOL. It's a pain in the ass, but it's nothing any reasonably intelligent person can't learn.
So why the lack of COBOL programmers and the desperate dependence on the old timers who are, inevitably, dying off?
Capitalism. You **COULD** fairly easily produce a corps of young programmers who are total COBOL experts. All that's needed is money. Offer a young programmer the chance to be a highly sought after specialist, high pay, and good benefits and they'd jump at the opportunity. It'd take a training period, but probably not as long as you might fear. Six months of intensive COBOL immersion would make for a fairly good COBOL programmer and after a year or two they'd be approaching expert level.
But no one makes the offer. They want people who are already COBOL experts. They don't want to pay what it takes to train those experts, they just want them to appear and take crap wages to boot.
Of course there's also the fact that legacy code is notoriously obnoxious to maintain. If you've got a person who actually wrote the damn program they'll have an easier time debugging it. But programmers have been dealing with poorly documented legacy code since forever, it's annoying but a solved problem. Mostly you just have to dive in and start immersing yourself in the code.
But again, that means you get a programmer who can do kind of OK right now but will take many months of being paid high wages and producing not so much for it before they can become experts not just in COBOL, but in your specific programs. And again, the companies don't want to pay.
Eventually they'll have to, when all the old COBOL programmers die.
Or, more likely, they'll complain endlessly that no one wants to work anymore, burn out the tiny handful of suckers who apply for the job by working them 80+ hours a week, and then wail as the code inevitably becomes worse and worse and buggier and buggier.
Hope you don't want to use an ATM thirty years from now...
posted by sotonohito at 4:23 PM on November 5, 2021 [18 favorites]
There's no dark art to COBOL, anyone who can code can learn it. I did, I was the very last person at my college who had a degree plan that required both assembly and COBOL. It's a pain in the ass, but it's nothing any reasonably intelligent person can't learn.
So why the lack of COBOL programmers and the desperate dependence on the old timers who are, inevitably, dying off?
Capitalism. You **COULD** fairly easily produce a corps of young programmers who are total COBOL experts. All that's needed is money. Offer a young programmer the chance to be a highly sought after specialist, high pay, and good benefits and they'd jump at the opportunity. It'd take a training period, but probably not as long as you might fear. Six months of intensive COBOL immersion would make for a fairly good COBOL programmer and after a year or two they'd be approaching expert level.
But no one makes the offer. They want people who are already COBOL experts. They don't want to pay what it takes to train those experts, they just want them to appear and take crap wages to boot.
Of course there's also the fact that legacy code is notoriously obnoxious to maintain. If you've got a person who actually wrote the damn program they'll have an easier time debugging it. But programmers have been dealing with poorly documented legacy code since forever, it's annoying but a solved problem. Mostly you just have to dive in and start immersing yourself in the code.
But again, that means you get a programmer who can do kind of OK right now but will take many months of being paid high wages and producing not so much for it before they can become experts not just in COBOL, but in your specific programs. And again, the companies don't want to pay.
Eventually they'll have to, when all the old COBOL programmers die.
Or, more likely, they'll complain endlessly that no one wants to work anymore, burn out the tiny handful of suckers who apply for the job by working them 80+ hours a week, and then wail as the code inevitably becomes worse and worse and buggier and buggier.
Hope you don't want to use an ATM thirty years from now...
posted by sotonohito at 4:23 PM on November 5, 2021 [18 favorites]
So last year she was excited to describe how the bank found some junior staff, offered them a little extra pay, and sent them all through a class to be COBOL programmers.
Can confirm. My brother almost got pulled into one of these programs. No idea whether or not it was at the same bank.
posted by paper chromatographologist at 4:30 PM on November 5, 2021 [1 favorite]
Can confirm. My brother almost got pulled into one of these programs. No idea whether or not it was at the same bank.
posted by paper chromatographologist at 4:30 PM on November 5, 2021 [1 favorite]
Vernor Vinge's book A Deepness in The Sky is a space opera set many thousands of years in the future and one of the characters is (because of relativity) centuries older than anyone else. He is able to perform near-magical hacks on the computer systems because everything is legacy systems built on legacy systems emulating still older legacy systems, and he knows about layers that nobody has understood for hundreds of years.
Reading this thread, I'm realizing it was probably the most realistic part of the whole book.
posted by straight at 4:54 PM on November 5, 2021 [20 favorites]
Reading this thread, I'm realizing it was probably the most realistic part of the whole book.
posted by straight at 4:54 PM on November 5, 2021 [20 favorites]
Very depressing - I understand in Australia a lot of healthcare and social support payments are based on COBOL systems, so disastrous effects if they go wrong.
posted by quercus23 at 7:00 PM on November 5, 2021 [1 favorite]
posted by quercus23 at 7:00 PM on November 5, 2021 [1 favorite]
Oh, man, fader hits the bank type thing on the nose. I used to be involved in hiring systems people in the Chicago area, and every bank person whose resume came over the transom for years... if you didn't know it before hand, the absolutely blank stares you'd get back when you asked a slightly improvisational question would tell you.
posted by wotsac at 7:59 PM on November 5, 2021
posted by wotsac at 7:59 PM on November 5, 2021
I remember this as a problem when I graduated from college 15 years ago, and at the campus recruitment fair there was a booth from, I think, either a large bank or a government agency, and they were looking for mathematics and physics majors interested in learning FORTRAN and COBOL. And I thought about it, because I had a physics degree and the job security seemed extremely good, but I didn't feel I was smart enough or good enough at math to learn these languages. Anyway I think this "shortage" is something that's been well known for a long time.
posted by subdee at 7:59 PM on November 5, 2021 [1 favorite]
posted by subdee at 7:59 PM on November 5, 2021 [1 favorite]
COBOL datasets to train AI would involve a lot of GOTO instructions. I am not certain what that would do but it fills me with a sense of unease.
posted by Ignorantsavage at 8:02 PM on November 5, 2021 [1 favorite]
posted by Ignorantsavage at 8:02 PM on November 5, 2021 [1 favorite]
I thought they just paid people in India to deal with this now.
posted by panama joe at 9:04 PM on November 5, 2021 [1 favorite]
posted by panama joe at 9:04 PM on November 5, 2021 [1 favorite]
Among the reasons for there still being big cobol estates at the core of banks and insurance companies is that the legacy systems work (not a lot of innovation in ledgers), and that the complex eco-system of risk management, accounting, BI and regulatory reporting that has grown over the decades rely on the specific idiosyncratic features (and sometimes bugs) of the core systems.
Another reason is that switching ledgers is difficult (I’m using understatement here), and when you’re done you have the same functionality as before. Perhaps wrapped in a better set of services (for eg reducing process times for customers etc), but you can get most of that by wrapping your legacy ledger with more flexible systems.
But I believe the main thing is this: no major bank has been able to show that you can make cost go down by replacing the core systems with something more modern. The legacy systems
are fully depreciated so they are very cheap to keep to keep running, whereas any new system will end up putting a lot of money on the balance sheet that must subsequently be written off, making costs go up. And since cost/income ratio is a major determining factor in share price for incumbent banks there’s just no business case. It’s not a case of does it require a large investment, it’s a case of no one can show that that large investment actually will end up making the bank money. I’ve waited for (and spent some time agitating about) a challenger bank with no legacy making a serious dent in profits for incumbents, as I believe this is the only thing that can move the needle on this.
Instead we are stuck with challenger banks with great customer satisfaction and enormous valuations (since measured on potential growth) but making practically no money, and incumbents with abysmal customer satisfaction, valuations based on return on equity (since measured on actual business) making enormous profits.
Sorry for rambling, I’m currently on gardening leave having previous been head of group lending at a multinational bank so this has been on my agenda all day every day for quite some time (and now I have time to vent about it). I’ve previously led a move of a national taxation system from mainframe to Java so I know it can be done. I believe a banking system to be at least an order of magnitude more complex though.
posted by boogieboy at 4:52 AM on November 6, 2021 [10 favorites]
Another reason is that switching ledgers is difficult (I’m using understatement here), and when you’re done you have the same functionality as before. Perhaps wrapped in a better set of services (for eg reducing process times for customers etc), but you can get most of that by wrapping your legacy ledger with more flexible systems.
But I believe the main thing is this: no major bank has been able to show that you can make cost go down by replacing the core systems with something more modern. The legacy systems
are fully depreciated so they are very cheap to keep to keep running, whereas any new system will end up putting a lot of money on the balance sheet that must subsequently be written off, making costs go up. And since cost/income ratio is a major determining factor in share price for incumbent banks there’s just no business case. It’s not a case of does it require a large investment, it’s a case of no one can show that that large investment actually will end up making the bank money. I’ve waited for (and spent some time agitating about) a challenger bank with no legacy making a serious dent in profits for incumbents, as I believe this is the only thing that can move the needle on this.
Instead we are stuck with challenger banks with great customer satisfaction and enormous valuations (since measured on potential growth) but making practically no money, and incumbents with abysmal customer satisfaction, valuations based on return on equity (since measured on actual business) making enormous profits.
Sorry for rambling, I’m currently on gardening leave having previous been head of group lending at a multinational bank so this has been on my agenda all day every day for quite some time (and now I have time to vent about it). I’ve previously led a move of a national taxation system from mainframe to Java so I know it can be done. I believe a banking system to be at least an order of magnitude more complex though.
posted by boogieboy at 4:52 AM on November 6, 2021 [10 favorites]
In conclusion, "Weird" Al Yankovic should be inducted into the Rock and Roll Hall of Fame.
And since Cleveland is the COBOL of American cities
posted by waving at 7:51 AM on November 6, 2021 [1 favorite]
And since Cleveland is the COBOL of American cities
posted by waving at 7:51 AM on November 6, 2021 [1 favorite]
Gotta tell my 81 year old dad to put down his tuba and come out of retirement. I remember playing with the punch card machine when I was a kid.
posted by umbú at 8:06 AM on November 6, 2021 [2 favorites]
posted by umbú at 8:06 AM on November 6, 2021 [2 favorites]
Two of the biggest employers in my city are banks, PNC and BNY Mellon, so I've known a lot of people who have done stints at them and they're just the worst places to work as a software developer. The pay sucks, the processes are horribly hierarchical and antiquated, technical decisions have more to do with who the managers golf with than with their actual merit and the office environment is straight out of the 1980s.
posted by octothorpe at 8:58 AM on November 6, 2021 [2 favorites]
posted by octothorpe at 8:58 AM on November 6, 2021 [2 favorites]
In contrast, self-driving cars is generally regarded as better than humans, but public skepticism over the tech will likely mean more dead commuters than otherwise would result.
Only regarded that why by naive or tricked people.
The initial reports tesla put out where all incorrect, they had done the statistics wrong, it was not safer. Look into accidents compared by mile, but also by driving locations (city vs highway)
Tesla does the same with car safety too. They compare driver deaths vs all existing cars, but if you compare tesla driver deaths to cars from the same year that cost the same, it actually turns out they have a 4 times higher death rate.
posted by Iax at 10:43 AM on November 7, 2021 [3 favorites]
Only regarded that why by naive or tricked people.
The initial reports tesla put out where all incorrect, they had done the statistics wrong, it was not safer. Look into accidents compared by mile, but also by driving locations (city vs highway)
Tesla does the same with car safety too. They compare driver deaths vs all existing cars, but if you compare tesla driver deaths to cars from the same year that cost the same, it actually turns out they have a 4 times higher death rate.
posted by Iax at 10:43 AM on November 7, 2021 [3 favorites]
The tales of ancient programmers desperately needed to maintain COBOL are an example of capitalism killing itself.
Capital-ism survived Y2k, and will be just fine even if a few capital-ists are bankrupted in the course of events. The business cycle is seemingly an intractable part of capitalism, and so firms are born and die over time. Individual firms are allowed to make different decisions, and in the process society basically tests many solutions to a problem in parallel. COBOL is just another dimension on that front.
I'm not a banker, nor an executive, but I imagine there's a high level question about whether it's worth it for the First National Bank of Wamego, KS to modernize their computer system versus other options. In the event of a bank failure, there are countless others ready to pick over the corpse, take over deposits, and serve their customers. Perhaps by firms that have already applied the modernization process every other industry frequently applies. M&A in banking isn't new, and will keep happening for years to come:
If you've been a banking customer over the past 20 years you've likely witnessed the M&A trend first hand as a small regional bank is acquired and rebranded. Even internet darling ING Direct was sold off to Capital One, who at the very least is playing a very good marketing game with their tech stack.
And as kjs3 mentions, it's not like the people who wrote the business continuity rules for banks don't know about COBOL -- their software development booklet mentions it by name. People are ringing alarm bells, but I'm not sure 'nobody knows COBOL' rises to the level of systemic risk*.
And quite frankly, I don't know why anyone in Mefi would want a bank to make a long term investment in more COBOL over modernizing on something newer. If you're going to invest the money into tech, maybe instead of making Mt. Tech Debt taller, invest in something new? Even from a junior talent perspective, banking COBOL seems ripe for monopsony whereas a Python bootcamp graduate will have multiple industries to choose from.
tl;dr: A hypothetical bank closure due to COBOL talent waning is an indictment of management not capitalism as a whole.
Hope you don't want to use an ATM thirty years from now...
In the past 5 years I have used an ATM twice, and once was just to prove to myself the debit card works. Even the laundromats and gas station tire pumps here take cards and mobile payments. Every time I do use cash, I wonder in the back of my mind if I'm just enabling money laundering or tax evasion. And using more rigorous datasets, the pandemic did a number on cash as a payment method. I'm sure the coin shortage didn't help.
posted by pwnguin at 11:05 AM on November 7, 2021 [3 favorites]
Capital-ism survived Y2k, and will be just fine even if a few capital-ists are bankrupted in the course of events. The business cycle is seemingly an intractable part of capitalism, and so firms are born and die over time. Individual firms are allowed to make different decisions, and in the process society basically tests many solutions to a problem in parallel. COBOL is just another dimension on that front.
I'm not a banker, nor an executive, but I imagine there's a high level question about whether it's worth it for the First National Bank of Wamego, KS to modernize their computer system versus other options. In the event of a bank failure, there are countless others ready to pick over the corpse, take over deposits, and serve their customers. Perhaps by firms that have already applied the modernization process every other industry frequently applies. M&A in banking isn't new, and will keep happening for years to come:
"A lot of customers haven't walked into a bank branch for years. Regional banks are in many cases seeking economies of scale to invest in technology."While larger deals between publicly traded banks continue to drive M&A, smaller players are also taking part among the roughly 5,000 banks in the U.S. including small community banks.
If you've been a banking customer over the past 20 years you've likely witnessed the M&A trend first hand as a small regional bank is acquired and rebranded. Even internet darling ING Direct was sold off to Capital One, who at the very least is playing a very good marketing game with their tech stack.
And as kjs3 mentions, it's not like the people who wrote the business continuity rules for banks don't know about COBOL -- their software development booklet mentions it by name. People are ringing alarm bells, but I'm not sure 'nobody knows COBOL' rises to the level of systemic risk*.
And quite frankly, I don't know why anyone in Mefi would want a bank to make a long term investment in more COBOL over modernizing on something newer. If you're going to invest the money into tech, maybe instead of making Mt. Tech Debt taller, invest in something new? Even from a junior talent perspective, banking COBOL seems ripe for monopsony whereas a Python bootcamp graduate will have multiple industries to choose from.
tl;dr: A hypothetical bank closure due to COBOL talent waning is an indictment of management not capitalism as a whole.
Hope you don't want to use an ATM thirty years from now...
In the past 5 years I have used an ATM twice, and once was just to prove to myself the debit card works. Even the laundromats and gas station tire pumps here take cards and mobile payments. Every time I do use cash, I wonder in the back of my mind if I'm just enabling money laundering or tax evasion. And using more rigorous datasets, the pandemic did a number on cash as a payment method. I'm sure the coin shortage didn't help.
posted by pwnguin at 11:05 AM on November 7, 2021 [3 favorites]
(laughs in MUMPS)
Jesus Christ, man! Don't speak its name or it'll slouch forth from the VA's health care system to confuse and enrage us all!
posted by Mr. Bad Example at 2:40 PM on November 7, 2021 [5 favorites]
Jesus Christ, man! Don't speak its name or it'll slouch forth from the VA's health care system to confuse and enrage us all!
posted by Mr. Bad Example at 2:40 PM on November 7, 2021 [5 favorites]
Everybody always says COBOL works well enough, but then it took like 6 months for some people to get relief checks during the pandemic and if I recall correctly 2 states unemployment systems failed.
I guess it depends on who you mean when you say 'the code still works'.
posted by The_Vegetables at 12:33 PM on November 8, 2021 [1 favorite]
I guess it depends on who you mean when you say 'the code still works'.
posted by The_Vegetables at 12:33 PM on November 8, 2021 [1 favorite]
Migration does not necessarily lead to a better system, either in terms of maintainability or capacity. Florida's complete piece of shit UI system is relatively new, but is broken AF. (And designed to be difficult for users) Last I checked, user access is still being cut off every night and all weekend for "processing" after it was mostly unusable for a couple of months last year.
Point being that the issue isn't so much age of the code, but lack of competence or outright malfeasance among decision makers who dictate how systems are run.
posted by wierdo at 2:06 PM on November 8, 2021 [5 favorites]
Point being that the issue isn't so much age of the code, but lack of competence or outright malfeasance among decision makers who dictate how systems are run.
posted by wierdo at 2:06 PM on November 8, 2021 [5 favorites]
pwnguin: And quite frankly, I don't know why anyone in Mefi would want a bank to make a long term investment in more COBOL over modernizing on something newer. If you're going to invest the money into tech, maybe instead of making Mt. Tech Debt taller, invest in something new? Even from a junior talent perspective, banking COBOL seems ripe for monopsony whereas a Python bootcamp graduate will have multiple industries to choose from.
I'm in an industry that's full of Python technical debt because huge gobs of interdependent code were written in Python 2. I expect to be writing and maintaining Python 2 until the end of my career.
Which gets me thinking about what appear to be shorter and shorter cycles of programming language/environment/system obsolescence. Is there any language you could pick today for your banking system that won't be viewed as a disgusting pile of technical debt 50 years from now?
posted by clawsoon at 5:15 PM on November 8, 2021 [4 favorites]
I'm in an industry that's full of Python technical debt because huge gobs of interdependent code were written in Python 2. I expect to be writing and maintaining Python 2 until the end of my career.
Which gets me thinking about what appear to be shorter and shorter cycles of programming language/environment/system obsolescence. Is there any language you could pick today for your banking system that won't be viewed as a disgusting pile of technical debt 50 years from now?
posted by clawsoon at 5:15 PM on November 8, 2021 [4 favorites]
Is there any language you could pick today for your banking system that won't be viewed as a disgusting pile of technical debt 50 years from now?
LISP, because nearly no one uses it now so they userbase can't shrink much?
posted by wenestvedt at 5:53 PM on November 8, 2021 [1 favorite]
LISP, because nearly no one uses it now so they userbase can't shrink much?
posted by wenestvedt at 5:53 PM on November 8, 2021 [1 favorite]
Use of "Use of GOTO Considered Harmful" Considered Harmful
posted by thelonius at 9:16 PM on November 8, 2021
posted by thelonius at 9:16 PM on November 8, 2021
outright malfeasance among decision makers who dictate how systems are run
And maybe some ignorance/laziness/corruption on the part of civil servants who would rather sign big contracts with Oracle or Deloitte worth hundreds of millions for software that will be abandoned without ever having been used than hire a few tens of tech workers to develop and maintain systems over the long term.
posted by klanawa at 12:04 PM on November 9, 2021 [1 favorite]
And maybe some ignorance/laziness/corruption on the part of civil servants who would rather sign big contracts with Oracle or Deloitte worth hundreds of millions for software that will be abandoned without ever having been used than hire a few tens of tech workers to develop and maintain systems over the long term.
posted by klanawa at 12:04 PM on November 9, 2021 [1 favorite]
That's certainly part of the issue, klanawa. In Florida, though, Deloitte delivered exactly what they were asked to provide. There are even documents showing them pushing back on some of the requirements because they knew it was going to be a shit show. The blame they deserve in that particular case is their decision to continue when the state insisted on a system designed to make it as difficult as possible to claim benefits and to fail when claim volumes are high in order to make as many people as possible give up and go away.
posted by wierdo at 5:15 PM on November 9, 2021 [4 favorites]
posted by wierdo at 5:15 PM on November 9, 2021 [4 favorites]
We got this: COBOL in 100 seconds
posted by ChurchHatesTucker at 9:52 PM on November 14, 2021 [3 favorites]
posted by ChurchHatesTucker at 9:52 PM on November 14, 2021 [3 favorites]
« Older Glue predates anatomically modern humans and other... | cosmic inflation Newer »
This thread has been archived and is closed to new comments
Is Weird Al a standard measure of time these days?
posted by goHermGO at 6:30 PM on November 4, 2021 [65 favorites]