Adobe Flash update knocks out train system in Dalian, China, for 20 hour
January 21, 2021 3:13 PM Subscribe
Recently, there was news that the train dispatching system of Dalian Railway in Liaoning Province was directly paralyzed due to the suspension of Flash, after which technical staff installed a reduced version of pirated Flash to solve the problem.
Was there a legal reason that Adobe put in a kill routine? That's weird they did that. Usually EOL means not supported.
posted by geoff. at 3:34 PM on January 21, 2021 [2 favorites]
posted by geoff. at 3:34 PM on January 21, 2021 [2 favorites]
Yeah, the point is to kill Flash dead, not just leave shitty code sitting around to be compromised.
Adobe needed to put an impermeable clay cap over the Flash code so it wouldn't leak out and cause cancer in the surrounding community.
posted by ryanrs at 3:36 PM on January 21, 2021 [28 favorites]
Adobe needed to put an impermeable clay cap over the Flash code so it wouldn't leak out and cause cancer in the surrounding community.
posted by ryanrs at 3:36 PM on January 21, 2021 [28 favorites]
epony-choo-choo-sterical SHURELY
posted by lalochezia at 3:43 PM on January 21, 2021 [10 favorites]
posted by lalochezia at 3:43 PM on January 21, 2021 [10 favorites]
Was there a legal reason that Adobe put in a kill routine? That's weird they did that. Usually EOL means not supported.
It wouldn't matter, the same shit would eventually happen. Also, the notification that the "kill" was coming happened months and months, maybe even years, ago.
My employer declared we would eventually not allow 32bit applications to run on a future OS version seven years ahead of time. Hell, the writing was written on the wall when 64bit applications were announced ~twelve year ahead of time. Users even got a pop up that said "This shit ain't going to work in the near future!" (OK I'm paraphrasing) for an entire year before the deadline.
Did all developers take heed and update their shit? No, they told their customers they had just no idea this could ever happen and that my employer was just a big meanie that just wanted to make the lives of developers miserable.
So, even if they EOL'd it for a decade, no one was going to fix anything unless Adobe forced them. Might as well just get the pain over now.
Also, the plugin was such a gaping security hellhole and something like a dozen exploits had to be fixed a month. Wouldn't surprise me if a "No more updates, you are on your own lol!" approach made their legal people too uneasy when it came to litigation exposure.
posted by sideshow at 3:51 PM on January 21, 2021 [19 favorites]
It wouldn't matter, the same shit would eventually happen. Also, the notification that the "kill" was coming happened months and months, maybe even years, ago.
My employer declared we would eventually not allow 32bit applications to run on a future OS version seven years ahead of time. Hell, the writing was written on the wall when 64bit applications were announced ~twelve year ahead of time. Users even got a pop up that said "This shit ain't going to work in the near future!" (OK I'm paraphrasing) for an entire year before the deadline.
Did all developers take heed and update their shit? No, they told their customers they had just no idea this could ever happen and that my employer was just a big meanie that just wanted to make the lives of developers miserable.
So, even if they EOL'd it for a decade, no one was going to fix anything unless Adobe forced them. Might as well just get the pain over now.
Also, the plugin was such a gaping security hellhole and something like a dozen exploits had to be fixed a month. Wouldn't surprise me if a "No more updates, you are on your own lol!" approach made their legal people too uneasy when it came to litigation exposure.
posted by sideshow at 3:51 PM on January 21, 2021 [19 favorites]
Was there a legal reason that Adobe put in a kill routine?
The Monitor prophesied that Flash must die to save the world during the Crisis on Infinite Earths crossover event.
posted by otherchaz at 3:55 PM on January 21, 2021 [37 favorites]
The Monitor prophesied that Flash must die to save the world during the Crisis on Infinite Earths crossover event.
posted by otherchaz at 3:55 PM on January 21, 2021 [37 favorites]
Also, the notification that the "kill" was coming happened months and months, maybe even years, ago.
Announced in 2017.
posted by mr_roboto at 4:03 PM on January 21, 2021 [3 favorites]
Announced in 2017.
posted by mr_roboto at 4:03 PM on January 21, 2021 [3 favorites]
Do we know that it was because of a kill switch and not because their browser got hijacked because they clicked on an advertisement?
posted by Joe in Australia at 4:24 PM on January 21, 2021
posted by Joe in Australia at 4:24 PM on January 21, 2021
I am rather surprised that the Chinese government relies so heavily on a foreign software platform. I thought it was bad enough one of our sister companies refused to update their timeclock system until December 31; at least they aren't trying to run on a pirated version!
posted by JawnBigboote at 4:48 PM on January 21, 2021
posted by JawnBigboote at 4:48 PM on January 21, 2021
Also, the notification that the "kill" was coming happened months and months, maybe even years, ago.
The deprecation announcement happened in 2017, but to my knowledge the hard-kill decision was only announced -or at least, was only fully appreciated - in early 2020. In addition, there is an enterprise support plan available offering support for Flash available until 2023, and there is a China-only long-term-support version of Flash whose terms I don’t understand.
A year to update production heavy-industrial software is... not a lot. But building it on Flash, geez. What a way to run a railroad.
posted by mhoye at 4:58 PM on January 21, 2021 [1 favorite]
The deprecation announcement happened in 2017, but to my knowledge the hard-kill decision was only announced -or at least, was only fully appreciated - in early 2020. In addition, there is an enterprise support plan available offering support for Flash available until 2023, and there is a China-only long-term-support version of Flash whose terms I don’t understand.
A year to update production heavy-industrial software is... not a lot. But building it on Flash, geez. What a way to run a railroad.
posted by mhoye at 4:58 PM on January 21, 2021 [1 favorite]
Why were they running the trains on Flash? How were they running the trains on Flash? TFA did not answer this question, and I desperately want to know.
posted by snowmentality at 5:20 PM on January 21, 2021 [12 favorites]
posted by snowmentality at 5:20 PM on January 21, 2021 [12 favorites]
Announced in 2017.
Did no one see the timeline?
...I'll get my coat.
posted by avoision at 5:27 PM on January 21, 2021 [2 favorites]
Did no one see the timeline?
...I'll get my coat.
posted by avoision at 5:27 PM on January 21, 2021 [2 favorites]
You have to use just the right combination of specific builds of Flash, Internet Explorer 6, and Java, but when you do so your computer is like Mr. Burns when all his diseases are in perfect balance and cancel each other out. Anyway, that’s the only configuration our ERP system runs in. Hop on eBay and pick up another motherboard, would you?
posted by Huffy Puffy at 6:00 PM on January 21, 2021 [12 favorites]
posted by Huffy Puffy at 6:00 PM on January 21, 2021 [12 favorites]
I loved these articles and the translation so much. And the fact that automatic updates at 10:00pm totally hosed their progress! It makes me feel better about my server issues at work, goddamn automatic updates fucking things up overnight.
From the first article: Do the IT staff of this company have obvious problems? ... Isn’t it 2021? Isn’t there something called the Internet? To solve the problem of disabling Flash, why should I install another computer and then transport it to the station for testing? Isn’t this a solution that IT staff used only 10 years ago?
My distance-learning child is obsessed with this web game he gets to play for music class called Isle of Tune. Last week he came to me and told me the website was “broken because of a flash.” He learned a sad lesson in Adobe Flash obsolescence, quite possibly on the same day as the Dalian railway workers.
posted by Maarika at 6:06 PM on January 21, 2021 [6 favorites]
From the first article: Do the IT staff of this company have obvious problems? ... Isn’t it 2021? Isn’t there something called the Internet? To solve the problem of disabling Flash, why should I install another computer and then transport it to the station for testing? Isn’t this a solution that IT staff used only 10 years ago?
My distance-learning child is obsessed with this web game he gets to play for music class called Isle of Tune. Last week he came to me and told me the website was “broken because of a flash.” He learned a sad lesson in Adobe Flash obsolescence, quite possibly on the same day as the Dalian railway workers.
posted by Maarika at 6:06 PM on January 21, 2021 [6 favorites]
Anyway, that’s the only configuration our ERP system runs in.
Not that long ago, my employer, a company that defines much of the modern web, had one of their biggest money-making products only accessible via IE 6. Many, many employees did not even have Windows machines, they had Macs or Linux laptops.
So we ran a large Citrix installation all for the sole purpose of running IE 6 which in turn ran our very popular IE6-specific web app. So you had an app, which was really a remote session, running in a remote VM, running Windows, running IE 6.
But we did eventually rewrite the whole thing from scratch because of course we did.
posted by GuyZero at 6:55 PM on January 21, 2021 [3 favorites]
Not that long ago, my employer, a company that defines much of the modern web, had one of their biggest money-making products only accessible via IE 6. Many, many employees did not even have Windows machines, they had Macs or Linux laptops.
So we ran a large Citrix installation all for the sole purpose of running IE 6 which in turn ran our very popular IE6-specific web app. So you had an app, which was really a remote session, running in a remote VM, running Windows, running IE 6.
But we did eventually rewrite the whole thing from scratch because of course we did.
posted by GuyZero at 6:55 PM on January 21, 2021 [3 favorites]
At work we changed timecard systems maybe a year or so ago. Bizarrely (IMO), we went from a janky IE6-only interface to one requiring Flash. It was only recently that they rolled out the non-Flash version; this thread makes me realize why. Save for the EOL, they would probably have been happy to let the Flash thing molder forever.
posted by Standard Orange at 9:12 PM on January 21, 2021
posted by Standard Orange at 9:12 PM on January 21, 2021
Hop on eBay and pick up another motherboard, would you?
Reminds me of the time I had to trade my computer to one of our clients because they were using an ancient Dialogic ISA card to run a mission critical IVR system and it was the only thing available that day with an ISA slot. I did end up getting a shiny new 700MHz Athlon system out of the deal, though, so I wasn't sad about it at all.
They got a pretty decent deal out of it, too, given that they were still using that K6-3 and its rigged up power switch to make it turn on again after power loss nearly a decade later.
posted by wierdo at 10:18 PM on January 21, 2021 [2 favorites]
Reminds me of the time I had to trade my computer to one of our clients because they were using an ancient Dialogic ISA card to run a mission critical IVR system and it was the only thing available that day with an ISA slot. I did end up getting a shiny new 700MHz Athlon system out of the deal, though, so I wasn't sad about it at all.
They got a pretty decent deal out of it, too, given that they were still using that K6-3 and its rigged up power switch to make it turn on again after power loss nearly a decade later.
posted by wierdo at 10:18 PM on January 21, 2021 [2 favorites]
I am rather surprised that the Chinese government relies so heavily on a foreign software platform.
Many different things to unpack here, but in my experience two key aspects in play here.
1) appearance vs. reality. The "Chinese Government" is not some monolithic hi-tech entity, it is held together with spit and glue. I doubt there is much budget for official MS licenses, or Adobe, or whatever, at the Dalian hi-speed rail offices... It comes down to some low level office manager or software dev team leader working with shit equipment on an unrealistic schedule putting in crazy hours and just expected to "get it done".
2) "Chabuduo" which would loosely translate as "close enough" or "good enough" permeates all aspects of building and manufacturing in China. You see highway overpasses just falling over, buildings looking OK for a few years until all the marble starts falling off, etc. etc. If it looks OK, and it works, and we managed to do it cheap, nobody is going to ask too many questions.
The fact that some key bit of infrastructure suddenly crashed because dead Flash does not surprise me at all.
posted by Meatbomb at 10:53 PM on January 21, 2021 [17 favorites]
Many different things to unpack here, but in my experience two key aspects in play here.
1) appearance vs. reality. The "Chinese Government" is not some monolithic hi-tech entity, it is held together with spit and glue. I doubt there is much budget for official MS licenses, or Adobe, or whatever, at the Dalian hi-speed rail offices... It comes down to some low level office manager or software dev team leader working with shit equipment on an unrealistic schedule putting in crazy hours and just expected to "get it done".
2) "Chabuduo" which would loosely translate as "close enough" or "good enough" permeates all aspects of building and manufacturing in China. You see highway overpasses just falling over, buildings looking OK for a few years until all the marble starts falling off, etc. etc. If it looks OK, and it works, and we managed to do it cheap, nobody is going to ask too many questions.
The fact that some key bit of infrastructure suddenly crashed because dead Flash does not surprise me at all.
posted by Meatbomb at 10:53 PM on January 21, 2021 [17 favorites]
Meatbomb is, alas, right on the money. I have seen software in China that has shocked me with how awful it was. Example: recently a friend had to register for some school related stuff. The website required Internet Explorer...that was it. Didn't work with anything else. Not even Microsoft Edge! It _had_ to be IE. And that honestly is not that bad for the software I have seen here.
posted by wooh at 11:20 PM on January 21, 2021 [2 favorites]
posted by wooh at 11:20 PM on January 21, 2021 [2 favorites]
The commuter train Homestar Runner was ready to go after the software "Cheat", technicians reported.
posted by benzenedream at 11:28 PM on January 21, 2021 [6 favorites]
posted by benzenedream at 11:28 PM on January 21, 2021 [6 favorites]
Tangentially relevant is the fact that Flash is not actually dead in China. Adobe has licensed Flash to a firm so that the business of Flash continues inside China but not elsewhere. Before the global end-of-life, there was already a geolock built in Flash, so that
Since the kill-date, Flash keeps on being updated in China (but I'm not sure whether those updates are the result of good-faith bug-fixes or just new ways to packaging stuff; see below for more). If you want to go there, here's the website: flash.cn
The new business model, beside "enterprise support", is a familiar one: shareware (Flash) plus drive-by downloads (some sort of game centre). With a modernized twist, if the user accepts the drive-by download, they're also endowed with the privilege of giving up tons of personal information, all enshrined in the EULA.
A recent update of the flash.cn-Flash, according to the website, removes from the Flash player the ability serve as a video player for the majority of operating systems. The new direction seems to be entirely focused on gaming and legacy intranet app support.
So from the point of view of a hypothetical IT person (who might be the nephew of the manager, FWIW), the situation wasn't that of "we've been warned about it since X years ago", but "since Adobe promised Flash would live on in some way at least in China, so what's the worst that could happen?" They knew Flash would never be dead, and they don't even have to pony up for the "legacy support". Anyway, what makes one believe that the right kind of "support" from a shady company some 1,700 km away in Chongqing would be there within 20 hours? They have the backup already, so it's just a matter of pulling the plug and restore (which didn't quite go according to the plan, but with a little bit of help from warez there's nothing in the way really).
posted by runcifex at 12:16 AM on January 22, 2021 [1 favorite]
- an installation of the "Adobe" edition of Flash, when it detected that it is running from China, would stop working
- users in China were directed to install the locally licensed version instead.
Since the kill-date, Flash keeps on being updated in China (but I'm not sure whether those updates are the result of good-faith bug-fixes or just new ways to packaging stuff; see below for more). If you want to go there, here's the website: flash.cn
The new business model, beside "enterprise support", is a familiar one: shareware (Flash) plus drive-by downloads (some sort of game centre). With a modernized twist, if the user accepts the drive-by download, they're also endowed with the privilege of giving up tons of personal information, all enshrined in the EULA.
A recent update of the flash.cn-Flash, according to the website, removes from the Flash player the ability serve as a video player for the majority of operating systems. The new direction seems to be entirely focused on gaming and legacy intranet app support.
So from the point of view of a hypothetical IT person (who might be the nephew of the manager, FWIW), the situation wasn't that of "we've been warned about it since X years ago", but "since Adobe promised Flash would live on in some way at least in China, so what's the worst that could happen?" They knew Flash would never be dead, and they don't even have to pony up for the "legacy support". Anyway, what makes one believe that the right kind of "support" from a shady company some 1,700 km away in Chongqing would be there within 20 hours? They have the backup already, so it's just a matter of pulling the plug and restore (which didn't quite go according to the plan, but with a little bit of help from warez there's nothing in the way really).
posted by runcifex at 12:16 AM on January 22, 2021 [1 favorite]
When I was working at Apple in the early 2000s, Flash was responsible for the majority of crashes on Mac OS X.
Think about that. Flash caused more crashes than all other Mac software combined.
posted by ryanrs at 1:52 AM on January 22, 2021 [3 favorites]
Think about that. Flash caused more crashes than all other Mac software combined.
posted by ryanrs at 1:52 AM on January 22, 2021 [3 favorites]
Flash, Flash, I love you, but we only have fourteen hours to save the Earth Railway!
posted by Lanark at 2:26 AM on January 22, 2021 [5 favorites]
posted by Lanark at 2:26 AM on January 22, 2021 [5 favorites]
It could be worse, apparently their bus-scheduling software is just an animate gif.
posted by snofoam at 4:34 AM on January 22, 2021 [1 favorite]
posted by snofoam at 4:34 AM on January 22, 2021 [1 favorite]
Why were they running the trains on Flash? How were they running the trains on Flash? TFA not answer this question, and I desperately want to know.
A train dispatch system is kind of like an air traffic control system but... for trains. The signals and trains would likely have been ok but the signalers would have had no idea where anything was.
posted by onya at 5:11 AM on January 22, 2021
A train dispatch system is kind of like an air traffic control system but... for trains. The signals and trains would likely have been ok but the signalers would have had no idea where anything was.
posted by onya at 5:11 AM on January 22, 2021
Why were they running the trains on Flash? How were they running the trains on Flash?
There's not much chance Flash handled any of the industrial logic, and it was almost certainly "just" the display layer, but it turns out that's a pretty important layer.
posted by mhoye at 5:48 AM on January 22, 2021 [6 favorites]
There's not much chance Flash handled any of the industrial logic, and it was almost certainly "just" the display layer, but it turns out that's a pretty important layer.
posted by mhoye at 5:48 AM on January 22, 2021 [6 favorites]
China apparently has 46 nuclear power plants. I hope they're not running flash.
posted by TrialByMedia at 7:40 AM on January 22, 2021 [5 favorites]
posted by TrialByMedia at 7:40 AM on January 22, 2021 [5 favorites]
Dear god no - nuclear power plants run on VBasic.
posted by pseudophile at 9:38 AM on January 22, 2021 [11 favorites]
posted by pseudophile at 9:38 AM on January 22, 2021 [11 favorites]
How were they running the trains on Flash?
Sometimes it isn't the main application, but instead a minor, management back-end. I have seen that happen at $WORK.
posted by wenestvedt at 10:53 AM on January 22, 2021
Sometimes it isn't the main application, but instead a minor, management back-end. I have seen that happen at $WORK.
posted by wenestvedt at 10:53 AM on January 22, 2021
Adobe was trying to position Flash as an app platform ("Flex") in the early 2000's. They built a GUI control library and a app-oriented xml format for laying out interfaces. But there wasn't really any support for building apps on the back-end, so most likely this was a Flash front-end client for the actual train controlling software.
posted by zixyer at 12:03 PM on January 22, 2021
posted by zixyer at 12:03 PM on January 22, 2021
Oh God, I had successfully repressed memories of Flex all these years.
posted by signal at 4:38 PM on January 22, 2021
posted by signal at 4:38 PM on January 22, 2021
« Older What does that click mean? | Not a three hour tour Newer »
This thread has been archived and is closed to new comments
The brief summary for other readers similarly confused: the problem was indeed the suspension of Flash. A more general Adobe update included a kill routine that actually disabled Flash on the given date, however (as opposed to what happens with most no longer supported software, which just keeps becoming clunkier and accumulating small problems as other software stops interfacing with it).
posted by eviemath at 3:25 PM on January 21, 2021