...like Xerxes whipping the sea for swallowing his shitty bridge.

November 16, 2015 6:49 PM

Marcin Wichary writes (among other things) articles about crazy, complicated computer bugs and what they say about the world. Come to read about how a bug in Google's famous Pac-Man doodle led to all manner of chaos, stay to learn how a confluence of history, typography, and OS weirdness broke the Polish S (previously), or how a fluke of CSS unearthed a decades-old font.

(Wichary on the blue previously, with a wonderful investigation of Medium's office building)

posted by Itaxpica (37 comments total)

These are generally awesome stories! The "fluke of CSS" one is a bit weird to me, though. As someone who routinely uses font-pickers because I can't seem to make up my mind what to use in various applications I use every day? It wouldn't have occurred to me that other people didn't realize System was still there. Actually, the original System, even older, is still there, as Fixedsys, and I literally only just now realized that it's no longer the Notepad default. Though now I'm squinting at Wikipedia... has that really been true since XP? Maybe I'm just elderly.

But I love, overall, seeing how other people solve problems like this when weird things happen.
posted by Sequence at 7:05 PM on November 16, 2015 [1 favorite]

Unintended Consequences: It's the law.
posted by Bringer Tom at 7:08 PM on November 16, 2015 [3 favorites]

The System CSS one is super-weird, because it implies they never tested the new css on a Windows machine before pushing to production.
posted by timdiggerm at 7:13 PM on November 16, 2015 [8 favorites]

His explanation of how the Pac Man bug totally isn't his fault is total nonsense. What kind of web designer (working for Google no less) implements something with autoplaying audio on the most visited site on the internet?
posted by dilaudid at 7:49 PM on November 16, 2015 [11 favorites]

Finding bugs is fun.

One device I worked on had a bug when you pressed two buttons 1 to 2 milliseconds apart (but not at the same time, and not > 2ms apart). We found it using high precision relays. The developers said "That will never happen!" Then we found a surgeon who could reproduce it about 50% of the time. It got fixed.

Another surgeon finished a pig surgery with a prototype device, and casually poured a pint of blood out of the device. QA guy just smiled and nodded like that was supposed to happen, and there were emergency redesigns for fluid ingress for the next week.

Had a tin whisker growing into a fiber optic path in humid climates. Everything would be fine for a few weeks, and then slowly the part would just stop working as the whisker grew out to block the light. After a co-worker figured out what was happening, the short term fix was "Just go give the box a good whack". This would break off the whiskers and everything would work again for a while.

The testers I now work with have a subtle little "time to first bug" contest every brand new release, and can be heard chuckling evilly the whole time. Usual first bug: about a minute and a half. Catching them all during development is hard!
posted by underflow at 8:00 PM on November 16, 2015 [6 favorites]

Probably my all-time favorite computer glitch story: Real Life Tron on an Apple IIgs (previously)
posted by Rhaomi at 8:09 PM on November 16, 2015 [15 favorites]

Rhaomi, that is a fantastic story!
posted by nushustu at 8:43 PM on November 16, 2015 [1 favorite]

> never tested ... on a Windows machine

Google dumped Internet Explorer 6 in 2010 (finally), and the rest of the industry followed suit. Without the requirement to test on Internet Explorer, there was no need for Windows anymore, which meant website designers were free to OS X and website developers that wanted to could also switch to OS X, or Linux.

While casting aspersions on Medium.com's professionalism might be more fun (screenshot tools exist that could have exposed the problem before it went into production), it's more interesting to point out that there are large swaths of the high-tech industry in which Microsoft (of old) is no longer relevant. Microsoft (of new, under Nadella) is making a strong effort to catch up though.

My favorite 'bug' of lore, is of the 500 mile email.
posted by fragmede at 9:12 PM on November 16, 2015 [13 favorites]

I hear three simultaneous games of Pac-Man in this coffee shop. I kind of love you, Google.

♥
posted by JHarris at 9:29 PM on November 16, 2015

That "Real life Tron" bug is exactly the sort of thing that would happen to me once in a while coding on the C64. On a 1mhz, 64K machine, you don't have a surplus of either cycles of memory, so you cut every corner you can. So you use the display as double-duty, both as the graphic representation shown to the player and as the playfield data that your code operates on. If you want to see if there's ground underneath a platformer guy's feet, you actually look at the location in screen memory corresponding to underfoot and see if there's a ground character there.

To a computer, all memory is really the same, the only difference is how code or hardware interprets it. If the code doesn't check to make sure Platformer Guy doesn't fall out of screen memory (hey, that's extra cycles!), and there's no line of wall characters to stop him (a whole line of the display, wasted!), then he might very well trample over important memory locations, "falling" through your program's code until he finds the magic number or messes up the code so badly that things get WEIRD.
posted by JHarris at 9:37 PM on November 16, 2015 [4 favorites]

I honestly would have simply assumed medium was going for a retro 8-bit vibe and not even noticed that it was the old school capital-S System font.
posted by GuyZero at 9:52 PM on November 16, 2015

Finding bugs is hell. I found a bug where two different teams had differing opinions on how to implement a super-specific part of the AAC spec and it's really hard to reproduce so I can't even tell if it's an encoder or a decoder issue.

YAY
posted by GuyZero at 9:54 PM on November 16, 2015 [1 favorite]

Hah, I think I actually experienced that font bug in an article linked from here, but nobody knew what I was talking about when I mentioned it. I am vindicated! Vindicated, I tell you!
posted by Joe in Australia at 9:55 PM on November 16, 2015 [1 favorite]

The last link's comparison of the two fonts named "San Francisco", one from 1984 and one from 2015, is the most concise picture of why so many people are upset about what's happened to their city.
posted by Harvey Kilobit at 9:55 PM on November 16, 2015 [14 favorites]

Which is the right thing to do.

Well, no, evidently not, for reasons that it is the purpose of the article to describe.
posted by JHarris at 9:59 PM on November 16, 2015

The System CSS one is super-weird, because it implies they never tested the new css on a Windows machine before pushing to production.

Hmm. Apparently, I inadvertently discovered the System font issue back in September. I thought it was just a bug with Windows, and it crapped out a really old and ugly version of Arial. Who knew?
posted by a lungful of dragon at 10:21 PM on November 16, 2015

Also:
The other fun part is that, back in 2010, I also had to reintroduce a bug from the original Pac-Man code… but that’s a whole different article.

I have never worked at Google, but I'm pretty sure I know the bug of which he speaks, because I'm intimately familiar with Pac-Man's workings both from obsessive reading of The Pac-Man Dossier (which, oh no, seems to be down right now!) and developing my own clone/expansion of Pac-Man, Octropolis, that to ensure perfect compatibility with the original game's AI the doodle would have to duplicate.

Namely, there is a bug in the AI for both the Pink and Blue ghosts that, whenever Pac-Man faces up, changes the location they target to a location somewhat to the left of where it should be. Don Hodges explains the bug here.
posted by JHarris at 10:29 PM on November 16, 2015 [2 favorites]

Catching them all during development is hard!

Yeah, not so much hard, as impossible. The best you can hope for is to find the bugs that the vast majority of users will run into before a product ships.

If you ever think that you've found and fixed all the bugs before a product has shipped, you may want to look into getting better QA people.*

*I am a better QA person.
posted by el io at 10:36 PM on November 16, 2015 [2 favorites]

autoplay audio embedded in webpages is a bad idea! who knew? I mean in 2010 it had only been a definite annoyance for, what, twelve years? fifteen?
posted by mwhybark at 12:53 AM on November 17, 2015 [3 favorites]

I had a job once that was supposed to involve converting a creaky old system written in MUMPS (anathema upon it; yea, even unto the end of time) into something more modern like Java with an Oracle back end.

My second day, I was watching over the shoulder of one of the old-hand MUMPS programmers, who couldn't figure out why the program he wrote was only processing every other record. I, having never seen MUMPS before, figured out in about two minutes that he was accidentally double-incrementing a counter.

It...didn't bode well for the rest of the job.
posted by Mr. Bad Example at 1:23 AM on November 17, 2015

When I was an editor at a gaming magazine in Poland, Marcin wrote us a cover CD menu and a bunch of other utils we regularly used. He was in high school then. I was extremely pleased to discover a few years back that he works for Google - but not surprised.
posted by hat_eater at 1:54 AM on November 17, 2015 [2 favorites]

"system" is not a generic CSS font-family, so when you specify "system" you're asking the browser to use a font called "system" if it exists. Don't do that if you don't want to use that particular font.
posted by Pruitt-Igoe at 3:38 AM on November 17, 2015 [3 favorites]

Google dumped Internet Explorer 6 in 2010 (finally), and the rest of the industry followed suit. Without the requirement to test on Internet Explorer, there was no need for Windows anymore, which meant website designers were free to OS X and website developers that wanted to could also switch to OS X, or Linux.

But Microsoft didn't discontinue IE, they continued to develop it. What about IE 7, 8, 9, 10, 11...? None of those browsers render pages in the same way as webkit does.

There is millions of institutional users using IE. I've been one of them, and when I was I really appreciated websites that worked. There is still a need to test on IE if you target any of those users.

I agree that nobody should need to test on IE6 anymore, but for a website as large as Medium not testing on any IE version is just laziness. Windows still makes up almost 90% of the desktop internet.
posted by leo_r at 4:06 AM on November 17, 2015 [2 favorites]

What about IE 7, 8, 9, 10, 11...? None of those browsers render pages in the same way as webkit does.

Ain't that the truth. Yes they should have tested it. As a web developer though, I entirely understand anyone not firing up the Windows VM to try out every little fix on Microsoft's still-broken mess.
posted by iotic at 4:39 AM on November 17, 2015

On the other hand, Apple defining an in-house CSS font-family rule, for which the assumed non-proprietary version isn't compatible with Windows/IE, is a bigger oversight.
posted by iotic at 4:42 AM on November 17, 2015 [2 favorites]

The problem was assuming the generic version of a proprietary value is safe or at least currently-meaningless.

The whole point of proprietary namespacing is to restrict a rule or value to a controllable subset. It's not on Apple or any other web browser maker to assume that the generic version of their proprietary value is meaningful somewhere else. Especially since, per Craig Hockenberry's research, it's undocumented and possibly meant to be a private setting rather than a publicly usable one; like the private-namespaced .SFNSDisplay font name, it's unsupported and subject to unannounced changes.
posted by ardgedee at 5:00 AM on November 17, 2015

It's not on Apple or any other web browser maker to assume that the generic version of their proprietary value is meaningful somewhere else.

Perhaps not. But one might consider it good manners.
posted by iotic at 5:05 AM on November 17, 2015

While casting aspersions on Medium.com's professionalism might be more fun (screenshot tools exist that could have exposed the problem before it went into production), it's more interesting to point out that there are large swaths of the high-tech industry in which Microsoft (of old) is no longer relevant.

There's still no excuse for not testing on all platforms. Yeah, no one wants to test on IE, I certainly don't, but it's there and hundreds of millions of people still use it.
posted by octothorpe at 5:07 AM on November 17, 2015

Apple was already observing proper protocol regarding the creation of "system" as a generic value. It's really weird to hold them accountable for the unintended consequence from some developer genericizing an unpublished privately namespaced value and putting it into production untested.
posted by ardgedee at 5:16 AM on November 17, 2015

I'm a fan of Marcin's, he's got remarkably diverse talents. I was one of the lucky ones to see his screening of Alternatywy 4, a 1980s Polish sit-com. He'd spent a year subtitling the whole TV show and editing the 8 hours down to a 2.5 hour coherent thing we could watch in a movie theater. A phenomenal amount of work well done. He even coded a simulation of the Polish TV broadcast clock. As you do.
posted by Nelson at 8:08 AM on November 17, 2015 [3 favorites]

From TFA:

And, since –apple– is a typical vendor prefix for an experimental feature, we included a more future-proof system as the first entry


Right. So you "future-proofed" an experimental setting and then didn't test it because it worked fine on your laptop.
So fascinating!

And that's after you deployed a delayed audio autoplay to millions of people and blame the users for their plug-ins when it does exactly what you intended?



Google dumped Internet Explorer 6 in 2010 (finally), and the rest of the industry followed suit. Without the requirement to test on Internet Explorer, there was no need for Windows anymore, which meant website designers were free to OS X and website developers that wanted to could also switch to OS X, or Linux.


I just opened paint.net on a pretty much Vanilla windows 10 install and "system" is in the middle of the list. Its not "unearthed", its right fucking there.

This is just shitty coding and shitty QA, that it comes from someone who has pushed prominent pages just shows how entrenched and common these attitudes are.
posted by lkc at 9:54 AM on November 17, 2015 [4 favorites]

and my favorite bug is ARMM.
posted by lkc at 10:46 AM on November 17, 2015

IE's combined marketshare's fallen to ~50% from an estimated 90+% back in the IE6 days (according to wikipedia), so compatibility with IE 7-11 is just not the requirement it once was.

As far as "System" still existing in Windows 10 and not needing to be "unearthed"... I reiterate, where am I going to get a Windows?

It should be noted that Microsoft themselves can't do any better. Their License Advisor only works in IE, and plain does not work in Firefox or Chrome.
posted by fragmede at 12:02 PM on November 17, 2015

On the other hand, Apple defining an in-house CSS font-family rule, for which the assumed non-proprietary version isn't compatible with Windows/IE, is a bigger oversight.

Microsoft continuing to package Windows with a generically-named font that should no longer be in use is perhaps a bigger problem.
posted by a lungful of dragon at 12:11 PM on November 17, 2015

I dunno, I've long known about System and kind of liked it. It's interesting that it still exists, and it's very small. Microsoft is throwing out Windows traditions left and right, I'm in favor of keeping whatever we can keep.
posted by JHarris at 1:17 PM on November 17, 2015

The "System" bitmap font is VGASYS.FON. Well, it's that if you're using the default font scaling (96 DPI, referred to as "Smaller (100%)" in the control panel.) If you choose one of the other options (Medium/125%/120 DPI or Larger/150%/144 DPI) then you get 8514SYS.FON as your System, which is slightly larger. I think it's 12 pixels tall compared to 10 for the VGA resolution one. Funny story, that 8514 in the filename refers to the IBM 8514/A graphics adapter, originally a Micro Channel card for the IBM PS/2. Released in 1987, I think it was one of the first adapter/monitor combinations to support the 1024x768 resolution (at a nice headache-inducing, soul-crushing 43.5 Hz interlaced refresh, but still.)

So the idea was that there were two versions of these fonts, one for normal VGA resolutions and one for higher resolutions which meant higher effective DPI and hence the need for the font to be bigger to remain legible. If you check your %WINDIR%\Fonts directory of your modern Windows machine you'll still see good old 8514SYS.FON, still unchanged from the mid 1980s. (If you're using Explorer, you might have to configure it to show files with Hidden and/or System attributes, which it likes to hide from you by default, although Explorer probably won't let you view the fonts directory as a regular directory, so copy *.fon into a temporary location using something other than Explorer and then open that location with Explorer if you want to be able to see them as normal files.) You'd probably see it if you listed the directory of one of the installation floppies of Windows 2.0, except it would have probably been named 8514SYS.FO_ since it would have been cab-compressed. I find it amusing that a tiny remnant of an ancient computer system that's long dead still lives on in modern Windows.

BTW, the cousins of System are Fixedsys (VGAFIX.FON and 8514FIX.FON) and Terminal (something like VGAOEM.FON and 8514OEM.FON but things get murky here because of all the different code pages.)
posted by Rhomboid at 3:21 PM on November 17, 2015 [4 favorites]

It seems really weird to call this a bug. I mean, you made a change without even trying it in one of the more popular browsers... What did you think would happen? BrowserStack is cheap, ya know.

And this is why everything I do has to go through about 100 layers of QA to get anywhere near a customer....
posted by miyabo at 6:50 AM on November 18, 2015