The Case of the 500-mile Email

March 2, 2019 12:33 PM

"We can't send mail farther than 500 miles from here," he repeated. "A little bit more, actually. Call it 520 miles. But no farther."

posted by Sokka shot first (42 comments total)

Fucking sendmail. Thinking about sendmail.cf is giving me hives.
posted by axiom at 12:36 PM on March 2, 2019 [10 favorites]

I agree about sendmail

But hang on a minute. It needs a full roundtrip to establish a connection, so it should only be able to connect to something 1.5 light-milliseconds away, right?
posted by aubilenon at 12:42 PM on March 2, 2019 [1 favorite]

It is of course complete B.S. at best and a very poorly diagnosed network issue at worse.

It was passed around among network admins (including myself) at the time for the same reason the mouse balls memo was. It was vaguely plausible but mostly an entertaining concept.
posted by Tell Me No Lies at 12:46 PM on March 2, 2019 [7 favorites]

There's a timeout at 500 miles
Or a timeout with a few miles more
It's the kernel that times out after those miles
Do make the packets slower
posted by RobotVoodooPower at 12:47 PM on March 2, 2019 [5 favorites]

sendmail.cf

Huh TIL what Central African Republic's TLD is and yes someone did register it.
posted by Space Coyote at 12:48 PM on March 2, 2019 [6 favorites]

During my time with sendmail.cf configuration files I'd suggest that it was actually a turing complete language and someone should (not me, omg no no no) write a C compiler in sendmail.
posted by sammyo at 1:01 PM on March 2, 2019 [4 favorites]

As someone hearing about it after the fact, this is delightful.
posted by Going To Maine at 1:47 PM on March 2, 2019

This is funny.

Is this where we share our funny problem stories? Because I have a vaguely related story. One time at a conference I was having a conversation like this - stories of astonishing problems. A colleague started telling the story of how he (a scientist) was consulting with a company that made, IIRC, beer. Beer Company was having problems with the consistency of the beer varying with time, and they didn't understand why.

At this point I said, "Hmm. Could it have something to do with tides?"

My colleague stopped short and stared at me with his mouth open. Finally he said that it took him more than a month to solve the problem, which was indeed caused indirectly by tides. The brewery had two sources of water, and how much of each source was used varied with tides. The company hadn't realized this variation but it was affecting the final product.
posted by medusa at 2:05 PM on March 2, 2019 [40 favorites]

Oh I could mail five hundred miles
But I can't mail five hundred more
And if you're a thousand miles away
You won't get any mail no more

Da da da (da da da)
Da da da (da da da)
posted by Roentgen at 2:25 PM on March 2, 2019 [36 favorites]

I actually read this whole thing. Whew! It was almost funny! I mean, I almost understood it enough for it to be funny. That's saying something.
posted by luaz at 2:45 PM on March 2, 2019 [3 favorites]

Best of the web. This is going into my "hacker lore" bookmarks along with The Story of Mel and the Graphing Calculator Story.
posted by ragtag at 2:58 PM on March 2, 2019 [8 favorites]

The FAQ pointed to at the top is also kinda funny if you’re a technical person. He responds to people’s BS detectors going off by admitting that virtually every detail in the story was pulled out of his hiney. But he’s cheerful about it, so there’s that.
posted by Tell Me No Lies at 3:02 PM on March 2, 2019 [3 favorites]

One of my favorite oooollldd stories from the mists of times long forgotten:
The Story of Mel
From the good old Jargon File: http://www.catb.org/jargon/

Also: Moof!
posted by Cris E at 3:03 PM on March 2, 2019 [2 favorites]

oh wow this story cropped up at work last week after 15 years of me not even thinking about it lol...and now here? I'm curious where my work/metafilter life intersects and bounces across the US :-)
posted by nikaspark at 3:56 PM on March 2, 2019 [2 favorites]

To me, it seems like the core of the story is that a crummy, misconfigured version of sendmail had a default timeout of 'as soon as the system got impatient', and somehow the users worked out and reported a rough correlation to distance, which seems completely plausible to me.

I can totally buy that some of these old tools did things that seemed completely insane to us: I've been trapped in vi before.
posted by Merus at 4:06 PM on March 2, 2019 [4 favorites]

bounces across the US

Nah, 500 miles would only take you to Wichita.
posted by axiom at 4:08 PM on March 2, 2019 [1 favorite]

After the filing repository database at my work crashed twice in seven days, both times caused by a wild-card search that was less than five characters, resulting in angry emails from department heads using all of the available red bold fonts telling us to NEVER, EVER, EXECUTE THAT AGAIN...

This just seems plausible
posted by lineofsight at 4:20 PM on March 2, 2019

My colleague stopped short and stared at me with his mouth open as if turned to stone. Finally he said that it took him more than a month to solve the problem, which was indeed caused indirectly by tides. The brewery had two sources of water, and how much of each source was used varied with tides. The company hadn't realized this variation but it was affecting the final product.
posted by medusa

FTFY

But seriously, how did you guess that??
posted by jamjam at 4:26 PM on March 2, 2019 [2 favorites]

I'm curious where my work/metafilter life intersects and bounces across the US

I saw this bouncing around Mastodon some months back, so I'm guessing one of The Darn Kids discovered it and made it go "viral".
posted by suetanvil at 5:12 PM on March 2, 2019 [2 favorites]

Got this stuff makes me long for the early days of the internet, when it was like a giant puzzle box instead of a tool kit to monetize data.
posted by Ickster at 5:18 PM on March 2, 2019 [11 favorites]

I know you've received me, now here's a surprise
I know that you have 'cause there's packets in my eyes
I can send for miles and miles and miles and ... whoop, shit
posted by RobotVoodooPower at 5:31 PM on March 2, 2019 [8 favorites]

Along with the already linked Story of Mel and More Magic this is one of my favourite internet folklore stories. I reference More Magic all the time: "let's leave this switch in the more magic position" "Oops, looks like you turned off the more magic switch" and no one gets it. I'm not sure whether to hope it goes viral or not.

jamjam: "But seriously, how did you guess that??"

This sort of thing happens to me about every six months or so (IE: an at least partially accurate guess from what appears to be incomplete data that seems way out there at first glance) and it's a bit of a rush; like what I imagine a runners high feels like. A guy I worked with called them Sherlock moments. My brain is just really good a correlating data in this way. EG: I once guessed a person's middle name knowing only that her first name is Victoria and her parents were British immigrants. And sometimes it is just what seems like an obscure problem is well known to me.

I'm just impressed the tech managed to get such a coherent, accurate and well specified problem and not just "the internet is down".

[Reveal the name]

Anne. Admittedly not too much of a stretch but still impressed the heck of out of her.

posted by Mitheral at 9:22 PM on March 2, 2019 [3 favorites]

I love the "I'm looking for work" post script. This is a hell of a resume.
posted by es_de_bah at 10:01 PM on March 2, 2019

I love the "I'm looking for work" post script. This is a hell of a resume.

Not if you're looking to get hired anywhere with knowledgeable networking people.
posted by Tell Me No Lies at 11:38 PM on March 2, 2019

There are lots of places where internet traffic goes over point-to-point radio links to get to small islands where the cost of laying undersea cable hasn't been worth it to anyone.

You get multipath effects on these links, because radio waves can take both the direct path between the antennas and a reflected path from one antenna, off the surface of the sea, and back to the second antenna.

So the signal quality on these links varies with both weather and the tides. The graphs are pretty cool.
posted by automatronic at 2:17 AM on March 3, 2019 [1 favorite]

c in a medium such as a cable is significantly slower than it is in a vacuum.
posted by Bovine Love at 4:29 AM on March 3, 2019 [1 favorite]

Being able to successfully hack sendmail.cf was probably one of my most impressive achievements at my old job... but so few people in my life can truly appreciate the scope of my achievement.
posted by amtho at 7:16 AM on March 3, 2019 [1 favorite]

c in a medium such as a cable is significantly slower than it is in a vacuum

A C compiler implemented in a medium such as sendmail.cf is slower still.
posted by flabdablet at 7:26 AM on March 3, 2019 [1 favorite]

Debian bug #170857: Sendmail is not the only Turing-complete MTA

Linked from that bug report: Creative abuse of Exim
posted by flabdablet at 7:30 AM on March 3, 2019

Is this where we share our funny problem stories?

It is now!

My favorite from the land of software development: we'd implemented a new system that required user registration, and QA immediately bounced it back to us with the note "this thing fails at literally the first screen. I can't even register an account." We dutifully looked into it, and were unable to reproduce the failure--all the devs were able to register an account in the QA system. We talked to QA, and watched over their shoulders, and yup, it was failing for them with an generic 500 error whenever either of them tried to register. Dev could use the same keyboard and same machine, and not get the error. Snarky remarks about "oh, so you're just checking the isQA flag and failing, huh?" went back and forth. This had now burned half the morning, and the server error logs were no help diagnosing the error--it was being trapped and rethrown as an Uncategorized SQL Exception.

I finally tripped over the answer when I was mashing the keyboard to generate gibberish input into the registration screen, and triggered a failure. Cause of error: something was concatenating a suffix onto the username, and the column it was inserting into was short enough that sometimes it overflowed and failed. The magic length limit was 6. Every developer doing testing used their first name as a username, and every dev on the team had a name <5 letters long. (Bill, Erik, Evan, etc.) . QA was done by Candice and Lindsey, who also used their first names, which were all >6 letters.
posted by Mayor West at 10:36 AM on March 3, 2019 [6 favorites]

After the filing repository database at my work crashed twice in seven days, both times caused by a wild-card search that was less than five characters

Surprisingly, the timebomb in "glob" search patterns was discovered and solved fairly recently, despite being latent for decades. It turns out that even simplistic implementations can be worst-case quadratic, but traditional implementations were worst-case exponential, leading to ready DoS attacks across a wide variety of languages and platforms. (Better algorithms are linear or even sublinear, since "glob" patterns can be reduced to regular expressions, but the up-front costs are in most cases not worth it.)
posted by sjswitzer at 11:26 AM on March 3, 2019

But seriously, how did you guess that??
posted by jamjam

It was a wild-ass guess: the first thing that popped into my head for something that would change over days-weeks in a confusing way. I just was lucky, or maybe it was a superpower moment like Mitheral describes.
posted by medusa at 11:30 AM on March 3, 2019 [1 favorite]

I cannot believe this didn't have a previously. This has been going around forever!
posted by Chrysostom at 12:18 PM on March 3, 2019

I worked for a school dept. for a while. Bored kids would remove the mouse ball(s) and play with them, never returning them, of course. I saved them from dead mice at one job for the computers at the school job. Then they just glued the fitting so they couldn't be removed (or the rollers cleaned) and I had a mason jar of mouse balls. I kind of wish I still had that.
posted by theora55 at 12:21 PM on March 3, 2019

The magic-switch story is evergreen because it's plausible. There's no way to be sure, of course, but it's good for a solid 20 minute argument between engineers of various stripes as to how it might have worked. It doesn't matter whether the thing ever happened (my guess is probably it did, and more than once; I've seen things as least as apparently-nonsensical when I did big sound reinforcement systems).

(My personal theory is that the switch body was probably grounded via the physical connection to the machine's chassis, and the switch was either faulty or had been modified to short when in the "more magic" position. Thus, flipping the switch grounded the pin inside the machine that it was connected to, which would otherwise float.)

I've seen a working system that used a very long, very delicate wire as a de facto pull-down resistor on an unused input. Swapping it for a shorter wire—because why did it have this stupid long dangly piece?—caused it to stick in a programming mode (Harvard architecture). But that wire didn't, it worked fine. It took longer than you'd expect for someone to think of that wire as more than just an idealized zero-ohm connection to ground. The problem wasn't the wire, but the tendency to assume that things in the messy analog universe can be casually and predictably modeled and their behavior understood without empirical testing.
posted by Kadin2048 at 2:54 PM on March 3, 2019 [2 favorites]

At my last job I made a little lab equipment checkout system, mostly to teach myself JavaScript. I used the Google supported Angular.js package to make it pretty.

Tested it, did a limited rollout with just my group, no problem. Did a larger rollout and found out that fully half of SQA was getting corrupted dialog boxes. I noticed that many of the screenshots(*) featured IE so I thought maybe it was a version issue, but nothing seemed to fit.

The “Aha!" moment came when I went to organize a conference call and found that everyone with the problem had good availability at 4:30 PM on Sunday afternoon (PST). Also known as 8:30 AM in Shanghai.

Turns out I was loading Angular.js from google.com, and guess who’s Great Firewall blocks that host?


* The nice thing about giving SQA buggy software is that you get really really thorough bug reports
posted by Tell Me No Lies at 4:42 PM on March 3, 2019 [1 favorite]

That should have been "8:30 AM Monday morning in Shanghai", lest anyone think we were working weekends at that point.
posted by Tell Me No Lies at 5:08 PM on March 3, 2019

The problem wasn't the wire, but the tendency to assume that things in the messy analog universe can be casually and predictably modeled and their behavior understood without empirical testing

I firmly believe that the Law of Leaky Abstractions needs to be formally taught fairly early in every curriculum, STEM and humanities both.
posted by flabdablet at 6:10 PM on March 3, 2019 [2 favorites]

We had an audio broadcast system with an on-air indicator that would light up when the system was active. After a while it would randomly start turning on and off instead of staying illuminated.

After a few months I noticed that it seemed to get glitchy when whoever was driving the console stood up or sat down. The problem turned out to be static shocks in the oil column of the adjustable chair sending out small bursts of RFI that was messing with the sign’s electronics. Nobody believed me until they tried standing up and sitting down themselves.
posted by the duck by the oboe at 10:46 PM on March 3, 2019 [4 favorites]

I had the pleasure of hearing this story direct from Trey at a LinuxConf or other around the turn of the millennium. He told it quite well, but as I recall there were also some details at the time about how they were one of the infamous "sunsites".

If you can't remember that, well it was a series of high-capacity FTP mirrors of important Unix software. I don't remember if that got them funding somehow, but I always remember going to some sunsite.****.edu or other to download my Slackware floppy images.
posted by rum-soaked space hobo at 3:48 AM on March 5, 2019

Oh yes, and there was also a bit of detail about a configuration management problem that had landed an empty sendmail.cf on the professor's machine.
posted by rum-soaked space hobo at 3:51 AM on March 5, 2019

Back in college I had a work-study job in Admissions doing whatever random things needed to be done: tours, bulk mailings, dounut runs, whatever. This was a long time ago, the early 80s, when half the office work was still done on IBM Selectric typewriters and the other half was a parade of mail merges from Word Perfect.

One day the staff was having a lot of trouble getting a mailing printed. I was fine with that because I could sit around and gab until it was ready for me to stuff into envelopes. This was something that was done every day, so it was kind of curious that they couldn't get it to work, but one field was just not filling. Eventually I wandered over to take a look.

Green on black, 25x80 DOS computing was a much simpler world than what we have now. You could more or less see everything involved in a problem* and it was just a matter of spotting your issue and getting on with your work. In this case they had Name, Addr1, Addr2, City, etc data fields and Addr1 was not filling. We re-pulled the data, we ran a doc from yesterday, we had Pat retype the thing. It was weird. Then I had her re-type it again and watched closely. Ah! We fixed the invisible typo and they were off to the races.

What was the typo? Well our staff was still using typewriters every day, and one habit that old secretaries frequently had was using a lower case L instead of the number 1. Most machines before the Selectrics didn't have a 1/! key, and when IBM added it many typists didn't bother using it as Courier was very forgiving in that regard. PCs, on the other hand, noticed the difference, and they were quite particular about not using an L in your variable names.

* Unless you were farting around with extended or expanded memory. Blerg.
posted by Cris E at 7:31 PM on March 6, 2019