Metal whiskers: more dangerous than cute

May 19, 2020 12:27 PM

On May 21, 1998, the Los Angeles reported on the Galaxy IV satellite that suffered an on-orbit failure three days prior, which resulted in a massive telecommunications disruption on Earth. Around 45 million pagers went out of service that day, among other communication outages (L.A. times mentions radio listeners and gasoline buyers). The failure was attributed to tin whiskers (The Brown Daily Herald), a topic on which NASA has extensive documentation. [Previously: Lead-Free Solder - friend or foe?]
posted by filthy light thief (22 comments total)

I know that day as The Day Everyone's Pager at Canyon High School in Santa Clarita, California stopped working.

As a high school junior, it was a dark day indeed.
posted by sideshow at 12:34 PM on May 19, 2020 [3 favorites]

I fully thought that "tin whiskers" was the name of a cat in outer space. Then I clicked through.

Disappointed to learn that this was in reference to something a bit more scientific.
posted by Fizz at 12:36 PM on May 19, 2020 [3 favorites]

While Tin Whiskers may haunt satellites, its cousin Zinc Whiskers is fooking up shit in raised-floor data centers. "During a one-month period, a NASAdata center experienced at least 18 cat-astrophic power supply failures in newly installed mass memory storage devices." (Zinc Whiskers: Hidden Cause of Equipment Failure, 2004 IEEE publication, PDF).

I posted this for the interesting bit of history, the mildly amusing (and cat-like) name, and the fact that NASA's dedicated URL has a space in it. Also, the vintage-feeling web design.
posted by filthy light thief at 12:54 PM on May 19, 2020 [3 favorites]

Well, it certainly pricked up my ears, flt. :) I spent several years supporting the industry transition to Pb-free, and am still in the same field, even now looking at new solders on the horizon. Tin whiskers (and zinc!) are still a concern and they will be for a long time.
posted by blurker at 1:00 PM on May 19, 2020 [1 favorite]

Cool stuff. It’s wild that this is still an issue and still a matter of scholarly investigation. I kept reading, waiting for a paragraph like, “Manufacturers now coat copper wires with x.” C’mon, science! :-)
posted by Don.Kinsayder at 1:54 PM on May 19, 2020 [1 favorite]

Ha! I should get “C’mon, science,” tattooed on my face. :)

We’re trying, I swear! If only we didn’t have to worry about cost, processing, and manufacturability!
posted by blurker at 2:32 PM on May 19, 2020 [1 favorite]

I remember when a sysadmin I worked with called me crazy for worrying about Single Event Upsets, but who's laughing now, Steve???
posted by mikelieman at 2:32 PM on May 19, 2020 [4 favorites]

Around 45 million pagers went out of service that day
Buying weed was impossible.
posted by TrialByMedia at 3:21 PM on May 19, 2020 [4 favorites]

There is no way of determining the real cause of the Galaxy satellite failure because the satellite was never examined. Tin whiskers became a sort of catch-all excuse for unexplained failures in that era.

But even if you accept the tin whisker explanation, there was likely a contributing manufacturing defect. A lot of military spec electronics has a conformal coating of a varnish-like polymer applied to the entire board and components. The conformal coating eliminates most tin whisker problems. Examination of the manufacturer's conformal coating procedures showed that they were inadequate for providing full coverage.

High reliability stuff like satellites have pretty much solved this problem by refining their conformal coating processes.

On the list of component failure modes, tin whiskers is way down the list.
posted by JackFlash at 3:28 PM on May 19, 2020 [4 favorites]

This was a number of years before I began in public radio, but my boss was one of those guys that loved to tell war stories from the old days. Apparently when Galaxy IV failed, PRSS's solution was to create a phone tree of sorts with ISDN units. My station had a T1 for audio connections to NPR, so they used that to receive audio, and then used their 3 Zephyrs to feed other stations, each of whom in turn fed other stations. With, of course, encoding and decoding delay stacking up along the way. I have always felt bad for the on-air hosts who had to adjust their clock timing because of some nonsense up in space.
posted by god hates math at 3:43 PM on May 19, 2020 [3 favorites]

My Galaxy IV story:

I was in Dallas, TX on a business trip the day Galaxy IV failed. I worked in IT for a major retailer. That satellite carried only only pager traffic, but network traffic for adopters of TCP/IP over Very Small Aperture Terminal (VSAT) dishes.

I was part of a team reviewing stores and uncovering issues and areas ripe for improvement.

I walked into the first store that morning. Someone on my team said, "Hey - he's with IT."
A cashier replied, "Well tell Mr. IT that our credit cards aren't going through."

I walked around back, logged into the server running the store and checked network connectivity. There was none, so I called our Network Operations Center. And that's how I learned our bird was gone.

And that's why I spent the rest of the trip visiting stores, climbing on the roofs of commercial buildings, re-pointing dishes to a new satellite, and writing howto directions for others involving the purchase of a compass from a local store.

We got all of our stores back online in 3 days, which frankly shocked me. Those involved in the effort got tee shirts that said, "I survived Galaxy IV." I still have that shirt.
posted by grimjeer at 4:49 PM on May 19, 2020 [17 favorites]

I know it's likely to shave off a few units of measurement off my lifespan, but I'm glad my father stocked up with crazy amounts of lead solder, because I find the lead-free stuff harder to work with and now I'm having to do it with SMT components and ARGH.
posted by sonascope at 5:33 PM on May 19, 2020

Mmm..lead.

Interestingly, the risk of leaded solder is minimal to the hobbyist, assuming they aren't the sort who is happy to breathe the obvious flux fumes and has at least some ventilation as a result. The bigger issue is lead finding its way into the environment from e-waste.
posted by wierdo at 9:30 PM on May 19, 2020

I'm glad my father stocked up with crazy amounts of lead solder

It's not like it's carbon tet or something; you can still buy 60/40 tin/lead electronics solder at retail. You just can't sell the result in some markets.

MG Chemicals 60/40 63/37

Kester

WYCTIN

All sorts of Chinese brands
posted by Mitheral at 9:37 PM on May 19, 2020

Just yesterday I learned the term "stickybeak" in an otherwise serious technical document (PDF) about solder flux.
posted by clawsoon at 1:00 AM on May 20, 2020

Tin is just pretending to be a non-eldritch material. Look at tin cry.
posted by away for regrooving at 1:13 AM on May 20, 2020 [2 favorites]

I was going to talk a bit about how Galaxy IV failing and programmers being done with C-band home dishes combined to make a few thousand people unnecessarily sad, but it seems petty after the story about manually repointing VSAT dishes to get credit card transactions working again.
posted by wierdo at 1:36 AM on May 20, 2020

I wondered what the Sun was doing around the time of the Galaxy IV satellite failure, and found that May of 1998 had the greatest solar activity of that 11 year cycle:

A Week of Solar Blasts:
The Space Weather Event of May 1998

Aftermath:
Earth-dwellers probably thought the space storm had completely blown over. Indeed, the clouds of plasma had passed, and their immediate effects had died down; the longer-term effects, however, had just begun. Two rings of plasma called the Van Allen radiation belts normally lie trapped around the Earth between the orbits of high-flying geosynchronous satellites and the low-orbiting Space Shuttle. When the shock wave associated with the May 4 storm smashed into the radiation belts, the electrons trapped there were energized to new heights and were spun around the Earth to new locations. Satellite data showed that a new radiation belt had formed between the normal two by the end of May 4 and did not subside until the middle of June.
That the high-energy electrons of the radiation belts are known as "killer electrons" does not bode well for satellites flying near the belts. During the period of disturbed conditions that May, the Equator-S satellite failed; the Polar satellite suffered blackouts; four of Motorola's Iridium satellites were lost. The most famous failure, however, was that of the Galaxy IV communications satellite, handler of some 90 percent of U.S. pager transmissions and several television and radio feeds. At 6:00 PM EDT on May 19, Galaxy IV's onboard control system and backup switch failed, proving to at least 40 million pager customers across the United States the fragility of satellite communications. Over the next few days, PanAmSat, Galaxy IV's owner, scrambled to transfer signals to other satellites and fix the communications blackout.
Although PanAmSat dismissed any solar causes for the satellite outage, blaming instead circuit-damaging "tin whiskers", some scientists disagree. An October 1998 article in the newspaper Eos stated boldly that "killer electrons" flooded the satellite and damaged its electronics. Since high-energy electron levels were far above normal for at least three weeks, the article stated, Galaxy IV slowly fried to death. The true cause can never be determined, but the concept suggested is striking -- extended periods of space weather such as the week of activity in early May 1998 can create highly abnormal conditions for satellites and inconvenience millions of people at one blow.[my emphasis]

I also wonder about insurance issues; a solar storm would seem to be the archetype for 'an act of God', but a component failure due to "tin whiskers" might well have been covered.
posted by jamjam at 1:57 AM on May 20, 2020 [3 favorites]

Still lead-free here. Yes, it's a bloody nightmare to rework without a dedicated hot-air station. I try to move more slowly and check twice before soldering. I'm pleased to say I build a recent project (a single board DEC PDP-8 with weird 80s silicon) lead free, without rework and almost without additional flux.

At work I teach soldering to all ages, and it's all leaded there. The big safety thing there is wash you hands after soldering and before eating. Thankfully I'm working with the older than “don't chew the pretty solder wire” crowd. The cheap leaded that comes with the very cheap kits we use (hey, we're a charity, we have no money; our better iron is the Gao Jie 905D, still not fab) isn't great. My brother-in-law (an avionics engineer) gave me a huge roll of US-made Pb-Sn that makes shiny perfect joints unfairly easily.

The big issue with cheap irons and leaded solder is that they're really hard to keep cold enough not to burn out solder pads. The Handskit ones I linked to are knockoffs of Hakko's cheapest temperature-controlled unit, and they're clearly meant to use SnAgCu.

The solar storm vs tin whiskers insurance angle is an interesting one. The lead industry is not above doing shitty things to keep itself in business.
posted by scruss at 6:33 AM on May 20, 2020 [1 favorite]

My Galaxy IV story: Chicago, the offices of a financial software company (one that used to boast of being "First in Windows").

That satellite carried not only pager traffic, not only tv and radio signals, not only TCP/IP/VSAT for retails, but the satellite distribution of both S&P Comstock and PCQuote financial data.

Guess which two financial data providers were the core suppliers of data to our two-year-old server farm? And guess how we were getting that data?

Now, the end-users of our product were a short-tempered bunch at the best of times - i.e. late 90's internet cutting out between our servers and the end-users. So our phones were ringing off the hook, and no one was really happy that their trading for the day was impossible because of a satellite outage.

One client actually asked a co-worker of mine how we were gonna fix it. My answer, "Are you gonna buy us a space shuttle?" was not used.
posted by Mutant Lobsters from Riverhead at 2:39 PM on May 20, 2020

I'm betting I lost a few IQ points from hobby electronics in the 70s, 80s. And lead sinkers. My dad showed me how to bite the lead ball so it stuck to the fishing line. Oh, and pouring balls for black powder rifles. Lead!
posted by j_curiouser at 7:58 PM on May 20, 2020

Interesting Somehow, I'd never heard about this event. The document archive is great!

I'm lucky, I think, to work in an industry and country where we can still use lead solder for everything I do. (One-off research equipment, with appropriate ventilation and training.) On the other hand, the RoHS rules mean we can also buy cheap, mass-produced silver-coated PCBs which are absolutely perfect for wirebonds and didn't exist until fairly recently here. You can hand-tin them with lead-tin solder if you need them to superconduct. It's ugly, but it works and it's a lot easier than removing old-fashioned tinning from bond pads or shelling out for boards made with exotic metals.

Our subfield seems to have a five year time-constant for forgetting that purple plague exists, which means I'm usually the paranoid weirdo who insists on making processes harder than they need to be, possibly without cause. But, at least we don't have to worry much about tin whiskers.
posted by eotvos at 3:42 PM on May 21, 2020