I've Made a Huge Mistake
November 28, 2023 11:07 AM Subscribe
Chris Lewicki recounts a story about how he almost killed a half-billion-dollar Mars rover. Turns out cables are hard.
He’s the last person on Earth who would make this mistake again
That's the kind of boss everyone should be lucky enough to have.
posted by They sucked his brains out! at 11:50 AM on November 28, 2023 [13 favorites]
That's the kind of boss everyone should be lucky enough to have.
posted by They sucked his brains out! at 11:50 AM on November 28, 2023 [13 favorites]
You can also read it as a threat to put him on the next rocket if he does it again.
posted by echo target at 11:54 AM on November 28, 2023 [23 favorites]
posted by echo target at 11:54 AM on November 28, 2023 [23 favorites]
A piece of tape inscribed with the ancient runes DNFW and stuck to the multimeter might have given the author pause. But since this is JPL, it would probably have to be made from titanium and cost five figures
posted by scruss at 1:00 PM on November 28, 2023 [1 favorite]
posted by scruss at 1:00 PM on November 28, 2023 [1 favorite]
Yeah, you'd expect something that was performing a long term metering role to have had some additional molly guards on it. I assume it was measuring current feeding the rover, so without it in line the rover was only getting some other parasitic feed?
posted by Kyol at 1:15 PM on November 28, 2023
posted by Kyol at 1:15 PM on November 28, 2023
Yeah, you'd expect something that was performing a long term metering role to have had some additional molly guards on it.
I take it you haven't been in a lab in a while? Anything we're doing that's ad hoc is held together by Kapton tape and signs of various legibility saying "DON'T TOUCH" which are never removed so no one can tell which ones are still fresh. We have benches collapsing under the weight of "DON'T TOUCH" signs. I've had to run tests before (set up by other people) where I was carefully told that if you plug the two wires in one way, the test works, and if you plug them in the other way, the entire thing catches on fire.
I, too, use the question about failure during job interviews. I want to see that someone can learn from their mistakes. Don't do what I heard recently - a candidate came in and told me about a task that had no written procedures and he nearly killed someone due to it. I asked him what he learned. "Don't... do that again?" Nothing else? "mmm... nope!" Right, ok, have a nice day.
posted by backseatpilot at 1:23 PM on November 28, 2023 [12 favorites]
I take it you haven't been in a lab in a while? Anything we're doing that's ad hoc is held together by Kapton tape and signs of various legibility saying "DON'T TOUCH" which are never removed so no one can tell which ones are still fresh. We have benches collapsing under the weight of "DON'T TOUCH" signs. I've had to run tests before (set up by other people) where I was carefully told that if you plug the two wires in one way, the test works, and if you plug them in the other way, the entire thing catches on fire.
I, too, use the question about failure during job interviews. I want to see that someone can learn from their mistakes. Don't do what I heard recently - a candidate came in and told me about a task that had no written procedures and he nearly killed someone due to it. I asked him what he learned. "Don't... do that again?" Nothing else? "mmm... nope!" Right, ok, have a nice day.
posted by backseatpilot at 1:23 PM on November 28, 2023 [12 favorites]
Yeah, what struck me was not so much the mistake made but the one before it, having an essential bit of test gear without note or knowledge of its function. And that telemetry must have gone down a lot longer than "two seconds" before the test was run.
Fluke multimeters aren't cheap, especially as the ones in that lab hopefully are "space grade" (ie, not what you can find on ebay or mcmaster) but surely the lab must overstock them for just these scenarios. Your $1M a day lab can't go down because you're short a (I'm guessing) $20,000 multimeter. Having to pull one from another job?
posted by maxwelton at 1:25 PM on November 28, 2023 [2 favorites]
Fluke multimeters aren't cheap, especially as the ones in that lab hopefully are "space grade" (ie, not what you can find on ebay or mcmaster) but surely the lab must overstock them for just these scenarios. Your $1M a day lab can't go down because you're short a (I'm guessing) $20,000 multimeter. Having to pull one from another job?
posted by maxwelton at 1:25 PM on November 28, 2023 [2 favorites]
Back in the ‘70’s, I worked at a small company near Santa Barbara that made TCXO’s, temperature compensated crystal controlled oscillators. I was an electronics technician making a little above minimum wage. My job was to test, fix, and align the little box. One day my boss came in and told me I was to do a special little box that was to go up in a satellite. All of a sudden, all the steps and procedures had to be fully documented on paper. And the general pressure not to fuck it up went sky high, if you pardon the pun. The idea that what you are working on will go somewhere far away and unreachable and cost a ton of money makes things a lot different. This sort of suggests that such care might help in more mundane things, but it costs a lot more. The satellite went up in the sky and worked fine thanks in part to my minuscule contributions.
posted by njohnson23 at 1:29 PM on November 28, 2023 [9 favorites]
posted by njohnson23 at 1:29 PM on November 28, 2023 [9 favorites]
Your $1M a day lab can't go down because you're short a (I'm guessing) $20,000 multimeter. Having to pull one from another job?
I do not work in a space exploration context but every single day someone at my job jeopardizes tens of millions of dollars in order to save ~200 bucks here and there, so, zero surprise frankly.
posted by We put our faith in Blast Hardcheese at 1:32 PM on November 28, 2023 [17 favorites]
I do not work in a space exploration context but every single day someone at my job jeopardizes tens of millions of dollars in order to save ~200 bucks here and there, so, zero surprise frankly.
posted by We put our faith in Blast Hardcheese at 1:32 PM on November 28, 2023 [17 favorites]
Fluke multimeters aren't cheap, especially as the ones in that lab hopefully are "space grade" (ie, not what you can find on ebay or mcmaster) but surely the lab must overstock them for just these scenarios. Your $1M a day lab can't go down because you're short a (I'm guessing) $20,000 multimeter.
the meter in question was a model 87 III, their current successors, the 87 V cost around $500. they are the handheld DMM of choice everywhere I've ever worked. I'm not sure I would have used them in that particular role since they are battery powered, but I've put together (many)wonky setups before.
Even the Keysight 3458A, the best DMM ever built is actually under 20K.
posted by Dr. Twist at 2:03 PM on November 28, 2023 [6 favorites]
the meter in question was a model 87 III, their current successors, the 87 V cost around $500. they are the handheld DMM of choice everywhere I've ever worked. I'm not sure I would have used them in that particular role since they are battery powered, but I've put together (many)wonky setups before.
Even the Keysight 3458A, the best DMM ever built is actually under 20K.
posted by Dr. Twist at 2:03 PM on November 28, 2023 [6 favorites]
MetaFilter: if you plug them in the other way, the entire thing catches on fire
posted by hippybear at 2:10 PM on November 28, 2023 [8 favorites]
posted by hippybear at 2:10 PM on November 28, 2023 [8 favorites]
Even the factory I worked in as a teenager, back when you still smoked inside, safety gear was optional and you could turn up to work still drunk from the night before and the boss wouldn't even blink, had an effective system of signs to prevent others from fucking with a machine you had set up. You'd reckon NASA could sort something out.
posted by deadwax at 2:42 PM on November 28, 2023 [6 favorites]
posted by deadwax at 2:42 PM on November 28, 2023 [6 favorites]
I'm curious how a DMM measures voltage while inline.
I thought maybe he meant to say he was measuring current which you typically do with the DMM in-circuit and yanking the meter would indeed shut the circuit down but, lo and behold in that photograph, the DMM is set up to measure voltage. Perhaps the breakout box has a voltage divider somewhere inside there and yanking the probes would break the circuit?
posted by JoeZydeco at 3:06 PM on November 28, 2023 [2 favorites]
I thought maybe he meant to say he was measuring current which you typically do with the DMM in-circuit and yanking the meter would indeed shut the circuit down but, lo and behold in that photograph, the DMM is set up to measure voltage. Perhaps the breakout box has a voltage divider somewhere inside there and yanking the probes would break the circuit?
posted by JoeZydeco at 3:06 PM on November 28, 2023 [2 favorites]
Your $1M a day lab can't go down because you're short a (I'm guessing) $20,000 multimeter. Having to pull one from another job?
If you're interested in how the sausage gets made, here it is:
JPL is an FFRDC run by California Institute of Technology. Its sponsor is NASA. While JPL certainly has equipment of its own (that is, owned either by JPL itself or CalTech), it may also be the custodian of certain Government Furnished Property (GFP), which is owned by NASA. JPL has a responsibility to maintain the GFP, and depending on the contract, not "cross the streams" as it were.
I haven't worked at JPL, but I have worked at other FFRDCs and non-profits. The FFRDC status does allow for certain contracts that, say, private sector would not be allowed to bid on, but other contracts are competed publicly. JPL does not build rovers "on spec," NASA directs them to build them based on either internal "contracts" from the science people or from other research institutions who then pay NASA to develop vehicles for them, who subcontract that work to JPL.
This is all to say that the multimeter mentioned in the article very likely was not owned by JPL, but rather either by NASA or more specifically the contract that paid for the rover. This usually means that the tool in question cannot be used for any other purpose and is nominally returned to the owner (again, NASA) at the conclusion of the project. The further implications of this are that, yes, there may not have been other multimeters around to pull from stores because they were all similarly earmarked.
Have I run into issues before where I see several perfectly functioning pieces of equipment sitting on a bench but I can't touch them because they belong to another project funded by the same customer I'm working for? Yes I have. Is it wasteful to have to go buy another one? Arguably. But them's the rules when it comes to government contracting.
posted by backseatpilot at 3:15 PM on November 28, 2023 [10 favorites]
If you're interested in how the sausage gets made, here it is:
JPL is an FFRDC run by California Institute of Technology. Its sponsor is NASA. While JPL certainly has equipment of its own (that is, owned either by JPL itself or CalTech), it may also be the custodian of certain Government Furnished Property (GFP), which is owned by NASA. JPL has a responsibility to maintain the GFP, and depending on the contract, not "cross the streams" as it were.
I haven't worked at JPL, but I have worked at other FFRDCs and non-profits. The FFRDC status does allow for certain contracts that, say, private sector would not be allowed to bid on, but other contracts are competed publicly. JPL does not build rovers "on spec," NASA directs them to build them based on either internal "contracts" from the science people or from other research institutions who then pay NASA to develop vehicles for them, who subcontract that work to JPL.
This is all to say that the multimeter mentioned in the article very likely was not owned by JPL, but rather either by NASA or more specifically the contract that paid for the rover. This usually means that the tool in question cannot be used for any other purpose and is nominally returned to the owner (again, NASA) at the conclusion of the project. The further implications of this are that, yes, there may not have been other multimeters around to pull from stores because they were all similarly earmarked.
Have I run into issues before where I see several perfectly functioning pieces of equipment sitting on a bench but I can't touch them because they belong to another project funded by the same customer I'm working for? Yes I have. Is it wasteful to have to go buy another one? Arguably. But them's the rules when it comes to government contracting.
posted by backseatpilot at 3:15 PM on November 28, 2023 [10 favorites]
Have I run into issues before where I see several perfectly functioning pieces of equipment sitting on a bench but I can't touch them because they belong to another project funded by the same customer I'm working for? Yes I have. Is it wasteful to have to go buy another one? Arguably. But them's the rules when it comes to government contracting.
This is a thing I saw in my time working in public schools, which is nothing like NASA or other high tech things, but can also receive funds from various government programs. And in the elementary school I worked in a lifetime and a half ago, I saw instances where resources that were funded by one entity would sit unused while another funding entity within the same granting system for programs in that school was going to buy identical resources because you can't cross the funding streams or any material results therefrom.
posted by hippybear at 3:21 PM on November 28, 2023 [1 favorite]
This is a thing I saw in my time working in public schools, which is nothing like NASA or other high tech things, but can also receive funds from various government programs. And in the elementary school I worked in a lifetime and a half ago, I saw instances where resources that were funded by one entity would sit unused while another funding entity within the same granting system for programs in that school was going to buy identical resources because you can't cross the funding streams or any material results therefrom.
posted by hippybear at 3:21 PM on November 28, 2023 [1 favorite]
Yeah, you'd expect something that was performing a long term metering role to have had some additional molly guards on it
Yeah, it seems this isn't a thing around science professionals. Maybe they assume everyone is too smart to not realise that a random multi-meter is a critical component in a multi-billion dollar piece of equipment. In a more prosaic setting, the multimeter would be wrapped in masking/electrical/duct tape (depending on context) and be marked with something like 'touch this and I'll cut your fucking hand off'.
posted by dg at 3:26 PM on November 28, 2023
Yeah, it seems this isn't a thing around science professionals. Maybe they assume everyone is too smart to not realise that a random multi-meter is a critical component in a multi-billion dollar piece of equipment. In a more prosaic setting, the multimeter would be wrapped in masking/electrical/duct tape (depending on context) and be marked with something like 'touch this and I'll cut your fucking hand off'.
posted by dg at 3:26 PM on November 28, 2023
In a more prosaic setting, the multimeter would be wrapped in masking/electrical/duct tape (depending on context) and be marked with something like 'touch this and I'll cut your fucking hand off'.
The post about signage at Antarctica from a short while back is an excellent example of how actual signage needs to be absolute and to the point and unable to be mistaken by anyone who encounters it for the first time.
posted by hippybear at 3:32 PM on November 28, 2023 [1 favorite]
The post about signage at Antarctica from a short while back is an excellent example of how actual signage needs to be absolute and to the point and unable to be mistaken by anyone who encounters it for the first time.
posted by hippybear at 3:32 PM on November 28, 2023 [1 favorite]
The Antarctic solution would be to put a sign on the door saying "Two Mars rovers and a multimeter"
posted by tigrrrlily at 3:36 PM on November 28, 2023
posted by tigrrrlily at 3:36 PM on November 28, 2023
It’s not going to be a $20k multimeter. It’s a Fluke, they’re all built to the same standard (mil spec something or other) and they’re all suitable for just about any kind of production, development, or field use. And they keep models in production for a really long time so when the Fluke you use in your half billion dollar thing goes bad you can buy a new one. That’s why they still exist even though you can get something with the same specs but no name for $20.
posted by doomsey at 4:07 PM on November 28, 2023 [3 favorites]
posted by doomsey at 4:07 PM on November 28, 2023 [3 favorites]
This usually means that the tool in question cannot be used for any other purpose and is nominally returned to the owner (again, NASA) at the conclusion of the project.
In the late 70s, I interviewed at Martin Marietta, and I was shown tons of high end computer equipment that was going unused because they're acquired it for a certain contact that had ended. It really turned me off to the idea of working there.
posted by CheeseDigestsAll at 4:25 PM on November 28, 2023
In the late 70s, I interviewed at Martin Marietta, and I was shown tons of high end computer equipment that was going unused because they're acquired it for a certain contact that had ended. It really turned me off to the idea of working there.
posted by CheeseDigestsAll at 4:25 PM on November 28, 2023
I had to track down the previous thread:South Pole signage. To find this gem. Its a hand written note on the wall/device that says "OPEN CAREFULLY High Air Pressure."
posted by zenon at 6:17 PM on November 28, 2023 [3 favorites]
posted by zenon at 6:17 PM on November 28, 2023 [3 favorites]
I thought maybe he meant to say he was measuring current which you typically do with the DMM in-circuit and yanking the meter would indeed shut the circuit down but, lo and behold in that photograph, the DMM is set up to measure voltage.
The caption on the photo says it was taken in January 2003. The introduction to the article indicates that the event took place in February 2003, so I assume that the photo was representative, but does not show the actual incident.
posted by yuwtze at 7:24 PM on November 28, 2023
The caption on the photo says it was taken in January 2003. The introduction to the article indicates that the event took place in February 2003, so I assume that the photo was representative, but does not show the actual incident.
posted by yuwtze at 7:24 PM on November 28, 2023
My dad had an early-ish DMM (it had a "hold" button" and a temperature probe setting) which he won in a trade show raffle. Because he frequently took it to work it had a huge "NOT CALIBRATED: DO NOT USE FOR OFFICIAL TESTING" sticker affixed to the side.
posted by RonButNotStupid at 5:28 AM on November 29, 2023
posted by RonButNotStupid at 5:28 AM on November 29, 2023
In the late 70s, I interviewed at Martin Marietta, and I was shown tons of high end computer equipment that was going unused because they're acquired it for a certain contact that had ended. It really turned me off to the idea of working there.
So the flip side to the argument of "this is all going to waste" is that... What if you need to solve a problem ASAP? Like in this rover case, JPL builds mockups, ground spares, and other hardware-in-the-loop test and evaluation units at various levels. If the rover's on Mars and the ops team wants to find out right now why one of the motor voltages is acting weird, it's good to know that all the tooling and instruments are sitting around where you expect them to be and haven't wandered off to be used on someone else's project. Or in something like Lockheed Martin's case, when the government comes back after the contract is over and says "BTW, could you build us another hundred aircraft?" (this does happen fairly frequently) then the tooling is right there.
This is one of the major challenges of cost estimating, especially for these very long-duration contracts. It might take a decade or more for JPL to develop, build, and test a new space vehicle. If I asked you today to price out a Mars rover that's going to be delivered in 2035, how many multimeters do you think you would budget for?
posted by backseatpilot at 7:53 AM on November 29, 2023 [2 favorites]
So the flip side to the argument of "this is all going to waste" is that... What if you need to solve a problem ASAP? Like in this rover case, JPL builds mockups, ground spares, and other hardware-in-the-loop test and evaluation units at various levels. If the rover's on Mars and the ops team wants to find out right now why one of the motor voltages is acting weird, it's good to know that all the tooling and instruments are sitting around where you expect them to be and haven't wandered off to be used on someone else's project. Or in something like Lockheed Martin's case, when the government comes back after the contract is over and says "BTW, could you build us another hundred aircraft?" (this does happen fairly frequently) then the tooling is right there.
This is one of the major challenges of cost estimating, especially for these very long-duration contracts. It might take a decade or more for JPL to develop, build, and test a new space vehicle. If I asked you today to price out a Mars rover that's going to be delivered in 2035, how many multimeters do you think you would budget for?
posted by backseatpilot at 7:53 AM on November 29, 2023 [2 favorites]
I spent 35 years @ NASA KSC and am overly familiar with the entire scenario. I have two points:
- People who say "Failure is not an option" are too risk adverse which, at NASA at least, is understandable when a failure can have you on front pages and in front on Congress. You don't get anywhere if failure isn't an option.
- One of my favorite stories happened a couple of weeks after I started at NASA. One of the smartest men I have ever known asked me to help finish fabricating what at the time was a new chassis (VME) that you slide various boards into to perform a function. It wouldn't work and we spent 45-60 minutes trying to figure out why. Turns out he hadn't connected the power pins to the supply. I learned two valuable lessons that day that I still carry. 1) The smartest people can make the silliest mistakes, and 2) always make sure the O-N/O-F-F switch is in the O-N position.
posted by jeporter99 at 8:58 AM on November 29, 2023 [2 favorites]
- People who say "Failure is not an option" are too risk adverse which, at NASA at least, is understandable when a failure can have you on front pages and in front on Congress. You don't get anywhere if failure isn't an option.
- One of my favorite stories happened a couple of weeks after I started at NASA. One of the smartest men I have ever known asked me to help finish fabricating what at the time was a new chassis (VME) that you slide various boards into to perform a function. It wouldn't work and we spent 45-60 minutes trying to figure out why. Turns out he hadn't connected the power pins to the supply. I learned two valuable lessons that day that I still carry. 1) The smartest people can make the silliest mistakes, and 2) always make sure the O-N/O-F-F switch is in the O-N position.
posted by jeporter99 at 8:58 AM on November 29, 2023 [2 favorites]
always make sure the O-N/O-F-F switch is in the O-N position.
I call this "is it plugged in". Check that the thing you are working on is connected to the problem, and that really basic systems are actually connected, even if this seems stupid.
Car won't start? Check if battery is connected. Check that you have the correct key.
99/100 times this is stupid and a waste of time just like you expect. 1/100 times it saves you HOURS of amazingly frustrating problem solving.
posted by NotAYakk at 11:08 AM on November 29, 2023
I call this "is it plugged in". Check that the thing you are working on is connected to the problem, and that really basic systems are actually connected, even if this seems stupid.
Car won't start? Check if battery is connected. Check that you have the correct key.
99/100 times this is stupid and a waste of time just like you expect. 1/100 times it saves you HOURS of amazingly frustrating problem solving.
posted by NotAYakk at 11:08 AM on November 29, 2023
Remember this feeling the next time you have to sign-off that something is OK.
Late one night my now-wife-then-girlfriend, Flight Hardware, Do Not Touch's phone rings; she sits up next to me, takes out her laptop and starts typing.
The voice on the phone explains how "Yeah we're about to send out this batch of commands over X-band, and we need someone to say yay or nay"
I just tried to make myself as non-distracting as possible.
They sent the commands, the spacecraft didn't brick itself, and we got to go back to sleep.
posted by tigrrrlily at 11:32 AM on November 29, 2023 [3 favorites]
oh wow that's a great user name. I hope it is in reference to this sort of sign/situation.
yes yes you know it: Pence pictured touching a metal doodad that is clearly labeled
yes yes you know it: Pence pictured touching a metal doodad that is clearly labeled
Critical Space Flight Hardwareposted by zenon at 1:43 PM on November 29, 2023 [1 favorite]
Do Not Touch
Mod note: This fabulous post has been added to the Best Of blog!
posted by Brandon Blatcher (staff) at 9:19 AM on December 5, 2023
posted by Brandon Blatcher (staff) at 9:19 AM on December 5, 2023
« Older Ceasefire now | If you sit by the riverside, you see a culmination Newer »
This thread has been archived and is closed to new comments
I love that line, but this one in particular jumped out at me: "I still remember the shock when Project Manager Pete delivered the decision and the follow-on news: ‘These tests will continue. And Chris will continue to lead them as we have paid for his education. He’s the last person on Earth who would make this mistake again.’ "
This is the kind of thing you can only hear at an exceedingly mature and professional organization.
posted by mhoye at 11:37 AM on November 28, 2023 [59 favorites]