Surviving an In-Flight Anomaly: What Happened on Ingenuity’s Sixth Flight
May 27, 2021 7:57 PM Subscribe
"Telemetry from Flight Six shows that the first 150-meter leg of the flight went off without a hitch. But toward the end of that leg, something happened: Ingenuity began adjusting its velocity and tilting back and forth in an oscillating pattern. This behavior persisted throughout the rest of the flight." Written by Håvard Grip, Ingenuity Mars Helicopter Chief Pilot
"Prior to landing safely, onboard sensors indicated the rotorcraft encountered roll and pitch excursions of more than 20 degrees, large control inputs, and spikes in power consumption." [Emphasis mine for clarity and reassurance.]
"Prior to landing safely, onboard sensors indicated the rotorcraft encountered roll and pitch excursions of more than 20 degrees, large control inputs, and spikes in power consumption." [Emphasis mine for clarity and reassurance.]
Very disappointed to learn that it was not aliens.
posted by Joe in Australia at 8:14 PM on May 27, 2021 [3 favorites]
posted by Joe in Australia at 8:14 PM on May 27, 2021 [3 favorites]
imagining those little coffee-drinking bug guys from the Men in Black movies joyriding on it and going yee-haw when it started to oscillate.
posted by Halloween Jack at 9:31 PM on May 27, 2021 [8 favorites]
posted by Halloween Jack at 9:31 PM on May 27, 2021 [8 favorites]
Nintendo cartridges were tested against well specified production hardware which had never traveled more than a few thousand miles on a planet with a functioning magnetosphere. These folks built a system robust enough to safely land a helicopter on Mars even after things went pear shaped, because that's what engineers learn these days. Cut the kids some slack; they're doing great.
posted by phooky at 10:52 PM on May 27, 2021 [44 favorites]
posted by phooky at 10:52 PM on May 27, 2021 [44 favorites]
That's a fair point. The Perseverance rover weighs the same as two grand pianos and had to be lowered from a crane hovering on four rocket jets. The software to do that must be incredible. It's mind-blowing that it's worked the two times they've tried it. Enough to give you very high expectations of what they can do, enough to make you wonder how bugs got into the easy part.
posted by adept256 at 12:13 AM on May 28, 2021 [4 favorites]
posted by adept256 at 12:13 AM on May 28, 2021 [4 favorites]
enough to make you wonder how bugs got into the easy part.
That's the secret, though. When your machine is millions of miles from home, there is no easy part.
posted by tclark at 12:36 AM on May 28, 2021 [30 favorites]
That's the secret, though. When your machine is millions of miles from home, there is no easy part.
posted by tclark at 12:36 AM on May 28, 2021 [30 favorites]
It's not called "Ingenuity" for nothing!
(That flight video looks an awful lot like Luke Skywalker in his X-Wing making the final approach on the Death Star; maybe Ingenuity just turned off the targeting computer and used the Force.)
posted by chavenet at 12:46 AM on May 28, 2021 [4 favorites]
(That flight video looks an awful lot like Luke Skywalker in his X-Wing making the final approach on the Death Star; maybe Ingenuity just turned off the targeting computer and used the Force.)
posted by chavenet at 12:46 AM on May 28, 2021 [4 favorites]
ample 'stability margin'
sounds comforting to me
Sounds like my walk to the bathroom when I first wake up.
posted by srboisvert at 4:25 AM on May 28, 2021 [5 favorites]
sounds comforting to me
Sounds like my walk to the bathroom when I first wake up.
posted by srboisvert at 4:25 AM on May 28, 2021 [5 favorites]
Optical assist would certainly decrease the reliability of that manoeuvre at that time for me as well.
posted by flabdablet at 5:52 AM on May 28, 2021
posted by flabdablet at 5:52 AM on May 28, 2021
We use this same sort of guidance and navigation system on lots and lots of vehicles on and around Earth, too - anywhere that GPS is unavailable/unreliable, basically. Inertial systems tightly coupled with "imaging" (I'm using this term loosely) can be incredibly accurate.
But yes, as the article mentions, timing is critical, and it's incredibly important to know if an update arrives late or not at all. The systems I work on have the luxury of being large enough to afford space for multiply redundant, highly accurate oscillators to provide very precise timing to make sure all the different parts are in sync. (Some of our labs are smaller than the installed environment, so we have to do things like hide huge coils of wire under the floor to make sure the cable runs are the same length, otherwise our timing inputs will be off.)
It's basic dead reckoning updated for highly advanced sensors and computers, and I think that's pretty neat.
posted by backseatpilot at 5:59 AM on May 28, 2021 [7 favorites]
But yes, as the article mentions, timing is critical, and it's incredibly important to know if an update arrives late or not at all. The systems I work on have the luxury of being large enough to afford space for multiply redundant, highly accurate oscillators to provide very precise timing to make sure all the different parts are in sync. (Some of our labs are smaller than the installed environment, so we have to do things like hide huge coils of wire under the floor to make sure the cable runs are the same length, otherwise our timing inputs will be off.)
It's basic dead reckoning updated for highly advanced sensors and computers, and I think that's pretty neat.
posted by backseatpilot at 5:59 AM on May 28, 2021 [7 favorites]
I'm sort of surprised that the optical flow can manage to get timestamps and optical data separated like that. I assume the IMU used the short term data to go "I don't _care_ what the flow sensor says, we're exceeding our flight tolerances, return to level flight and try again."
Still, pretty interesting, considering what sort of hardware they're working on and how unmanaged the flights have to be, right? I mean, we fly drones on earth that seem 100% IMU managed, but then I remember, right, they have a human at the controls constantly managing the flight if the IMU tumbles for a bit.
posted by Kyol at 5:59 AM on May 28, 2021
Still, pretty interesting, considering what sort of hardware they're working on and how unmanaged the flights have to be, right? I mean, we fly drones on earth that seem 100% IMU managed, but then I remember, right, they have a human at the controls constantly managing the flight if the IMU tumbles for a bit.
posted by Kyol at 5:59 AM on May 28, 2021
I'm sort of surprised that the optical flow can manage to get timestamps and optical data separated like that.
I have no idea what their implementation looks like, but it's pretty common not to use literal timestamps but frame counters. So, nav systems powers on, starts counting once it senses takeoff, and then each subsystem is in agreement that every x milliseconds a new frame starts. Since it doesn't use optical for takeoff or landing, the avionics computer expects camera data at some predetermined frame, when the camera starts dumping images that are indexed in some way. Each frame is subdivided into message windows where subsystems report in order - a frame may last a total of, say, 300 ms, and you get 5 ms of header information (including the frame counter), then another 10 ms of health and welfare data, 50 ms of IMU data, etc.
The error could have been introduced in a couple of ways. (Again, not knowing their implementation but familiar with others) The camera probably uses a timing pulse as a "shutter release" and won't do anything until it receives that pulse. If it misses a pulse, it misses taking an image, so all subsequent images are indexed off by one. It's also possible that it took the image at the right time, indexed it correctly, but it missed its transmission window during a frame for some reason (probably less likely, since the data would just be thrown out). Or, even more unlikely but still possible could be radiation-induced error and the error correction mechanisms in place failed for whatever reason.
posted by backseatpilot at 6:14 AM on May 28, 2021 [9 favorites]
I have no idea what their implementation looks like, but it's pretty common not to use literal timestamps but frame counters. So, nav systems powers on, starts counting once it senses takeoff, and then each subsystem is in agreement that every x milliseconds a new frame starts. Since it doesn't use optical for takeoff or landing, the avionics computer expects camera data at some predetermined frame, when the camera starts dumping images that are indexed in some way. Each frame is subdivided into message windows where subsystems report in order - a frame may last a total of, say, 300 ms, and you get 5 ms of header information (including the frame counter), then another 10 ms of health and welfare data, 50 ms of IMU data, etc.
The error could have been introduced in a couple of ways. (Again, not knowing their implementation but familiar with others) The camera probably uses a timing pulse as a "shutter release" and won't do anything until it receives that pulse. If it misses a pulse, it misses taking an image, so all subsequent images are indexed off by one. It's also possible that it took the image at the right time, indexed it correctly, but it missed its transmission window during a frame for some reason (probably less likely, since the data would just be thrown out). Or, even more unlikely but still possible could be radiation-induced error and the error correction mechanisms in place failed for whatever reason.
posted by backseatpilot at 6:14 AM on May 28, 2021 [9 favorites]
The Action Lab recently did a small video on the issues of flying a helicopter in a near-vacuum. If you consider the kinds of forces and mechanics that have to work juuuust right to get the thing off the ground, it's pretty much a miracle of engineering.
As an aside, I wonder if Ingenuity has a Jesus nut?
posted by JustSayNoDawg at 7:19 AM on May 28, 2021 [1 favorite]
As an aside, I wonder if Ingenuity has a Jesus nut?
posted by JustSayNoDawg at 7:19 AM on May 28, 2021 [1 favorite]
We use this same sort of guidance and navigation system on lots and lots of vehicles on and around Earth, too
It's in every optical desktop mouse, for a start.
posted by flabdablet at 2:24 PM on May 28, 2021 [1 favorite]
It's in every optical desktop mouse, for a start.
posted by flabdablet at 2:24 PM on May 28, 2021 [1 favorite]
Looks like a sophomore's PID with bad parameters in their first control theory course, but in spaaaaace
posted by MengerSponge at 7:47 PM on May 28, 2021 [1 favorite]
posted by MengerSponge at 7:47 PM on May 28, 2021 [1 favorite]
It's not called "Ingenuity" for nothing!
And I thought it got it’s name from the quote about early helicopters, "The helicopter symbolizes the victory of ingenuity over common sense.”
posted by jmauro at 9:51 AM on May 29, 2021 [1 favorite]
And I thought it got it’s name from the quote about early helicopters, "The helicopter symbolizes the victory of ingenuity over common sense.”
posted by jmauro at 9:51 AM on May 29, 2021 [1 favorite]
« Older A map of the unseen unknown aka dark matter | Surviving IDEO Newer »
This thread has been archived and is closed to new comments
I remember when a Nintendo cartridge went through every test but throwing it in a river to see if it floats, because there's no possibility of an update and if it doesn't work then it costs the company millions in recalls. Now beta testing bleeds into the deployment part of the SDLC and that's what engineers learn these days.
posted by adept256 at 8:12 PM on May 27, 2021 [3 favorites]