Engineer and astro-futurist Chris Lewicki wrote an essay this week - that would not be out of place in our Who, Me? archives - about a testing mistake that nearly transformed half a billion dollars worth of Mars Rover into spacecraft scrap.

Lewicki posted the tale to his personal site, describing work on the Spirit Mars Rover at NASA's Jet Propulsion Laboratory (JPL) in Pasadena, California more than two decades ago.

It was February 2003, and a mere two weeks remained before the trundlebot would be packed up for shipment to Florida for its launch.

The spacecraft needed to be checked and rechecked for the last time on Earth. Everything had to be verified as being in perfect working order; the rovers were, at the time, among the most complex spacecraft ever built. They also represented nearly a billion dollars of NASA investment.

"No pressure," Lewicki noted.

He was into his unofficial second shift, having already logged 12 hours that Wednesday, and was tasked with verifying the integrity of the motors in the Rock Abrasion Tool (RAT) attached to the end of Spirit's robotic arm.

After each round of testing, taking apart the motor wasn't practical, but it was possible to check for any problems by looking at the electrical performance. The team could disconnect and plug the motor into an external power supply. An exponential decrease in electrical current during spin-up was good. Blips were bad.

Lewicki had performed the test, which required a device called a "break-out-box," many times. He wrote: "My various roles on the project had given me the experience to decipher the maze of diagrams mapping the 10,000 pin-to-pin connections that made everything on the spacecraft work, and my responsibility in writing the instructions on how to connect and control all the motors on the rovers made me the obvious choice for this test campaign."

So, just another day in the office. Or cleanroom. He had a cabling expert deal with sorting out the connectors and inserting test equipment. A quick pulse to a reference motor verified all was good. Spirit's RAT motor was jumpered-in and Lewicki was good to go.

He explained: "To get the clearest signal and reveal the smallest of imperfections from the motor, the standard procedure is to give it as much power as it wants. This makes it vitally important to send the inrush of electrons to the right place.

"A wrong connection could do blue-smoke-releasing catastrophic damage."

The pulse was sent to the motor. The result - as usual - was immediate. However, it was not the result the team was expecting. It didn't look like a broken motor. It didn't look like a working motor. It looked like... something else.

"My mind raced for explanations and in what seemed like an instant, arrived at the most likely explanation. My eyes followed the wires from our breakout box on the test cart to the spacecraft, and the reason for the unfamiliar signal landed like a dagger through my heart.

"All that power we just released did not go into the RAT-Revolve motor. Due to a mistake I had made with the break-out-box, it went the other direction on the connector interface, sending a surge of electricity straight into the spacecraft, instead of the motor.

"Ooooohhhh ssshhhhiiiiitttt."

At around the same time, all telemetry from the rover ceased. It appeared that Lewicki had killed Spirit. With only two weeks remaining until launch, the options to fix the seemingly stricken trundlebot were few and far between.

There was, however, a ray of hope. Sure, the telemetry had stopped around the time Lewicki had blasted the poor thing with electricity where it was not wanted. But maybe, just maybe, the surge had gone somewhere that could handle the expected extra energy. Perhaps all that had done was cause a temporary glitch. Perhaps, just perhaps, a power cycle would clear the problem.

And so the spacecraft was booted... and nothing happened. No telemetry. No heartbeat. Nothing. Spirit did indeed seem dead.

Lewicki went home, and we can't imagine how his night went. He said: "What I do remember is the feeling of emotional devastation that followed me home where I recounted the story to my wife. I was convinced I would lose my job in the morning and space exploration history would attach my name to a particular chapter of infamy."

Back at JPL in the morning, the previous night's events were pieced together as the team attempted to recover the situation. It all seemed hopeless until a step involving a multimeter was remembered. Lewicki had needed the device for his test, and one was liberated from its apparent role of monitoring bus voltage. He had carefully disconnected it and then gone on with his trundlebot termination.

However...

"The monitoring multimeter I disconnected was actually completing the circuit that powered the spacecraft's ground test telemetry. I inadvertently disabled the connection the instant I removed the leads."

The device was restored. The spacecraft was powered up…

Lewicki recalled: "It worked. There was a collective gasp as the telemetry flickered back to life - Spirit was not dead after all!"

Naturally, there were discussions on whether the rover would be OK to launch. History tells us it was, and the mission proved to be a tremendous success.

As for Lewicki, he wasn't fired. In fact, the tests continued with him leading them - after all, he had learned some important lessons and was arguably the last person to ever make such a mistake again.

Lewicki went on to have a stellar career, which includes becoming flight director for the mission. In the book Roving Mars, Steve Squyres described Lewicki, then 29, as very young to be "Flight" during the landing on Mars. But goes on to say, "But the guy is a major hotshot... scary smart and poised beyond his years. He's the right man for the job tonight." ®

https://www.theregister.com//2023/11/29/jpl_engineer_spirit_mars_rover/