Authors: Rod Pyle
In the final analysis, it appears that a programming error—a human failing—caused the ship to turn into an improper orientation
with regard to the sun, overheating the onboard batteries and causing them to fail. With no power (as the solar panels were not correctly aimed at the sun) and dying batteries, the craft had little time to reacquire a signal with JPL. And while its programming likely resulted in its searching for a proper orientation to talk to its masters, this software did not include a way to keep itself safe from the rays of the sun, destructively hot even this far away.
4
While the craft died what was ultimately deemed an unnecessary death, the mission was a vast success. MGS operated four times beyond its primary mission, and for almost ten years, longer than any other Mars orbiter. It provided evidence of huge masses of flowing water on Mars (heretofore thought unlikely, despite the visual evidence), and found deposits of minerals that further indicated massive water flows in the past, helping to select future Mars rover landing sites.
The oversights that resulted in its demise also triggered a thorough review of the safety features loaded onboard future spacecraft, and this is how places such as JPL refine their approach to space exploration. While all contingencies are considered as thoroughly as possible in advance, it is by failures that we learn to avoid such future debacles. And this paved the way for even more successes ahead.
B
ob Brooks is an icon. He wears his trademark dark T-shirt and flannel overshirt, with his long hair tied back in a pony-tail. The hair is not a conceit, nor is it an homage to the 1960s. He says that it's just easier to see what you are doing that way.
When not at JPL guiding spacecraft to their fates, he can often be found in a darkened room in his Van Nuys home, fiddling with high-powered lasers and other dark technical arts. He is an inveterate tinkerer, a thirty-three-year veteran of the lab, and a passionate voice for planetary exploration.
“I went to USC, got my bachelor's in astronomy, and then a friend arranged for me to get an interview with his boss at JPL…. [N]ext thing I knew, I had a job working on Voyager. I stayed on that mission for several years. It was really fascinating, traveling to the outer planets. Then there was a new project called Mars Observer, and I got very interested in that. It was the beginning of a long-term involvement with Mars.
“Well, we lost MO. There was a lot of concern about that. So the lab wanted to try to do a better job next time, and the director at NASA headquarters sold [NASA] on the idea that we would actually have two projects: one would be Mars Global Surveyor, which would be basically to do the same things that MO was supposed to do, and the other was something called MSOP, or Mars Surveyor Operations Project, which was not a flight project, but a way to start development of a multi-mission operations center at
the laboratory, which would result in cost savings and higher efficiency.
“So that's what we did, we built MGS up, and in the same time frame we also built MSOP, in order to operate it. As we built MSOP, more and more projects afterward got interested in using the things that we offered, not just the software, but the people. So at one point MSOP was flying several projects at once, and doing it very inexpensively. It was a great way to manage spaceflight.
“I was one of the founders of MSOP, and was asked to come up with a good operational strategy that would save money and work in a multi-mission environment. Working with two colleagues from Mars Observer, we had developed an entity called the ASP, or automated sequence processor, which is still in use almost twenty years later. This was a way for remote science and spacecraft team users to send commands in from wherever they were, anywhere in the world. These commands would be automatically processed through the ground system, and if they were good, then they would be prepped for radiation to the spacecraft. On the other hand, if there were problems with them, then the ASP software would send back a nasty-gram to the requester saying ‘You have to fix this!’ This was something that was available 24/7 and required no personnel sitting and validating commands, it just ran all by itself. That was a big deal at the time, and everybody loved it. It saved anywhere from 1.5 to 5 million bucks per project, per year. Ultimately, we could have four or five projects running at once. Previously this would have been prohibitively expensive, but this allowed the projects to come in and very efficiently run their operations.”
1
MSOP was a revolution in unmanned spacecraft command and control. But “JPL'ers” live to explore, and that means flying specific missions. And Mars was calling across the void to Brooks: “But with the MSOP operating well, I wanted to go back to Mars and help to redeem ourselves from what happened with MO. So the MGS mission was that chance. One of the unique things that
we decided to do with it was called
aerobraking
, which was very new. We had done a short dip into aerobraking at Venus with Magellan, but for MGS we had to conduct this risky maneuver for months to skinny-down the capture orbit, which was forty-four hours long, to our two-hour mapping orbit.
“Now, aerobraking is a very complex and, potentially, personnel-intense procedure. But this automated sequence processor that we had designed made it possible to do this cost effectively. So while the aerobraking took several months, we picked one person who was the best at it, and he built up an input file for the ASP software which generally automated the procedure. Once we made sure that it was okay, the commands could come through in the middle of the night, the ASP would grab it, run it, finish it, get it ready to go, and send it on. It sounds complicated, but it actually saved a lot of steps, a lot of work-hours, and allowed us to pull off missions that would otherwise have been too expensive.”
Of course, MGS had one major issue during aerobraking: the broken solar-power-panel support.
“What happened was the yoke, or frame, that connected the solar panel to the spacecraft body broke; it snapped on one side, and so if we did aerobraking the way we originally planned, we would have had problems. Remember that aerobraking involves flying down into the atmosphere a little bit and using the drag to slow down the orbit, and slowly your orbit gets smaller and smaller.
“Now these orbital spacecraft have solar panels spread out to the sides. No avoiding that, they need the power supplied by the sun. So the plan was to actually configure the solar panels facing in a certain direction, so when we dipped down into the atmosphere, the force exerted by the atmosphere actually pressed evenly against those panels, using them a bit like resisting wings. But with that broken mount, the force would have snapped the solar panel off, and that would have been the end of that spacecraft! So instead we had to prepare the spacecraft for each dip into the
atmosphere by rotating the broken panel 180 degrees, so the force was pushing now against the break back into position.
“As you are doing this you're worried about lots of things, including the motors that turn the solar panel. These things won't work forever. And because you don't know when you're over-amping (and potentially damaging) something, you want to minimize that kind of activity on your spacecraft so you don't just use up all of your resources before completing your assigned mission.
“Nonetheless, during each aerobraking maneuver, and this took months, we had to rotate the broken panel away from the sun, dip into the atmosphere, back it out, and rotate the panel back around. Ultimately we were fine, it was just a matter of coordinating all this activity as you did this aerobraking stuff. In the end it was actually quite the education for everyone, and that was really how aerobraking got perfected and it's something that's been used many times since. But as the first, and with that mechanical problem, MGS was what forced us to do some really crazy things to make it happen.”
Once MGS got into its stable, round orbit, the images—and discoveries—started to pour in. Many were memorable, even these many years later: “I remember a picture that showed a crater wall with what looked like a dried streambed from liquid water that appeared to have gushed out of the wall. That was just one thing. We found all kinds of neat stuff using the science instruments on that spacecraft. But in the end, MGS was really about efficient planetary operations, and therefore low cost, and about trying to recover the science that we lost because of the failure of Mars Observer.”
But the thrill of discovery was punctuated with some true white-knuckle moments…ones that required immediate attention: “So I'm in bed one night, this was during aerobraking, we were in a forty-hour orbit, and had a long way to go. The phone rings at about 2 a.m., and I picked up and it's Glenn, the project manager, and he's asking if I have any idea why the spacecraft just
executed the same course correction maneuver twice and we're headed for an impact on Mars.
“That's an attention-getter, and I woke up really fast…. I said, ‘No I don't have any clue, but I'm on my way in right now.’ I jumped out of bed, got dressed, and hopped into the car. Soon I'm doing ninety miles an hour from my house to the lab, which is twenty-three miles. I think I got there in about fifteen minutes! I walk in and Glenn is standing there in the conference room looking at some telemetry, asking what I thought was going on. I said I had no idea (I had just woken up, after all), but I'm sure that it's some part of my ground system, so I just stayed there to dig into the problem. We had several hours, but it was still necessary to fix the problem (we were headed for impacting Mars), and so he woke up people to do an emergency maneuver.”
“Long story short, the spacecraft was saved because the flight team got up, everybody came in, they figured out what they had to do to get the spacecraft back into a normal orbit, and everything was fine. As it turned out, we discovered that a remote software delivery had installed a new memory map for the spacecraft, which tells it where to store things in the spacecraft's computer. This resulted in the maneuver commands being put into two different locations in the spacecraft's memory and those commands being executed twice…and that's not good.”
“Once I figured out what happened, the solution to that was really quite simple: don't let folks just deliver software when they feel like it. This was all a matter of timing. The error occurred because the ground software had been changed at the same time as a sequence was being processed through it, changing the intended result. I know some people were very unhappy about my policy changes because they were so used to doing things loosey-goosey, but it became a strong, hard-and-fast rule. ‘You will coordinate all software delivery with the operations team so that this doesn't happen again.’ If this had happened during one of the shorter [later] orbits, we would have lost that spacecraft.”
But, thanks to personal dedication by the flight team (and lax speed-limit enforcement of some Los Angeles-area freeways in the wee hours of the morning), MGS lived to fly on into history. One of its major discoveries was the verification of a lack of a global magnetic field around Mars.
“Either Mars doesn't currently have a molten core, or if it does, it's not a gigantic thing like the Earth's. If it exists, then it's probably heated by friction but it is too small to cause convection. And without this, you won't have a global magnetic field. Note that at one time Mars had to have had a fairly substantial molten interior, because you have these huge volcanoes on it, at least four of them. But for now we don't see much of a magnetic field and what we do see is localized and regional.”
And then, in November 2006, it was suddenly over. Unplanned, unanticipated, the mission came to a sudden halt: “The mission ended, but it ran four extended missions and operated for about seven years, so it was not like MGS died when it was at its prime. It was an old spacecraft, and while it was still producing perfectly good data, it had paid for itself several times over. Recall that each of those missions lasted a Martian year, or about two Earth years, so we got our money's worth out of this; it's all gravy after the primary mission, certainly after two or three extensions.”
That said, Brooks perceives a sacred trust within the JPL community, a bond between the taxpayer and those who fly the unmanned missions of discovery: “We have a mantra, and that is that spacecraft don't belong to us. It's not my spacecraft, it's not even really NASA's; it belongs to the American people, and because of that, we have to take very good care of it, and we have to get out of it everything we possibly can. We must basically milk the cow until it's dry, and that's exactly what we do with all of our missions. Look at what we did with the Mars Exploration Rovers. Eight years in, one of the rovers is dead, but the other one is still going along and doing its thing. That mission was supposed to
finish years ago, and yet still it's going. But let's go back even further: look at [the Voyager mission, both Voyager 1 and Voyager 2 are] still running, for probably another twenty years, until the RTGs' [Radioisotopic Thermal Generators'] power supplies aren't working anymore. That was launched in 1977, and it's still sending back data a quarter century later! That's value.
“When I go out and do talks, one of the things I tell people is that the cost of a large mission is, say, 750 million bucks. So you look at the population of the United States, and it's about 300 million people, and so for a bit over $2.50 per head, not even the cost of a Big Mac®, you pay for your part of a fantastic mission of discovery. Just skip one burger for a day in twenty or thirty years and you've done your part to pay for the mission. I think it's worth it.”
And while McDonald's may not enjoy the same sentiments, it seems to be more than a fair trade. I'll give up a burger for
two
days myself.