NASA: DOS Glitch Nearly Killed Mars Rover

[Renew Deal]

STANFORD, CALIF. -- A software glitch that paralyzed the Mars "Spirit" rover earlier this year was caused by an unanticipated characteristic of a DOS file system, a NASA scientist said Monday.

The flaw, since fixed, was only discovered after days of agonizingly slow tests complicated by the limited "windows" of communication allowed by the rotation of Mars, said Robert Denise, a member of the Flight Software Development Team at NASA's Jet Propulsion Laboratory.

On Jan. 21, the Spirit rover stopped communicating with the teams on Earth, beginning a cycle where the rover would reboot itself, over and over. After days of tests, the team finally discovered on Jan. 26 that the issue was tied to what was originally reported as corruption inside the rover's onboard flash memory.

In a presentation at the Hot Chips conference here, Denise said that the real issue was an embedded DOS file system whose directory structure kept growing and growing. When the rover's embedded operating system then told the flash memory to mirror the data structure in RAM, the unexpectedly large file caused a fatal error and an almost continuous reboot cycle, he said.
<snip>

http://www.extremetech.com/article2/0,1558,1639459,00.asp

Gotta love microsoft.

[amber dog democrat]

1. Friends don't let friends use

Microsoft. What did they expect ?

[trotsky]

2. Oh c'mon.

The last DOS version was over 10 years ago. You're telling me that OS programmers are supposed to know how their software is going to be used on a Mars rover a decade in the future?

Yes, Microsoft has its problems, but this is no reason to bash them. NASA testing should have found the error beforehand.

[Liberal Veteran]

3. Um, I didn't see Microsoft mentioned in this article.

DOS stands for Disk Operating System.

MS-DOS is Microsoft DOS.

Could have been DR-DOS. Caldera DOS. OpenDOS. FreeDOS. NovellDOS AmigaDOS. Or just some odd proprietary version of DOS altogether (most likely).

Realize people like to pick on Microsoft, but there is enough to bash them about without cheap shots.

[BootinUp]

4. Why they would use anything built on MSDOS is a complete

mystery to me. There actually are some reliable operating systems for embedded devices.

[BootinUp]

5. oops..."undisclosed software vendor"

Sorry Bill G. I just love bashin your products still.

[trotsky]

6. I used a single-floppy firewall for years

that was based on DOS and the packet drivers for network cards. It was absolutely bulletproof, and why? Because DOS was never written to be a network operating system. If it doesn't have the capabilities, there are none to be exploited. Then my cable company went and changed their DHCP configuration, and put the DHCP server too many hops away for the firewall program to get an address - no fault of DOS, though!

[BootinUp]

8. You understood its limitations

and used it appropriately nothing wrong with that.

The biggest problem with DOS was when it became so patched and cludged that even Microsoft couldn't figure it out anymore (win98). Second problem was trying to get enough memory under the 1M real mode limit to run all your device drivers and proggies. What a pain that could be.

[trotsky]

9. Oh yes, the art of memory managment.

DEVICEHIGH and all that. EMS vs XMS. Oh, I don't miss those days one bit. But viewed in the historical context, DOS was pretty decent. Assuming no one would ever need more than 640k of conventional memory was a pretty safe thing to do back in the early 80s, and with the prohibitive cost of hardware & memory, no one could afford to over-engineer their software.

[oly]

7. It is still accepted in tech these days that

customers debug the software that they purchase. MS is just the biggest user of this totally acceptable practice.

[zbdent]

10. "Ctrl-Alt-Delete" - NASA, we lost Mars

"Now where did the f-ing planet go????"

[hunter]

11. From the article:

Moving an actuator, for example, generates a large number of tiny data files. After the rover rebooted, the OSes heap memory would be a hair's breadth away from a crash, as the system RAM would be nearly full, Denise said. Adding another data file would generate a memory allocation command to a nonexistent memory address, prompting a fatal error.

Dynamic allocation of memory is considered a no-no in embedded systems, precisely because of the possibility of a system crash, attendees said. Denise acknowledged that JPL's tests only allowed for the addition of a small number of data files, and that the exception slipped by. "We made an exception and got bit by it," he admitted.

A quick summery is that the Mars Rover software project was a mess.

Electronic voting machines are worse, by the way, and some of the people writing code for electronic voting machines have been crooked.

[david_vincent]

12. Shoulda used CP/M

[Sarvis]

13. not MS

as has been mentioned here, the Rover isn't running MS products. They run VxWorks by Wind River Software, on a 20Mhz PowerPC with 128M RAM.

see http://www.computerworld.com/governmenttopics/government/story/0,10801,88734,00.html

More on VxWorks for the geeky types out there (raising my hand, of course...)
http://www.faqs.org/faqs/vxworks-faq/part1/

Sure, the faq's from the mid-90s, but so are the Rovers. :)