New Horizons enters safe mode 10 days before Pluto flyby

http://www.planetary.org/blogs/emily-lakdawalla/2015/07042044-new-horizons-enters-safe-mode.html

by dandelany • 11 years ago

154 47 11 years ago

robertfw• 11 years ago

I wouldn't want to be the one debugging this. Talk about pressure to deliver and difficult constraints!

stevewepay• 11 years ago

On the contrary, this is where you can really prove that you are worth your salt. There is no better arena to prove yourself than a real-live production-down situation.

Aldo_MX• 11 years ago

I beg to differ, a situation with a lot of factors you can't control is rarely a way to prove that you're worth your salt, It's the way you communicate your advancements what makes a difference.

JshWright• 11 years ago

This is a pretty common sentiment among firefighters/paramedics. You don't wish for something bad to happen to someone, but you absolutely want to be there if it does.

Sharlin• 11 years ago

"Proving oneself" is not a thing everyone values.

madaxe_again• 11 years ago

That assumes you're proving yourself to a rational engineer - rather more likely one would be attempting to prove oneself to a non-tech manager, and what happens there is that typically the fixer is so deeply associated with the problem that they end up being seen as the problem.

I mean, how many times have you fixed a downed server through no fault of your own to then have your head blown off by a client?

kabouseng• 11 years ago

It's a shame really, as the engineers fighting the crisis' gets a lot of attention, but very often those crisis' are caused by themselves.

It is the engineers who's projects run smoothly who is ultimately worth more, as they can predict and prevent problems before they become a crisis, but get no recognition for it.

verandaguy• 11 years ago

It might be a shame when your real-live production-down situation is a multimillion dollar space probe nine years into a mission to capture the first high-resolution imagery as well as collect mounds of scientific data about one of the (ex-)planets in our solar system we know the least about, right after discovering surface anomalies including four suspicious dark spots on one of the body's faces.

This isn't about proving you're worth your salt. It's about "let's get this thing working, ASAP, because who knows when we'll have another chance, since there's not exactly a plan B."

mdellabitta• 11 years ago

I think I figured it out. Someone put their book on the F8 key during a restart.

raverbashing• 11 years ago

Very likely it's cosmic rays causing a memory error

wkcamp• 11 years ago

Does anyone know which language(s) the NASA engineers write the spacecraft's software in?

themodelplumber• 11 years ago

Pretty interesting maybe-answer to that, from some searching:

http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit

Which language is used for programming New Horizon's flight software? How long is code? How do you make sure that it's gonna work?

We don't do the coding for the flight software, that's SciOps.

However here's what I know, each set of spacecraft commands is put up as a "command load", which has a name that's the year plus the day of year. So the 15188 load runs on July 7, and has commands for nine days.

Each load has a multi week coding and vetting process and is simulated on the ground, on a system called "NHOPS", pronounced "nops".

-AZ

I emailed some folk, here's an answer to #3, from Jillian, one of the members of our SciOps team.

3: We have a program called Statesim that checks to make sure constraints are not violated. For instance the Alice aperture door shouldn't be open within 20 degrees of the Sun. We also have something called NHOps which is a software EM of New Horizons and the commands are executed on it and data is downloaded and reviewed by the instruments. We also had a rehearsal in 2013 for the Pluto Closest Approach load.

-AZ

Okay, here's an answer from Helen Hart at Johns Hopkins University Applied Physics Lab (AZ):

We do not program New Horizons flight software. We store commands, packaged into macros, in the portion of C&DH memory designated for command storage, and execute the macros via a Time Tag.

See #1.

How are we sure that it¹s gonna work? That¹s a really, really big question, involving multiple layers, multiple platforms, and a lengthy, intensive Load Build and Review process. For more details on the process, talk to the Science Sequencers.

The command load is simulated in SEQGEN, which is also where it gets built. The Seqgen Modeling File contains hundreds of checks for problems with commands vs. the state of the spacecraft/instrument. The Seqgen modeling file also contains the Mission Operations Playback Data Volume models.

We have a SOFTWARE SIMULATOR called stateSim which models the response of the spacecraft and payload to the command sequence. stateSim generates the file that Mission Operations Command Sequencer turns into the Command Sequence Timeline - the excel spreadsheet that is distributed as part of the load review process.

We have HARDWARE SIMULATORS, called NHOPS-1 and NHOPS-2, which contains HARDWARE modules for the onboard processors, including C&DH-1, C&DH-2, SSR-2, SSR-2, G&C-1, G&C-2, P-Alice, LORRI, PEPSSI, Ralph, REX-1, REX-2, SDC, and SWAP.

The NHOPS and stateSim simulators have different weaknesses; for example: a: NHOPS gets all the details of slews correct; stateSim gets the slew start time and duration correct, but does not track the exact path of the track because that part of the CG&C cannot be modeled in software. b: stateSim correctly predicts C&DH Thermal Control, but NHOPS does not predict this at all c: SSR usage: stateSim does not model SSR usage. NHOPS does, but gets the wrong answer for anything that gets Compressed or Packetized. SEQGEN is tied directly to commanded data downlinks, and cannot be manipulated to produce interim data volumes; it correctly models downlink data volume TO THE ACCURACY OF the Compression Ratios specified by the Science Sequencers (lately, DataTrack).

-- Helen

natep• 11 years ago

The main command and data handling processor runs code written in C, as does the guidance and control computer. However, the G&C algorithms were 'written' in Simulink, and autocoded to C. I believe the instruments also use C, unless they are too resource constrained even for that, in which case, I'd guess Forth.

patcon• 11 years ago

Yeah, it's bad enough when there's latency and my SSH session is delayed several seconds... 3 billion miles is 4h at light speed.

Well hey, at least they're probably not using SSH ;)

dandelany3• 11 years ago

Alan Stern, the mission's P.I., dispelled rumors that contact had been lost on the USF forum, saying "Such rumors are untrue. The bird is communicating nominally."[0]

Also the Deep Space Network[1] page shows an ongoing 1kbps downlink from New Horizons; during the safe mode event it was at only 9bps. So that's a good sign! I'm sure they are still panicking a bit about what went wrong, but hopefully we're out of the woods on this particular anomaly.

[0] http://www.unmannedspaceflight.com/index.php?showtopic=8047&... [1] http://eyes.nasa.gov/dsn/dsn.html

magicalist• 11 years ago

Alan Stern, the mission's P.I., dispelled rumors that contact had been lost on the USF forum

Not sure what you mean? He was responding to the (supposed) rumor that contact had "been lost again or was never regained".

That Deep Space Network Now page is great. Thanks for sharing.

dandelany3• 11 years ago

We're saying the same thing. He dispelled rumors that contact had been lost [permanently]. Sorry if that was unclear - the article clearly states it was certainly lost the first time.

tptacek2• 11 years ago

The idea that we have an ongoing digital comms with something this far from Earth is completely fascinating to me. Are the encodings used documented anywhere? All I can find out is that it's X-band, 1kb/s.

throwaway_yy2Di• 11 years ago

    "...encodes block frame data from the spacecraft Command and
    Data Handling (C&DH) system into rate 1/6, CCSDS Turbo-coded
    blocks."

[pdf] http://www.boulder.swri.edu/~tcase/NH%20RF%20Telecom%20Sys%2...

bigiain• 11 years ago

Sadly, I'm guessing a scavenged satellite tv dish and a $12 SDR with a home built antenna is not going to let me listen in...

I wonder just how badly TCP would cope with 32million millisecond round trip ping times?

tsieling• 11 years ago

Edge. Of. Seat.

TheOtherHobbes2• 11 years ago

Edge. Of. Solar System.

kristofferR• 11 years ago

It won't reach the edge of the solar system for many decades, at the very least (if the edge is defined as interstellar space).

However, it truly won't leave the solar system entirely for at least 30 000 years: http://spectrum.ieee.org/tech-talk/aerospace/astrophysics/vo...

user• 11 years ago

Not Implemented Tor IP not allowed

Come on! What's the point of blocking tor for them?

biot• 11 years ago

  > and a less educational (but not catastrophic) gap in our
  > light curves for Nix and Hydra.

Can someone explain the importance of this? What data would this have provided?

greglindahl• 11 years ago

You can figure out the rotational period, and get an idea of what the surface looks like, from a light curve. For example, if half of it reflects a lot more light than the other half, then you'll see a repeating brighter/dimmer/brighter/dimmer pattern with a period equaling the rotational period.

If you have a gap in the light curve, it increases your uncertainty.

nathanb• 11 years ago

That is a really well-written article. It explains the problem, explains what the remediation plans are, and puts a human face on it...while remaining positive and optimistic.

Looking forward to seeing what the problem turned out to be and how they solve it!

dandelany3• 11 years ago

I highly recommend the planetary.org blogs for space news, especially Emily Lakdawalla's - they are always excellent.

zatkin• 11 years ago

Does anyone know how good of a quality the cameras are on New Horizons?

imrehg• 11 years ago

Here's some detailed description[1] of New Horizon's science payload. Also technical details link on the bottom of the page. I guess not all the information is in the form of a layman (like me too) asking "just how good they are?", so might need some "science->masses" translation.

There's also a writeup on ArXiv[2], that seems to be even more detailed, though just found it and skimmed so far.

[1] http://pluto.jhuapl.edu/Mission/Spacecraft/Payload.php

[2] http://arxiv.org/abs/0709.4261

elahd2• 11 years ago

Wikipedia says the camera is 1024x1024. There are a bunch of other non-optical sensors on board, as well.

https://en.wikipedia.org/wiki/New_Horizons

kenrikm• 11 years ago

1024x1024 seems rather low res even by 2006 standards. Any idea what special/expensive/practical reason it needed to be that low res?

r721• 11 years ago

"NASA’s New Horizons mission is returning to normal science operations after a July 4 anomaly and remains on track for its July 14 flyby of Pluto.

The investigation into the anomaly that caused New Horizons to enter “safe mode” on July 4 has concluded that no hardware or software fault occurred on the spacecraft. The underlying cause of the incident was a hard-to-detect timing flaw in the spacecraft command sequence that occurred during an operation to prepare for the close flyby. No similar operations are planned for the remainder of the Pluto encounter."

http://www.nasa.gov/nh/new-horizons-plans-july-7-return-to-n...

New Horizons enters safe mode 10 days before Pluto flyby

Related Stories