NASA puts jet propulsion lab on blast over Psyche mission failures

https://futurism.com/the-byte/nasa-jpl-psyche-independent-review

by zeristor • 4 years ago

138 173 4 years ago

cardamomo2• 4 years ago

NASA put out a press release as well. It has similar details and a link to the report itself. https://www.nasa.gov/press-release/as-psyche-mission-moves-f...

mlindner• 4 years ago

Agreed. Your link is the better link. "Put on blast" is wording that shouldn't really be used on hacker news for a technical subject.

Also here's the pdf link that's at the bottom of that page to the full set of comments along with NASA's responses: https://www.nasa.gov/sites/default/files/atoms/files/psyche_...

Unfortunately it's a PDF of raw image scans.

geuis• 4 years ago

Agreed about the wording. I understand adapting reporting to appeal to a wider, younger audience, but it also cheapens and lessens the importance of the reporting.

kbenson• 4 years ago

There's a long history in news of using a play on words for your story title to make it more catchy. I wonder if it's just that it's a less common term in your own experience that makes it more objectionable?

I could see Ars Technica making the same pun, and I wonder if they would get as much flak.

ChrisMarshallNY• 4 years ago

Can that be made the official link?

I really don’t like the other site (we’re not supposed to complain about that kind of thing, so I won’t).

everybodyknows• 4 years ago

Indeed. "Put on blast" is first of all slang and therefore anti-intellectual, ageist, and soon to become antiquated itself. Worse, it's ambiguous: A web search turns up alternate meanings, either public vilification, or publicizing of personal details (aka "doxxing).

edlinfan• 4 years ago

Important for these COVID-19-related findings was the lack of informal communication that the project team suffered in the absence of face-to-face interactions. In the past, JPL's success typically relied on senior members of projects and technical line organizations "walking the floor", dropping in for conversations at office doorways, or chatting in the cafeteria. Without these informal communication mechanisms, contextual clues and situational awareness were lost. Team members working the floor found it difficult to report problems up the chain over Webex, especially when attendees kept their cameras off."
... The lockdown conditions contributed significantly to the question of why Psyche and JPL leadership did not know of the severity of the problems with GNC and V&V until it was too late to correct course.
... The IRB recommends that, given these exceptional circumstances, the team should minimize remote work conditions.

JPL has been suffering a brain drain over the last few years, and it sounds like the remaining cadre of managers are not so competent. Remote work is a convenient scapegoat, but it is far more likely that the problem is an ossified chain of command and a lackluster set of communication tools (Webex, shudder).

tjr225• 4 years ago

WebEx does exactly the same thing as any of the other video chat clients. At least it did the last time I used it.

quacked• 4 years ago

I worked at NASA JSC for three years in the ISS cargo division; although I never interacted with the JPL teams, I did coordinate with a ton of people across SpaceX, Northop-Grumman, JAXA, ESA, Boeing, etc.

I simply cannot explain the level of organizational dysfunction in that institution. In many cases there were thoughtful and talented people in the job roles, but there were many blinding, glaring failures of basic information storage and availability across the entire org. My favorite example is having a hard time finding all of the requested cargo transactions for resupply flights in the cargo database because the "desired flight" field was a free-text entry instead of a drop-down menu connected to a central list of NASA flight title strings. I hammered on that issue for fifteen months straight and got absolutely nowhere with it, and I had a list of similar issues about seventy items long. If all of my issues had been addressed we probably could have reduced the number of people in the loop in cargo logistics by half.

The article is talking about leadership "walking the floor"--we had something similar happen, where I couldn't convince any of the major ISS divisions to fix their cargo data so we didn't have to interact face-to-face, but their higher-ups would call and email me to double check the status of a cargo request. They were "walking the floor" and "staying in touch", but only because they wouldn't (or couldn't) adopt any policies to make a streamlined system work. I don't know if the same thing is happening at JPL, but I have to assume that it is, given the pathological nature of the repeated communication failures I saw.

A well-organized institution should be able to provide clear visibility into their data and projects just through smart organization of their digital assets. If they don't have that basic competence, then they would be extra screwed by remote work.

dmix7• 4 years ago

Remote work is a convenient scapegoat

It's the best thing bosses have going for them in terms of scapegoats.

Although you can't blame them for being uncomfortable with this level of change. Remote work mainstreaming, without the COVID excuse anymore, is a whole new experiment being tried for the first time widely in society.

dtgriscom2• 4 years ago

For the past decade I've worked for a company whose headquarters is 500 miles away. I have 3-6 video meetings a day with engineers at the headquarters (as well as all over the world).

Until the pandemic, I visited headquarters every few months to catch up with the teams. I found that in informal chats (in the halls, walking into someone's office, having lunch) I discovered so much that never came up during video calls. It's similar to, but worse than, formal in-person meetings with agendas; there's an activation energy people need to overcome to bring up something that may seem small but is actually critical to the project.

Someone in a video meetings with their camera off is even worse; you have almost no emotional connection with them, and it's natural for them to only "speak if spoken to".

avalys• 4 years ago

“Put on blast”?

Never heard this idiom.

iepathos• 4 years ago

I think it comes from british slang where to blast someone means to exclaim annoyance at them. May have heard the exclamation like "oh blast it" when something goes wrong.

sammalloy• 4 years ago

Thanks. I’ve never heard this in the states.

detaro• 4 years ago

https://idioms.thefreedictionary.com/put+someone+on+blast

version_five• 4 years ago

  put me on blast on MTV
  "Yeah, he's cute, but I think he's married to Kim, hee-hee!"

https://genius.com/Eminem-the-real-slim-shady-lyric

From an eminem song, from ~the year 2000. That's the only place I'd heard it before

optimalsolver• 4 years ago

I think it was from urban slang in the 90s.

AnimalMuppet• 4 years ago

A bit obscure for the US, but fits well as a pun when the topic is NASA and JPL.

faizshah• 4 years ago

Common American (and online) slang meaning calling out someone. I believe it originates from Chicago or the Midwest but it got popularized on social media like a lot of other slang in the 2010s and now is pretty much used everywhere especially among Gen Z.

kfajdsl• 4 years ago

As a Gen Z, TIL "put on blast" is not just a normal common phrase

Maursault2• 4 years ago

An informal definition of "blast" is a severe verbal reprimand, and it is unlikely you've never heard it used that way because it originates in the US in the late 19th century. "Put on blast" means to embarrass publicly by admonishing, and also originates in the US, but is much more recent. Compare to "put on full blast," (sound familiar?) which has a different but related meaning, but like all these uses originate from Old English blæst "a blowing, a breeze, puff of wind," and later, blast furnace, the idea of forcing air to increase combustion for smelting iron.

VyseofArcadia• 4 years ago

Another sign of getting old is it's not just the whippersnappers using slang I don't understand, now it's showing up it news headlines too.

Also, current slang in headlines seems like an odd choice anyway. I get there's a not-great pun there and all, but the article doesn't seem like it's written for an audience that already knows what "on blast" means.

bearmode• 4 years ago

You're > 22 years out of date on that one then, a google search showed 'on blast' being used since at least the year 2000

clnq4• 4 years ago

The culture of ignoring clearly-voiced problems has killed so many projects and companies.

cardamomo2• 4 years ago

I'm curious which projects and companies you're thinking of. Would you mind sharing?

saagarjha• 4 years ago

Probably easier to make a list of products that don’t suffer from this.

fragmede• 4 years ago

It would take a college class on the subject to do the topic justice, but the one that comes to mind readily for me is the Space shuttle O-ring disaster.

clnq4• 4 years ago

I can mainly think of companies and projects from my professional experience, and it wouldn't be my place to publicly speak about these things. But I believe that many engineers like me have worked on death march projects where their critical problems have been brought up with management and thoroughly ignored.

Still, to try and give you some examples, Google Glass was known to be a failed product early on due to technical and practical problems voiced by engineers. But there was a culture of optimism or wilful ignorance at Google about Glass, so the project continued until it died due to these problems. And there are some articles on Hacker News about it. Right now there are murmurs online that Meta engineers are very aware that the metaverse is not engaging and VR efforts should pivot elsewhere. Yet the management seems to ignore this, even at tremendous cost to the company value. The whole VR effort at Meta could die because of this.

The interesting aspect of this culture is that it's not a "hindsight is 20-20" situation. In complete contrast, it's that engineers bring up problems that will significantly harm a project or a company in the future, but are ignored.

lisper3• 4 years ago

B737-MAX. (It didn't kill the company, just a few hundred passengers, but I still think that qualifies.)

throwawaysleep• 4 years ago

My anecdotal experience is that things just fall off the table if not constantly raised again and again.

It is ignored because something else came up and that one was forgotten.

sitkack2• 4 years ago

It was forgotten because the people receiving the message do not understand. They don't have a solid mental model that they can use to calculate cause and effect. They are unable to gauge severity or risk.

They might not even be able to differentiate from risks and deliverables. Tasks falling off any list shows that the relative importance of the task is not known, only recency is.

sidewndr462• 4 years ago

And in the case of NASA: quite a few astronauts

seibelj• 4 years ago

Solution - gotta give NASA and other government agencies money. That is the only solution

yarg2• 4 years ago

There was no reason that the Challenger disaster could not have been prevented.

https://www.npr.org/sections/thetwo-way/2016/01/28/464744781...

https://www.npr.org/sections/thetwo-way/2016/03/21/470870426...

If I were Ebeling, I'd've punched my manager.

joshuahedlund• 4 years ago

I don’t have any insight about the delay but this year I read A Portrait of the Scientist as a Young Woman, a memoir of the project manager behind the original proposal for this mission. While not exclusively about Psyche it was a great book with an interesting behind-the-scenes into the NASA mission selection process. It got me excited for the mission and sad about the delay.

Maursault2• 4 years ago

A Portrait of the Scientist as a Young Woman

Obviously this title is word play, taken from A Portrait of the Artist as a Young Man by James Joyce, his first novel. As I recall, the protagonist becomes so popular he can't get any work done.

colordrops2• 4 years ago

Scifi story: private interests bribed management at JPL to scuttle the mission so they could send a secret mission to collect the loot for themselves... or protect their precious metal cartel from devaluation.

hobs• 4 years ago

This is already the plot of - spoilers - Don't Look Up.

colordrops2• 4 years ago

I watched it, and either my memory sucks, or that wasn't the plot.

xqcgrek2• 4 years ago

Different NASA centers have very different cultures, with their pros and cons. JPL is known to have a tendency to overpromise and underdeliver.

lisper3• 4 years ago

I used to work at JPL and I have been a vocal critic of theirs at times but I think you are completely off the mark here. Most of the time JPL delivers. (It's actually pretty amazing when you have seen how the sausage gets made.)

mistletoe• 4 years ago

Can you give any insight on how Psyche failed? I mean we have the report but I still don't understand how any of that could happen.

gateorade4• 4 years ago

This is kind of a weird take. I used to work at JPL but I didn't stay due to it not being the right cultural fit for me.

With that said, my perspective as a former NASA insider is that JPL is known for the exact opposite, underperforming and overdelivering.

I mean, they have a nearly perfect track record of mission success (sorry Beagle) and when the missions do succeed they often persist for many times over the original lifetime requirement.

Additionally, and yes this is a vast oversimplification, if you just look generally at space exploration missions, they are far more prolific than any other NASA center.

Melting_Harps2• 4 years ago

This is kind of a weird take. I used to work at JPL but I didn't stay due to it not being the right cultural fit for me.

Any war stories you want to share?

electriclove• 4 years ago

What about the culture made it not a good fit?

Kukumber• 4 years ago

- no effort

- no responsibility

- overpaid with free money

The society you deserve..

This team of former NASA employees are a relic of the past unfortunately, they should investigate other companies too, including Microsoft, that would be fun :)

stefan_• 4 years ago

Well the review board found that senior executives failed in their responsibilities and had no project insight (but remain today), so on that part you are correct, but they also found that the actual engineers were understaffed and underpaid in face of aggressive hiring from space startups. The solution they propose, of course, is to cancel remote work, the root of all problems when the worker bees won't budge.

oses• 4 years ago

that part definitely stood out to me as well. I mean, clearly there are some parts of spacecraft / mission design that are best done in person, but the recommendations of the IRB definitely won't help the staffing issues.

fakethenews2022• 4 years ago

That is bias on your part. The review also found that lack of informal communication was a factor and recommended minimizing remote and hybrid work.

Pages 32, 33

https://www.nasa.gov/sites/default/files/atoms/files/psyche_...

Animats• 4 years ago

Surprising that Musk doesn't have a self-funded project to check out that asteroid and bring samples back. Looks profitable.

pencilguin20• 4 years ago

Wouldn't be. The "$ quintillion" figure is just clickbait.

Ore on Earth is worth a lot more, and is right here, ready to be dug up.

imiric3• 4 years ago

Wouldn't be. The "$ quintillion" figure is just clickbait.

Source? At this point we can't know for sure, which this mission is supposed to determine. But we know its size, and estimating its composition puts it into trillion or quadrillion dollar territory, which is still a crazy amount.

And we're talking about a _single_ asteroid. The possible value found in the asteroid belt is—literally—astronomical.

Ore on Earth is worth a lot more, and is right here, ready to be dug up.

Not quite. Ore on Earth is limited, and increasingly difficult to mine, requiring costly processing, deep sea expeditions, and producing toxic side effects for the environment. It's also politically problematic, as countries control access and exportation.

Asteroid mining would be much more expensive, but the ROI could potentially be much more lucrative. Once the space operation is setup, costs would eventually come down, and it opens up many further space exploration opportunities.

SpaceX is in a unique position to be one of the first companies to launch such missions, that I'm also surprised that Musk is not jumping at the opportunity.

It would still be a political challenge, and it creates new conflict arenas, but this is inescapable human nature.

Asteroid mining is inevitable and we'll get there eventually. Beltalowda!

basementcat• 4 years ago

Not surprising at all. SpaceX has no experience building and delivering a functioning payload beyond the orbit of the Earth.

mortehu• 4 years ago

"Functioning" is doing some work here, since Starman is orbiting the sun, slightly beyond Mars at aphelion.

dmix7• 4 years ago

VERITAS Venus Mission is going to have to be delayed from 2028 to 2031 due to issues caused by JPL's failure to launch Psyche

That's extremely disappointing. Venus is easily the 2nd most interesting planet "close" to us besides Mars. I was really looking forward to that one.

From the standpoint of total mass delivered to Venus, the best launch opportunities occur in 2029 and 2031.

It's too bad Russia's ambitious space programs are mosty vaporware. They pinoeered Venus exploration and have their own (pretend) Venus program that AFAIK isn't going anywhere.

a proposed Russian space mission to Venus that would include an orbiter and a lander to be launched in 2029.

https://en.wikipedia.org/wiki/Venera-D?wprov=sfti1

pencilguin20• 4 years ago

Venus is more interesting to me than Mars. We already know more about Mars than I cared to know. Venus is still packed with mystery. Hot mystery, yes.

dmix7• 4 years ago

It's also really interesting purely from a physics or human adventure perspective. The way it crushes anything that goes near the surface.

Mars seems like an introductory class to Venus' horror story. But yes there's plenty to learn, especially from a statistical analysis for finding life in interstellar space type of thing.

ramraj07• 4 years ago

I highly recommend anyone with any interest in space follow https://youtube.com/c/MarsGuy / he gives a weekly update on what the rovers find in great detail and it’s fascinating.

At the same time I continuously question if the findings they make wit these rovers continue to justify the multi billion dollar expense we make on these missions. I’d much rather we spend our limited resources on Venus and Europa, and actually starting to send interstellar probes than continuing to double down on Mars. It’s become a vanity project for billionaires anyway so let them figure out putting a man there. NASA can start focusing on the next step in exploration of space.

wallfacer120• 4 years ago

There's no mystery to Mars, just letting its crust hang out there for all to see.

dr_dshiv3• 4 years ago

The moon is more interesting than either. Focus on asteroid belts. Big gravity wells suck.

lordgrenville• 4 years ago

It's kind of funny to hear someone disappointed about a delay from 2028 to 2031. To my brain those both just fall into the same "long time away" bin.

plonk• 4 years ago

Not that far away now.

netsharc• 4 years ago

As a kid I remember seeing a project that had a projected finish date of 2002, and thinking like that..

Remember 2017? That's just as far as 2028 is.

dehrmann3• 4 years ago

which is set to investigate a rare metal asteroid that could be worth a reported $10 quintillion dollars if mined.

World GDP is ~100T per year. I'm guessing that estimate was just the amount of rare earth metals times the current price, but the magnitude it ignored economic realities is comedic.

kragen3• 4 years ago

The asteroid is 16 Psyche, which is a 220-km-diameter M-type asteroid. https://en.wikipedia.org/wiki/16_Psyche doesn't say what metals it's supposed to be comprised of, but rare-earth metals are not very likely and not suggested by the article. A more likely set of "rare metals" is platinum-group metals.

16 Psyche is 2.3e19 kg, according to WP, so 10 quintillion dollars works out to 43¢ per kg. It's pretty plausible that any kind of metal would be worth at least 43¢ per kg, it wouldn't even have to be very rare. Even steel is worth abut that much. If you're going to use it on Earth you'd be better off mining iron ore on Earth, but you can't do that with platinum-group metals, and if you want to use steel for construction in space, it's a huge advantage if it's already there instead of having to be lifted out of a gravity well.

I don't think there's an obviously correct way to compare economic value across generations' worth of time. 100 years ago people didn't have antibiotics, so death from infected wounds was common, even wounds that would be minor today. But old-growth lumber was abundant.

zeristor4• 4 years ago

Whilst a salient point, it wouldn't be as though they turn up one day with $10Qn of metals to sell on the market.

It might take decades, even centuries to return that to Earth, and of course how would they be able to do that, the amount of fuel required itself would mean that the price of that would be huge.

I doubt solar sails would be that viable, and the idea of the oceans going dry in order to produce the fuel to return metals would be a tad far fetched.

zeristor4• 4 years ago

Although perhaps using the less valuable parts of the asteroid as reactive mass might be useful idea. That though would end up with billions of projectiles in random orbits around the Sun.

bsaul• 4 years ago

aren't there any "gravitational rivers" from that asteroids one could use to slowly have the mining cargos float to an area close to earth ?

mightybyte• 4 years ago

That line actually made me think the whole thing was satire for a bit. I actually came back to the HN comments to see if people here were taking the article seriously.

TobyTheDog123• 4 years ago

Some context - Laurie Leshin, the director of the JPL, recently joined the lab after resigning from the Worcester Polytechnic Institute. Some say she resigned as suicides rocked the school and complaints about mismanagement flooded in [1].

I'm curious if there's any sort of pattern here, and how much of the blame can fall on the director.

[1] https://www.insidehighered.com/news/2022/02/03/multiple-suic...

vietjtnguyen2• 4 years ago

Hard to put much blame on her I think. She only joined JPL this year. Psyche was selected back in 2017.

acyou• 4 years ago

https://www.nasa.gov/sites/default/files/atoms/files/psyche_...

I was impressed with the report, it seemed comprehensive. I think all of the answers are in there if you read carefully and read a little into what is said.

The depletion and lack of availability of experienced technical personnel is repeatedly cited as the leading cause of the failure to develop quality software on schedule. The paper's authors don't explicitly discuss the possibility that the issues might not just be administrative problems, but that there could also be technical and system design issues with the project. On a superficial level, the report's recommendations are evaluating and solving problems in administrative ways: hire more people, improve communication, reduce remote work, improve retention (hard to do those last two at the same time).

If the overall lack of experienced technical personnel were such an issue, why are the project failures on the software side, with zero issues reported on the mechanical side?

From a system design perspective, a key issue is running into a problem and deciding to "deal with it with the software". This is done without thinking through the issue, effectively kicking the can down the road. Problem solving is postponed. Once the software is being written, it's too late to solve the problem at a system design level.

Another way to see it is that deep structural system design flaws, or any other reason to suspect that the project would fail, could have prompted an exodus of experienced personnel that saw project failure coming early on in the project. The report doesn't explicitly evaluate the system design, but has a good overview of the project fundamentals. One hint is the slide talking about how they shouldn't be trying to integrate a third-party commercial orbiter designed for Earth operations for their deep-space operation. That would be an example of a top-down system design flaw, where it would be (and probably was for many) easier to just get another job or transfer to a different department than to try to fix the holes.

One of the most telling comments is calling for a more rigorous annual review of the JPL Director's performance, right there in the report. I cannot believe they did that, talk about speaking truth to power. Good for them! Then the JPL director's signature is on the document, right at the top. It's an unfortunate situation, I'm sure they all know exactly what convening a 15-person council and releasing a public report will do to project and organizational morale.

Here are my suggestions: 1. Project leaders (and organization leaders) should able to convince everyone around them that success will be achieved, see "Reality Distortion Field" 2. Complex and dynamic functionality needs to be replaced where possible with simple and static functionality. 3. It sounds like there are cultural issues coming from the top (and unfortunately, probably from above the top). 4. Technicians and leaders should be competent first and experienced or senior second. 5. If no one competent and experienced with a Reality Distortion Field wants to touch a project, then probably better to just cancel it. 6. On any project aspect, individual contributors would work best with clearly defined requirements and without distractions. Writing the requirements properly is probably most of the work.

vietjtnguyen2• 4 years ago

If the overall lack of experienced technical personnel were such an issue, why are the project failures on the software side, with zero issues reported on the mechanical side?

Slide 38 explicitly lists staffing challenges in GNC (guidance, navigation, and control), FSW (flight software), avionics, and systems engineer.

Slide 41 also talks about external factors: "...this has required the experienced cadre to be spread thin... the rise of the commercial space industry in recent years has resulted in competition for experienced personnel, particularly in the systems engineering, GNC, and FSW areas."

UberFly• 4 years ago

The possibility that the Psyche asteroid might be the ancient core of a long lost solar system planet is very cool. We won't know until it's visited, so the delay is unfortunate.

KeyXiote2• 4 years ago

Opinion: NASA puts JPL on Blast, maybe a more proactive approach should be putting themselves on blast for not having the ability to take responsibility for joint project failures, enough with blame games, this is suppose to be a collaborative endeavor to explore and advance our collective knowledge. We f'd it up as a partner with beurocractic mismanagement, time to own our own mistakes and learn from them instead of finger pointing all the time. Reminds me of that time we decided to code imperial and metric into a system that accidentally and successfully tested how to plow a large amount of money into the surface of Mars. Let's all do better together.

NASA puts jet propulsion lab on blast over Psyche mission failures

Related Stories