We found an undocumented bug in the Apollo 11 guidance computer code

https://www.juxt.pro/blog/a-bug-on-the-dark-side-of-the-moon/

by henrygarner • 4 months ago

439 195 4 months ago

josephg4• 4 months ago

Super interesting. I wish this article wasn’t written by an LLM though. It feels soulless and plastic.

ModernMech• 4 months ago

I'm starting to develop a physiological response when I recognize AI prose. Just like an overwhelming frustration, as if I'm hearing nails on chalkboard silently inside of my head.

voodooEntity• 4 months ago

I feel ya.... and i have to admit in the past i tried it for one article in my own blog thinking it might help me to express... tho when i read that post now i dont even like it myself its just not my tone.

therefor decided not gonne use any llm for blogging again and even tho it takes alot more time without (im not a very motivated writer) i prefer to release something that i did rather some llm stuff that i wouldnt read myself.

embedding-shape4• 4 months ago

Any specific sections that stick out? Juxt in the past had really great articles, even before LLMs, and know for a fact they don't lack the expertise or knowledge to write for themselves if they wanted and while I haven't completely read this article yet, I'd surprise me if they just let LLMs write articles for them today.

croemer16• 4 months ago

Here's one tell-tale of many: "No alarm, no program light."

Another one: "Two instructions are missing: [...] Four bytes."

One more: "The defensive coding hid the problem, but it didn’t eliminate it."

gcr4• 4 months ago

For what it’s worth, Pangram thinks this article is fully human-written: https://www.pangram.com/history/f5f68ce9-70ac-4c2b-b0c3-0ca8...

xmcqdpt23• 4 months ago

Then pangram isn't very good, because that article is full of Claude-isms.

croemer16• 4 months ago

Pangram doesn't reliably detect individual LLM-generated phrases or paragraphs among human written text.

It seems to look at sections of ~300 words. And for one section at least it has low confidence.

I tested it by getting ChatGPT to add a paragraph to one of my sister comments. Result is "100% human" when in fact it's only 75% human.

Pangram test result: https://www.pangram.com/history/1ee3ce96-6ae5-4de7-9d91-5846...

ChatGPT session where it added a paragraph that Pangram misses: https://chatgpt.com/share/69d4faff-1e18-8329-84fa-6c86fc8258...

timdiggerm2• 4 months ago

So you're saying Pangram isn't worth much?

Aurornis3• 4 months ago

The AI writing detectors are very unreliable. This is important to mention because they can trigger in the opposite direction (reporting human written text as AI generated) which can result in false accusations.

It’s becoming a problem in schools as teachers start accusing students of cheating based on these detectors or ignore obvious signs of AI use because the detectors don’t trigger on it.

ChrisRR• 4 months ago

It's not setting off any LLM alarm bells to me. It just reads like any other scientific article, which is very often soulless

bbstats• 4 months ago

the subheadings are extremely AI IMHO

Jolter• 4 months ago

It repeats a few points too many times for a professional writer to not catch it.

I don’t mind that they let an LLM write the text, but they should at least have edited it.

monooso5• 4 months ago

You have no evidence that it was.

NiloCK3• 4 months ago

This is the top reply on a substantial percentage of HN posts now and we should discourage it.

It is:

- sneering

- a shallow dismissal (please address the content)

- curmudgeonly

- a tangential annoyance

All things explicitly discouraged in the site guidelines. [1]

Downvoting is the tool for items that you think don't belong on the front page. We don't need the same comment on every single article.

[1] https://news.ycombinator.com/newsguidelines.html

monooso5• 4 months ago

No idea why you're being downvoted. I've done my bit to redress the balance, I hope others do the same.

masklinn• 4 months ago

Downvoting is the tool for items that you think don't belong on the front page.

You can’t downvote submissions. That’s literally not a feature of the site. You can only flag submissions, if you have more that 31 karma.

timdiggerm2• 4 months ago

It's not a shallow dismissal; it's a dismissal for good reason. It's tangential to the topic, but not to HN overall. It's only curmudgeonly if you assume AI-written posts are the inevitable and good future (aka begging the question). I really don't know how it's "sneering", so I won't address that.

bakugo3• 4 months ago

The site guidelines were written pre-AI and stop making sense when you add AI-generated content into the equation.

Consider that by submitting AI generated content for humans to read, the statement you're making is "I did not consider this worth my time to write, but I believe it's worth your time to read, because your time is worth less than mine". It's an inherently arrogant and unbalanced exchange.

josephg4• 4 months ago

The guidelines you linked says this:

Don't post generated comments or AI-edited comments. HN is for conversation between humans.

The same principle applies to submissions. If you couldn't be bothered to write it, don't ask me to read it. HN is for humans.

TruffleLabs2• 4 months ago

"Written by an LLM" based on what data or symptom?

mpalmer3• 4 months ago

I've seen way, way worse. Either someone LLM-polished something they already wrote, or they did their own manual editing pass.

The short sentence construction is the most suspicious, but I actually don't see anything glaring. It normally jumps out and hits me in the face.

bookofjoe4• 4 months ago

Hemingway's 4 Fast Rules For Effective Writing

1. Use Short Sentences

https://www.wordsthatsing.com.au/post/hemingway-rules

rudhdb773b2• 4 months ago

Not to single out your comment, but it feels like it's gotten to the point where HN could use a rule against complaining about AI generated content.

It seems like almost every discussion has at least someone complaining about "AI slop" in either the original post or the comments.

Gigachad2• 4 months ago

HN has gotten to the point where it’s not even worth clicking the link because of course it’s ai slop.

There is some real content in the haystack, but we almost need some kind of curator to find and display it rather than a vote system where most people vote on the title alone.

Aurornis3• 4 months ago

I disagree. I like to read articles and explore Show HN posts, but in the past 6 months I’ve wasted a lot of time following HN links that looked interesting but turned out to be AI slop. Several Show HN posts lately have taken me to repos that were AI generated plagiarisms of other projects, presented on HN as their own original ideas.

Seeing comments warning about the AI content of a link is helpful to let others know what they’re getting into when they click the link.

For this article the accusations are not about slop (which will waste your time) but about tell-tell signs of AI tone. The content is interesting but you know someone has been doing heavy AI polishing, which gives articles a laborious tone and has a tendency to produce a lot of words around a smaller amount of content (in other words, you’re reading an AI expansion of someone’s smaller prompt, which contained the original info you’re interested in)

Being able to share this information is important when discussing links. I find it much more helpful than the comments that appear criticizing color schemes, font choices, or that the page doesn’t work with JavaScript disabled.

chrisjj6• 4 months ago

You're suggesting this is the complainant's fault?

furyofantares2• 4 months ago

Stop voting up slop articles and I'll stop commenting on it.

croemer16• 4 months ago

And it turns out at least the part about Rust and locks is plain wrong. What a surprise: https://news.ycombinator.com/reply?id=47676938&goto=item%3Fi...

jandrese• 4 months ago

AI tends to write like it is getting paid by the word. This article wasn't too egregious but an editor could have improved it.

iJohnDoe2• 4 months ago

I did not get any “written by LLM vibes”. I enjoyed it and it pulled me in to keep reading.

Who gives a crap if it was written by an LLM. Read it or don’t read it. Your choice.

If it conveys the idea and your learn something new, then it’s mission accomplished.

croemer16• 4 months ago

Incidental finding: another blog posts was written by Claude and they admit it openly in the last paragraph (not earlier):

   A Note on the Process
   To be clear about what happened here: Claude wrote this article.

https://www.juxt.pro/blog/what-we-learned-from-34-clojure-in...

yodon• 4 months ago

This is so insightfully and powerfully written I had literal chills running down my spine by the end.

What a horrible world we live in where the author of great writing like this has to sit and be accused of "being AI slop" simply because they use grammar and rhetoric well.

dotancohen• 4 months ago

I was completely riveted the whole read. The description of Collins' dilemma is the first time I've seen an actual real world scenario described that might cause him to return to Earth alone.

If an LLM wrote that, then I no longer oppose LLM art.

breakingcups• 4 months ago

I thought that was the least likeable part of the article. They speculated wildly, somehow making the leap that a trained astronaut would not resort to a computer reset if the problems persisted to weave the narrative that this bug was super-duper-serious indeed. They didn't need that and it weakened the presentation.

jwpapi• 4 months ago

Has someone verified this was an actual bug?

One of AI’s strengths is definitely exploration, f.e. in finding bugs, but it still has a high false positive rate. Depending on context that matters or it wont.

Also one has to be aware that there are a lot of bugs that AI won’t find but humans would

I don’t have the expertise to verify this bug actually happened, but I’m curious.

throwaway274484• 4 months ago

It's not even clear if AI was used to find the bug: they mention modeling the software with an "ai native" language, whatever that means. What is not clear is how they found themselves modeling the gyros software of the apollo code to begin with.

But, I do think their explanation of the lock acquisition and the failure scenario is quite clear and compelling.

Aurornis3• 4 months ago

It's not even clear if AI was used to find the bug

The intro says “We used Claude and Allium”. Allium looks like a tool they’ve built for Claude.

So the article is about how they used their AI tooling and workflow to find the bug.

jll29• 4 months ago

It's not even clear if AI was used to find the bug: they mention modeling the software with an "ai native" language, whatever that means.

Could the "AI native language" they used be Apache Drools? The "when" syntax reminded me of it...

https://kie.apache.org/docs/10.0.x/drools/drools/language-re...

(Apache Drools is an open source rule language and interpreter to declaratively formulate and execute rule-based specifications; it easily integrates with Java code.)

Qwuke2• 4 months ago

It's not even clear if AI was used to find the bug

It's not even clear you read the article

caminante2• 4 months ago

How did you pick out AI native and miss the rest of the SAME sentence?

We found this defect by distilling a behavioural specification of the IMU subsystem using Allium, an AI-native behavioural specification language.

ks2048• 4 months ago

They have some spec language and here,

https://github.com/juxt/Apollo-11/tree/master/specs

have many thousands of lines of code in it.

Anyways, it seems it would take a dedicated professional serious work to understand if this bug is real. And considering this looks like an Ad for their business, I would be skeptical.

wg0• 4 months ago

Someone please amend the title and add "using claude code" because that's customary nowadays.

chrisjj6• 4 months ago

Also add "AI can make mistakes". Thank you.

sgt• 4 months ago

Thank you for your attention to this matter.

riverforest• 4 months ago

Software that ran on 4KB of memory and got humans to the moon still has undiscovered bugs in it. That says something about the complexity hiding in even the smallest codebases.

whiplash451• 4 months ago

My guess is that in such low memory regimes, program length is very loosely correlated with bug rate.

If anything, if you try to cram a ton of complexity into a few kb of memory, the likelihood of introducing bugs becomes very high.

pooloo• 4 months ago

Yet here we are compounding the issues by adding more and more layers to these systems... The higher the level it becomes the more security risks we take.

SoftTalker• 4 months ago

Well you don't have room for a lot of "defensive" code. You write the program to function on expected inputs, and hope that all the "shouldn't happen" scenarios actually don't happen.

airstrike• 4 months ago

^ This is slop. Typical platitude that really means nothing.

pvdebbe• 4 months ago

Also contrast with the busy beaver problem and how much can be done with a small handful of instructions.

MeteorMarc• 4 months ago

Are there any consequences for the Artemis 2 mission (ironic)?

ChicagoBoy11• 4 months ago

For anyone who liked this, I highly suggest you take a look at the CuriousMarc youtube channel, where he chronicles lots of efforts to preserve and understand several parts of the Apollo AGC, with a team of really technically competent and passionate collaborators.

One of the more interesting things they have been working on, is a potential re-interpretation of the infamous 1202 alarm. It is, as of current writing, popularly described as something related to nonsensical readings of a sensor which could (and were) safely ignored in the actual moon landing. However, if I remember correctly, some of their investigation revealed that actually there were many conditions which would cause that error to have been extremely critical and would've likely doomed the astronauts. It is super fascinating.

deepsun• 4 months ago

And that's why it's harder (or easier?) to make the same landing again -- we taking way less chances. Today we know of way more failure modes than back then.

wat100004• 4 months ago

It's a miracle nobody died in flight during the program. Exploding oxygen tank, rockets shaking themselves to pieces during launch, getting hit by lightning on top of a flying skyscraper full of kerosene and liquid oxygen....

kevin_thibedeau• 4 months ago

They sent people up in a tin can with the bare minimum computational power to manage navigation and control sequencing. It was barely safer than taking a barrel over Niagara Falls. We do have much more capable and reliable technology.

nativeit• 4 months ago

Related topic on CuriousMarc and co.’s AGC restoration: https://news.ycombinator.com/item?id=47641528

russdill• 4 months ago

"popularly described" and how it's currently understood are two different things. Because it's hard to explain to lay people, it's popularly described in a number of simplified ways, but it's well understood.

garaetjjte2• 4 months ago

Since we are on HN, I think it could be explained there (before it's all consumed by AI slop):

For complex reasons, available CPU time during landing was lower than expected (it was stolen by radar pointing peripheral). This caused regularly scheduled job to spawn before previous instance finished. As such, this caused two effects: job instances were suspended before finishing by new instances in the middle of the routine, and that pilling up of the old instances eventually exhausted resources and caused kernel to panic and reboot. Rebooting during landing sounds scary, but that actually was fine: such critical tasks were specifically designed to automatically restart from previously saved checkpoint data in the memory.

What was more dangerous, was the suspended tasks before restarts occured. First, it meant routine wasn't executing to the end, which in actual flight caused blanked displays (as updating the display was the last thing routine was doing). Any more CPU time stolen, and it could be interrupted even earlier, eg. before it sends the engine commands.

Another issue is that in case of fluctuating load, new instances could actually begin running to the end, and then previously suspended job instance could be resumed, potentially sending the stale data to the displays and engine.

And finally, while each job instance had it own core and VAC set properly managed by the kernel (think of it as modern kernel switching between task stacks), that particular routine wasn't designed to be reentrant. So it was using various global variables ("erasables") for its own purpose, that when interrupted in unluckly place might have caused very bad behavior.

How likely all of above is to occur, depends on the exact profile of fluctuating load caused by the confused radar peripheral. I guess that's why Mike Stewart is trying to replicate these issues with real CDU.

chrisjj6• 4 months ago

The specs were derived from the code itself

Oh dear. I strongly suggest this author look specification up in a dictionary.

perching_aix2• 4 months ago

It's (what they're describing is) just reverse engineering. That's what reverse engineering is.

chrisjj6• 4 months ago

Fortunately reverse engineering too is in the dictionary - to help anyone mistaking it for spec generation.

buredoranna2• 4 months ago

Still my all time favorite snippet of code.

    TC    BANKCALL    # TEMPORARY, I HOPE HOPE HOPE
    CADR  STOPRATE    # TEMPORARY, I HOPE HOPE HOPE
    TC    DOWNFLAG    # PERMIT X-AXIS OVERRIDE

https://github.com/chrislgarry/Apollo-11/blob/master/Luminar...

donkeyboy• 4 months ago

Can you explain this to me?

buredoranna2• 4 months ago

Wish I could... but I know of it from a previous HN post, where there is some discussion on its purpose.

https://news.ycombinator.com/item?id=22367416

dylan604• 4 months ago

I think the point was the comments more than any of the code requiring explanation. There's nothing more permanent than a temporary solution

f1shy• 4 months ago

Cadr here has no relation with lisp cadr, right?

jacquesm2• 4 months ago

https://news.ycombinator.com/item?id=16008239

jasomill• 4 months ago

Correct.

CADR is an AGC assembly directive defining a "complete address" including a memory bank, in this case a subroutine to be called by the preceding BANKCALL (TC = transfer control, i.e., store return address and jump to subroutine), which switches to the memory bank specified in the CADR before jumping to the address specified in the CADR.

For a brief explanation of AGC subroutine calls, see [1].

CAR and CDR in Lisp come from the original implementation on the IBM 704, where pointers to the two components of a cons cell were stored as the (C)ontents of the (A)ddress and (D)ecrement fields of a (R)egister (memory word).

(CADR x) is just shorthand for (CAR (CDR x)), i.e., a function that returns the second element of a list (assuming x is a well-formed list).

[1] https://epizodsspace.airbase.ru/bibl/inostr-yazyki/American_...

foxyv3• 4 months ago

I'm having a really bad Mandala effect right now where I remember some XKCD that wrote a poem about this. Maybe I'm thinking of another comic.

watt• 4 months ago

Oh, it's Mandala effect now? I could swear it was Mandela before.

alexjm• 4 months ago

Are you thinking of "The Codeless Code" poem about this?

https://www.thecodelesscode.com/case/234

shagie• 4 months ago

It's reference in The Codeless Code - https://thecodelesscode.com/case/234

iJohnDoe2• 4 months ago

Fascinating read. Well done. Everyone involved in the Apollo program was amazing and had many unsung heroes.

esafak• 4 months ago

An application of their specification language, https://juxt.github.io/allium/

It seems the difference between this and conventional specification languages is that Allium's specs are in natural language, and enforcement is by LLM. This places it in a middle ground between unstructured plan files, and formal specification languages. I can see this as a low friction way to improve code quality.

kmeisthax• 4 months ago

Rust’s ownership system makes lock leaks a compile-time error.

Rust specifically does not forbid deadlocks, including deadlocks caused by resource leaks. There are many ways in safe Rust to deliberately leak memory - either by creating reference count cycles, or the explicit .leak() methods on various memory-allocating structures in std. It's also not entirely useless to do this - if you want an &'static from heap memory, Box.leak() does exactly that.

Now, that being said, actually writing code to hold a LockGuard forever is difficult, but that's mainly because the Rust type system is incomplete in ways that primarily inconvenience programmers but don't compromise the safety or meaning of programs. The borrow checker runs separately from type checking, so there's no way to represent a type that both owns and holds a lock at the same time. Only stacks and async types, both generated by compiler magic, can own a LockGuard. You would have to spawn a thread and have it hold the lock and loop indefinitely[0].

[0] Panicking in the thread does not deadlock the lock. Rust's std locks are designed to mark themselves as poisoned if a LockGuard is unwound by a panic, and any attempt to lock them will yield an error instead of deadlocking. You can, of course, clear the poison condition in safe Rust if you are willing to recover from potentially inconsistent data half-written by a panicked thread. Most people just unwrap the lock error, though.

croemer16• 4 months ago

I've had a look at the (vibe coded) repro linked in the article to see if it holds up: https://github.com/juxt/agc-lgyro-lock-leak-bug/blob/c378438...

The repro runs on my computer, that's positive.

However, Phase 5 (deadlock demonstration) is entirely faked. The script just prints what it _thinks_ would happen. It doesn't actually use the emulator to prove that its thinking is right. Classic Claude being lazy (and the vibe coder not verifying).

I've vibe coded a fix so that the demonstration is actually done properly on the emulator. And also added verification that the 2 line patch actually fixes the bug: https://github.com/juxt/agc-lgyro-lock-leak-bug/pull/1

ErroneousBosh3• 4 months ago

However, Phase 5 (deadlock demonstration) is entirely faked. The script just prints what it _thinks_ would happen.

I see this a lot in AI slop, which I mostly get exposed to in the form of shitty pull requests.

You know when you're trying to explain Test-Driven Development to people and you want to explain how you write the simplest thing that passes the test and then improve the test, right? So you say "I want a routine that adds VAT onto a price, so I write a test that says £20+VAT is £24, and the simplest thing that can pass that test is just returning 24". Now you know and I know that the routine and its test will break if you feed it any value except £20, but we've proved we can write a routine and its test, and now we can make it more general.

Or maybe we don't care and we slap a big TODO: make this actually work on there because we don't need it to work properly now, we've got other things to do first, and every price coming up as £20+VAT is a useful indicator that we still have to make other bits work. It doesn't matter.

The problem is that AI slop code "generators" will just stop at that point and go "THERE LOOK IT'S DONE AND IT'S PERFECT!" and the people who believe in the usefulness of AI will just ship it.

djmips3• 4 months ago

I think it's interesting that they found what seems to be a real bug (should be independantly verified by experts). However I find their story mode, dramatization of how it could have happened to be poorly researched and fully in the realm of fiction. An elbow bumping a switch, the command module astronaut unable to handle the issue with only a faux nod to the fact that a reset would have cleared up the problem and it was part of their training. So it's really just building tension and storytelling to make the whole post more edgy. And yes, this is 100% AI written prose which makes it even more distasteful to me.

jcalvinowens• 4 months ago

An elbow bumping a switch [..] really just building tension and storytelling to make the whole post more edgy.

A guarded switch, no less.

But personally I'm trying to be more generous about this sort of thing: it is very very difficult to explain subtle bugs like this to non-technical people. If you don't give them a story for how it can actually happen, they tend to just assume it's not real. But then when you tell a nice story, all us dry aged curmudgeons tut tut about how irreverent and over the top it is :)

Finding the middle ground between a dry technical analysis and dramatization can be really hard when your audience is the entire internet.

parliament32• 4 months ago

Both the article and repo[1] are slop.

[1] In the repo, the "reproduce" is just a bunch of print statements about what would happen, the bug isn't actually triggered: https://github.com/juxt/agc-lgyro-lock-leak-bug/blob/c378438...

bsoles• 4 months ago

Another CTO "published" an AI slop to get attention to their vibe-coded company that will disappear in two years. Tell me something new...

callamdelaney• 4 months ago

More likely the llm misinterpreted something and hallucinated an error. Just yesterday Claude code hallucinated itself an infinite loop.

garaetjjte2• 4 months ago

This article is garbage.

The Apollo Guidance Computer (AGC) is one of the most scrutinised codebases in history.

What? AGC programs were developed by relatively small team and pretty much left alone since then. Architecture is rather quirky when viewed with modern sensibilities. There's not much people that are familiar with it. Compare it to widely used software like libcurl or sqlite. Or perhaps to Super Mario Bros, which was extensively analyzed for competitive speedruns reasons. Surely that dwarfs amount of knowledge about Apollo code.

2K of erasable RAM and a 1MHz clock. The AGC’s programs were stored in 74KB of core rope

How about picking a unit and staying with it? AGC has 2K words of RAM, where each word has 15 bits of usable data (physically it's 16 bits, but one bit is used for parity). Maximum amount of ROM that could be installed is 36K words. (but they switch to KB, which is not only inconsistent with previous sentence but the number is also wrong! It's 72 KiB, 73.728 KB or 67.5 KiB, 69.12 KB depending whether you include parity or not) (maximum of 64K ROM words could be addressed by architecture design, but isn't available in any real hardware)

And yes, there is 1.024 MHz clock in the system, which is revelant for peripherals, but you probably want to know how fast it executed instructions. One memory cycle takes 11.71875 μs (85 1/3 kHz), and most instructions take 2 such cycles (one for operation, second for fetching next instruction) (each memory cycle is long enough for read from ROM, or read and write to RAM. ROM speed was the limiting factor, by standard of core memories it wasn't particularly fast. AGS backup computer used core for both RAM and ROM and had memory cycle time of 5 μs) (in case you are confused, "core memory" and "core rope memory" refers to quite different things!).

If you think I'm nitpicking, try writing an emulator and wondering why you have to sift through all that slop. You could give the correct numbers, you know?

“My secret terror for the last six months has been leaving them on the Moon and returning to Earth alone”, Collins later wrote of the rendezvous. A dead gyro system behind the Moon, with Armstrong and Aldrin on the surface waiting for a rendezvous burn that depends on a platform he can no longer align, is exactly that scenario. A hard reset would have cleared it. But the 1202 alarms during the lunar descent had been stressful enough with Mission Control on the line and Steve Bales making a snap abort-or-continue call. Behind the Moon, alone, with a computer that was accepting commands and doing nothing, Collins would have had to make that call by himself.

You know what an orbit is? That it goes around? That you could just wait for a while and speak with Mission Control? What even is this scenario? That your guidance system failed, and you for some inexplicable reason are considiering immediately leaving back for Earth right now leaving your pals behind? (with a manual burn, I guess, since guidance is dead?) You just wait for contact with Houston and tell them what happened. They pore over the program listings and find the bug. They radio you back appropiate VERB and NOUN commands for poking right values into memory. The End. And besides, spacecraft can be tracked and orbit determined from Earth, so even if the PGNCS did fail completely LM would just get necessary orbit information from Mission Control. (also in case guidance fails in either LM or CM, either one can have active role during rendezvous. And LM have extra backup system, the previously mentioned AGS)

The whole thing of "we found a minor deadlock bug in AGC program, what a shock!" is bizzare. It's not a small program. If you have any experience with software, of course you know it has bugs! They iterated on the software, releasing new software for most missions, adding new features, and, fixing bugs they found. What a concept!

thewonderidiot• 4 months ago

Mike Stewart here! I led the restoration of the AGC documented on CuriousMarc's channel and co-administrate VirtualAGC. There is a lot to unpack here.

First: this is indeed a real bug in the AGC software. However, it did not go unnoticed for the whole program. It was discovered during level 3 testing of SATANCHE, and late development branch of the Command Module software COMANCHE. It was assigned anomaly number L-1D-02, and was fixed between Apollo 14 and 15. There are two known surviving copies of the L-1D-02 anomaly report:

* https://www.ibiblio.org/apollo/Documents/contents_of_luminar...

The fix described in the article is partially complete, but as noted in the anomaly report there's a little bit more to it. Rather than just adding the two instructions to zero LGYRO, they restructured the code a bit and also cause it to wake up pending jobs. You can compare the relevant sections of the Apollo 14 and Apollo 15 LM software here:

* Apollo 14: https://github.com/virtualagc/virtualagc/blob/master/Luminar...

* Apollo 15: https://github.com/virtualagc/virtualagc/blob/master/Luminar...

The bug would not manifest silently in the way described in the article. For starters, LGYRO is also zeroed in STARTSB2, which is executed via GOPROG2 on any major program change: https://github.com/virtualagc/virtualagc/blob/master/Luminar...

This means that changing from any program to any other program would immediately resolve the issue. This is almost certainly a large part of why it took them so long to notice. Hitting BADEND while actively pulse-torquing is quite rare, and avoided by normal procedure. The scenario presented in the article can't happen since the act of starting P52 will zero LGYRO.

Moreover, in the very specific scenarios in which the bug can be triggered and remain, it results in multiple jobs stacking up attempting to torque the gyros. Eventually the computer runs out of space for new jobs -- similar to what happened on 11 -- and a 31202 (the Apollo 12+ equivalent of 1202) is triggered.

Since the issue was found before the flight of Apollo 14, a further description of how it might occur and what the recovery procedure should be was added to the Apollo 14 Program Notes: https://www.ibiblio.org/apollo/Documents/LUM159_text.pdf#pag...

Some other notes:

Ken Shirriff has analysed it down to individual gates

I've done the bulk of the gate-level analysis. :)

the Virtual AGC project runs the software in emulation, having confirmed the recovered source byte-for-byte against the original core rope dumps.

We've only been able to do that in very specific circumstances and only for subsections of assorted programs, but never for a full program. Most AGC software either comes from a program listing, from a core rope dump, or from reconstruction using changelogs and known memory bank checksums. We've disassembled all of the rope dumps into source files that assemble back into the same binary, but the comments and labels will be different from what was in the original listing. And to be extra clear: I've never had the opportunity to dump a module containing Apollo 11 software for either vehicle. Our sole source for both programs is a pair of printouts in the MIT Museum's collection.

Margaret Hamilton (as “rope mother” for LUMINARY) approved the final flight programs before they were woven into core rope memory.

Jim Kernan was the rope mother for Luminary at least up through Apollo 11. Margaret was the rope mother for Comanche, the CM software, and was later promoted to lead the software division. Their positions at the time of 11 can be seen on this org chart: https://www.ibiblio.org/apollo/Documents/ApolloOrg-1969-02.p...

Their priority scheduling saved the Apollo 11 landing when the 1202 alarms fired during descent, shedding low-priority tasks under load exactly as designed.

This is a huge topic on its own, but the AGC software was not designed to shed low-priority jobs. Ironically, the lowest priority job during the landing was the landing guidance itself, with high-priority jobs being reserved for things that needed quick response like antenna movements or display updates. If the computer were to shed the lowest-priority jobs, it would shed the landing guidance. This memo contains a list of all jobs active during the landing and their priorities: https://www.ibiblio.org/apollo/Documents/CherryApollo11Exege...

For example, the ICD for the rendezvous radar specified that two 800 Hz power supplies would be frequency-locked but said nothing about phase synchronisation. The resulting phase drift made the antenna appear to dither, generating roughly 6,400 spurious interrupts per second per angle and consuming roughly 13% of the computer’s capacity during Apollo 11’s descent. This was the underlying cause of the 1202 alarms.

The frequency-lock prevents phase drift, so the phase is essentially fixed once the power supplies are up. Ironically, however, the bigger issue is that one reference was 28V while the other was 15V. Initial testing on actual Apollo hardware suggests that at least for Apollo 11, this voltage difference was the key contributor rather than the phase difference: https://www.youtube.com/watch?v=dT33c70EIYk

password4321• 4 months ago

The front page has moved on but this is pure gold, thanks for making the time to share all these details.

dang• 4 months ago

Added to https://news.ycombinator.com/highlights!

(I mention this so more people can know the list exists, and hopefully email us more nominations)

jacquesm2• 4 months ago

I was hoping you'd comment here. Thank you. Amazing bits of lore.

replwoacause• 4 months ago

One of the coolest replies I've ever seen on HN for sure. Thanks for taking the time to write this out!

croemer16• 4 months ago

Great comment!

The anomaly report pdf is quite long. The relevant pages for the bug report are 51 and 52: https://www.ibiblio.org/apollo/Documents/contents_of_luminar...

croemer16• 4 months ago

Waiting patiently for the blog post to be edited as it's now clear it makes wrong claims starting with the subheading: "How a specification found what fifty-seven years of scrutiny missed."

croemer16• 4 months ago

Update has happened

henrygarner• 4 months ago

Mike, thank you so much for such patient and thoughtful corrections. I've updated the article in response to each point you raised and added footnotes crediting you and Ron Burkey, who shared some of the same feedback via email. It's been extremely cool to learn more about the Apollo program lore in the process of pulling this all together.

Your experimental work on the voltage difference is fascinating, I appreciate you sharing the link to the demo. There's something about seeing results come off real hardware that you just don't get from computer simulation. Watching your setup brought back some of the excitement I felt driving my Makerbot via a stylus input 15 years ago (probably the last time I seriously engaged with hardware) [1]. Thanks!

[1] https://www.youtube.com/watch?v=RNvt9PGhnds

thiht• 4 months ago

We used Claude […]

Ughhhh… I know this is probably legit here, but reading these words make me lose interest sooo fast these days…

We found an undocumented bug in the Apollo 11 guidance computer code

Related Stories