FAA NOTAM System Outage

https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=13&adv_date=01112023&facId=DCC&title=NOTAM+SYSTEM+EQUIPMENT+OUTAGE_FYI&titleDate=01/11/23

by chucksmash • 4 years ago

422 198 4 years ago

fly.faa.gov

zrail2• 4 years ago

As of 06:24 US Eastern the appears to no longer be true. There is a NOTAM outage but no full ground stop is mentioned, just a few specific due to weather.

Edit: obviously a developing situation, see responses to this comment for current advisories.

emn132• 4 years ago

https://www.reuters.com/business/aerospace-defense/us-faa-sa...

Clearly still developing; we'll see.

callahad2• 4 years ago

I don't know how these things work, but the most recent advisory, 027, seems to imply that United Airlines has a full ground stop until 15:00 UTC / 10:00 Eastern.

Edit: Of course as I was typing that, Advisory 028 pops up with a nationwide ground stop until 9:30 Eastern: "GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS"

https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...

perihelions• 4 years ago

- "All US domestic flights have been grounded for several hours because of a glitch with the flight control system."

https://www.bbc.co.uk/news/world-us-canada-64236047

chucksmash3• 4 years ago

Yep, the situation has changed since the post was made. Parent comment was correct at the time, however.

At 5:40am Eastern, the agent at my departure gate told us that there was a ground stop affecting all flights, which was reflected in my original title. This was not reflected in the NOTAM I posted, as was pointed out by several early commenters, and in fact my flight was able to board and depart around 7:00am Eastern before the ground stop took effect.

nexus75563• 4 years ago

Any source for the claim of a ground stop? I see plenty of flights still taking off.

kzrdude2• 4 years ago

The most recent one seems to say something like that. ATCSCC ADVZY 023 DCC 01/11/2023 ALL UAL MAIN AND SUBS GROUND STOP

Not that I know how to read it. The event time is three minutes ago.

https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=23&adv_dat...

UhUhUhUh• 4 years ago

Reads United Airlines apparently.

nexus75563• 4 years ago

My understanding is that applies to United Airlines and it’s regional partners. It seems each airline is making their own decision.

This Reddit post seems to have more info: https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_...

eftychis• 4 years ago

Air Traffic Control System Command Center Advisory 23 by the Traffic Management Command Center (DCC) at date [..] - all united airlines ... ground stop.

Here is about ground stops: https://www.faa.gov/air_traffic/publications/atpubs/foa_html...

alamortsubite2• 4 years ago

I can see that East Coast UA flights are currently delayed because of this.

InsomniacL• 4 years ago

MESSAGE:

EVENT TIME: 11/1126 - 11/1500 UAL GROUND STOP ALL FLIGHTS / ALL DESTINATIONS PER USER REQUEST DESTINATION AIRPORT; ALL FACILITIES INCLUDED: ALL GROUND STOP PERIOD: UNTIL 1500Z REASON: USER REQUEST REMARKS: US NOTAMS SYSTEM DOWN

https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=23&adv_dat...

https://www.fly.faa.gov/adv/advADB.jsp

nexus75563• 4 years ago

UAL = United Airlines, not “all North American flights” as OP claimed

chucksmash3• 4 years ago

This was based on a P.A. announcement from a gate agent, so not authoritative. I've updated the title to reflect what is in the link.

callahad2• 4 years ago

Advisory 028 is a nationwide ground stop: https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...

sneak• 4 years ago

Nothing in the linked page confirms the ground stop mentioned in the title.

Reddit comments say that there is no FAA ground stop, but airlines are making individual decisions about it.

https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_...

Klonoar• 4 years ago

There was a brief ground stop (I'm currently flying and it momentarily held us up) but Delta at the least isn't stopping entirely.

emn132• 4 years ago

https://www.reuters.com/business/aerospace-defense/us-faa-sa...

harrego2• 4 years ago

I really like the advisory page -- all info, no sugar.

gonzo41• 4 years ago

https://lite.cnn.com/en

It's the best way to get the news. No filler, no adds and it works on a 2g mobile.

ceejayoz9• 4 years ago

See also: https://text.npr.org/

kfrzcode4• 4 years ago

Not sure CNN is the "best" way to get the news.

Try this: https://en.wikinews.org/wiki/Main_Page

No ads, no adds, no filler, no filters.

Make the news objective again. Repeal Smith-Mundt.

https://www.congress.gov/bill/112th-congress/house-bill/5736

kfrzcode4• 4 years ago

harrego2• 4 years ago

http://68k.news is a fun one too.

chucksmash3• 4 years ago

Updated title from "FAA Ground Stop for All North America Flights Due to NOTAM Outage" to reflect comments that initial title was not accurate.

xattt• 4 years ago

Confusing title, as Canadian departures with Canadian destinations do not appear to be affected.

testplzignore• 4 years ago

If I'm understanding the situation correctly: the system stopped processing updates yesterday afternoon and it wasn't a big deal, then they tried restarting it this morning to fix it and it didn't come back up, so now older NOTAMs from before the issue started are not available. Is that right?

Do airlines/planes keep a cached copy of these? How large is the NOTAM database?

stevehawk• 4 years ago

Not really. And for most commercial pilots this may come down to whether or not ForeFlight caches theirs or not. I'd actually be really curious to see unofficial commentary from ForeFlight developers about this because it's a pretty unique scenario in the age of electronic flight bags.

p_l2• 4 years ago

The problem isn't keeping a copy, it's sending updates that are very time critical (for example restricted areas around POTUS, any impactful event, etc.)

AYBABTME• 4 years ago

Are 2020s the years where the cold war is back as a lukewarm war, and also all of NA aviation needs to reconvene with its decrepit software? Next plot-twist, it was all ready to blow, but ultimately triggered by Russian/Chinese/North Korean hackers.

Ok, taking off my tinfoil hat. Still, maybe I'm suffering of nostalgia bias but the last few years really make for a lot of prime source material for docudramas.

RagnarD• 4 years ago

No need for a tinfoil hat. Such systems are prime targets for states hostile to the U.S. If this one wasn't before, it will be now.

heisenbit• 4 years ago

Forensic analysis will reveal that there were multiple hidden failures over the past years that were covered up by foreign hackers to ensure their backdoors were not revealed in an in-depth investigation.

peteradio• 4 years ago

Free maintenance!

yehosef3• 4 years ago

don't forget about this beauty

https://techcrunch.com/2021/02/23/solarwinds-hackers-targete...

ceejayoz9• 4 years ago

https://twitter.com/AUStinAirport/status/1613134228321087488

An FAA system outage is causing ground stops at AUS and other airports across the country. Arriving & departing passengers can expect delays this morning & through the day.

If you’re flying today, it’s gonna be a rough one.

edit: BBC now confirming a full ground stop. https://twitter.com/BBCBreaking/status/1613151796955856896

https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...

yehosef3• 4 years ago

in unrelated news - https://techcrunch.com/2021/02/23/solarwinds-hackers-targete... ;) /humor

oldgradstudent• 4 years ago

Next time my server is down I'll blame the Russians and Chinese too.

yehosef3• 4 years ago

If your org was hacked by the Russians/Chinese and was a national safety system, then you might be right to start there.

kzrdude2• 4 years ago

ATC can tell pilots what they need to know, right?

dx0345• 4 years ago

The question is, how will ATC know? I'm not that familiar with their systems, but maybe they rely on them to get critical information?

ceejayoz9• 4 years ago

ATC will be fine to get everyone on the ground safely. There are procedures for everything including ATC itself being down.

McDyver• 4 years ago

NOTAM notices are not only for planes but for other people and equipment using airspace, like drones, skydivers, ... Those are not always connected with ATC, but need the information from NOTAM to plan their activities

cjrp3• 4 years ago

The NOTAMs for a commercial flight from JFK to LAX, for example, would be several pages long. Warnings about construction work at either airport, closed taxiways, cranes in the vicinity of the airport, enroute navigational aids which are unusable, etc. ATC can't relay that for every departing flight. Additionally, the NOTAM system is where they'd retrieve these from, so if that's down they have nothing to relay.

VLM3• 4 years ago

Its a flight planning mystery.

Lets say you are not flying to, but flying by, my hometown airport. You decide arbitrarily to use it as a divert in case of awful weather or engine trouble enroute or you just want a bathroom break (this is more of a Cessna 172 airport than a 747 airport)

I found a cached list of nine current NOTAMs for my home airport. Because its always better to overinform than to underinform, until the system overloads and crashes:

A long ramble that boils down to you can't use one of the IFR plates for a helicopter approach because of (its a very long story but relates to the crane mentioned below)

The modulation level on the closest VOR for identification/verification is low, although in legal tolerance, so you can use it but please stop trying to open the equivalent of a trouble ticket. We know the volume is both technically legal although low and not "turned up all the way". The techs say its safe until new parts arrive.

There's three failed tower lights in the area (three? Supply chain problems? Labor problems? WTF?) and one temporary crane obstruction. The crane is flagged and lit, and could theoretically be 60 ft AGL which is normally below the legal limit but its on airport grounds 0.13 miles away from one runway off to the side, so please try not to hit it. Also please don't bother reporting a crane off to the side of the approach; we know all about it...

Clearance delivery for the "big city" airport nearby has a remote trans/rec site nearby that's broken. Legal coverage is adequate (kind of a RAID array of retrans sites) so please stop reporting the reduced signal strength, we already know.

There's some kind of flood on runway 10 when it very heavily rains; the groundskeepers will fix the drainage problem; the NWS reports 0% chance of rain today so it probably won't be an issue, but don't act all surprised. This is not a NOTAM for groundskeeper work, that would be a separate new NOTAM.

briandear2• 4 years ago

Nailed it exactly. Also be advised that birds are in the vicinity of the airport.

someguydave3• 4 years ago

Is someone who is planning multiple divert airports seriously expected to retain all of that information and recall it in the case of an emergency?

Seems like 10% of that info might be kinda useful to know and 90% of it is “BY FLYING IN OUR DIRECTION YOU AGREED TO ACCEPT …”

dx0345• 4 years ago

There's a ground stop for all flights now.

GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS DESTINATION AIRPORT; ALL FACILITIES INCLUDED: ALL GROUND STOP PERIOD: UNTIL 1430Z REASON: EQUIPMENT OUTAGE REMARKS: US NOTAMS SYSTEM DOWN

srmarm• 4 years ago

This might be a silly question, but is that a NOTAM saying that NOTAM system is down? How does it get out?

zrail2• 4 years ago

As I understand it there's a hotline and periodic teleconferences and webinars today that for sure everyone at the airlines is participating it.

ceejayoz9• 4 years ago

The website and the system that gets it out to all the planes are related but not the same.

capekwasright• 4 years ago

This is an Air Traffic Control System Command Center (ATCSCC) advisory: https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...

Buffout• 4 years ago

NOTAM is a notice to alert aircraft pilots of potential hazards along a flight route...

Command control can see that NOTAM system is out and publish advisory. (Not via NOTAM system)

ape4• 4 years ago

Reminds me of that screenshot of a windows popup unable to display an error.

Tepix• 4 years ago

Why don't they use lower case letters?

yrro• 4 years ago

https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...

josephd792• 4 years ago

Gonna be a bad day for those sys admins

causality0• 4 years ago

May none of our fuckups be bad enough to get an article written about them. Amen.

josephd792• 4 years ago

Amen.

chasd00• 4 years ago

heh, my boss was on a public sector project that went so far south she was stalked by news crews leaving the office. The experience left a pretty deep scar on her. This was before my time thank babyjeebus.

acdha2• 4 years ago

Kinda wondering the odds of there being one crying into their coffee over a stack of memos saying they should do something before it fails, or some greybeard angrily pointing to their quality control concerns on a new system. Both are pretty common with federal IT.

nkcmr• 4 years ago

Why do all aviation IT systems feel like they are so fragile?

It feels like not a single month can pass these days without some "Airline X has major outage and ruins tons of people's days!" story on the news.

What is the tech behind these frail systems?

dx0345• 4 years ago

More modern systems are also down once in a while. With aviation it's just more visible because one system can affect a lot of people.

I'm not sure when ATC systems (which I would count NOTAM to be one of) have last failed, I always had the feeling they're very stable. It's usually airlines that struggle with their applications.

joshuahedlund• 4 years ago

I never know how well it applies but headline outages always make me think of that “The Coming Software Apocalypse” Atlantic article from a few years back[0]

[0] https://www.theatlantic.com/technology/archive/2017/09/savin...

namdnay• 4 years ago

I think it's just a question of media visibility. Aviation outage = queues in airports = easy article to write

VLM3• 4 years ago

All hyper optimized systems become fragile until they collapse.

The NOTAM system is ridiculous, no surprise it collapsed ... you can lose your ticket for flying thru a MOA that went active AFTER you took off.

What could possibly go wrong with a "push" notification system for every tiny little detail of aerospace scheduling? No force is as destructive as the good idea fairy.

trelane• 4 years ago

For those who also didn't know, MOA: https://tfmlearning.faa.gov/publications/atpubs/AIR/air2501....

sokoloff2• 4 years ago

I'm not sure your specific example is a particularly good one. I can legally fly through an MOA that's active, without needing ATC or military authorization.

yuppie_scum2• 4 years ago

They need Kafka

briandear2• 4 years ago

MOAs aren’t the same as restricted areas. You can fly in them if you want.

ilyt• 4 years ago

That's the curse of niche software. Especially one made to fit, it's easy for penny pincher to decide "it works fine now, it doesn't need ongoing improvements" and you end up with some ancient shit needing java 1.7 (or worse) to run with no plan to upgrade.

yuppie_scum2• 4 years ago

Airlines view technology as a cost center.

ceejayoz9• 4 years ago

This is the FAA's system, not the airlines'.

tyingq2• 4 years ago

The software is tied to a dependency chain of real people (passengers, crew, etc), real equipment, and lots of constraints. So problems have a downstream effect that's more pronounced and visible than many other spaces.

lovecg• 4 years ago

When was the last time NOTAMs went down? It seems remarkably resilient actually.

samwillis• 4 years ago

Edit 2 - * Ground stop Lifted * Update from FAA:

Update 5: Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted.
We continue to look into the cause of the initial problem

https://mobile.twitter.com/FAANews/status/161317140080139468...

---

Edit - Update from FAA:

Update 4: The FAA is making progress in restoring its Notice to Air Missions system following an overnight outage. Departures are resuming at @EWRairport and @ATLairport due to air traffic congestion in those areas. We expect departures to resume at other airports at 9 a.m. ET.

https://mobile.twitter.com/FAANews/status/161316273572174643...

---

Flightradar24 are reporting:

Flights to US airports have been groundstopped by the FAA until at least 09:30 ET (1430 UTC). This means any flight not in the air already will not be allowed to depart. Image: flights over US now compared to same time last week. About 700 fewer flights airborne now.
Flights are being held as the FAA works to bring the NOTAM (NOtice To Air Missions) system back online.

https://mobile.twitter.com/flightradar24/status/161315347921...

Latest FAA update:

Update 3: The FAA is still working to fully restore the Notice to Air Missions system following an outage.
The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information.

https://mobile.twitter.com/FAANews/status/161314857927045939...

Statement from the white house:

The President has been briefed by the Secretary of Transportation this morning on the FAA system outage. There is no evidence of a cyberattack at this point, but the President directed DOT to conduct a full investigation into the causes. The FAA will provide regular updates.

https://mobile.twitter.com/PressSec/status/16131535612899328...

I believe this means that nothing can take off, but aircraft already in the air can continue to their planned destinations.

dx0345• 4 years ago

Looks like they published some kind of a "master NOTAM" now with all important information to restart as soon as possible.

https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=31&adv_dat...

FiloSottile• 4 years ago

That's a routine ATCSCC.

rwmj• 4 years ago

Fun stuff about NOTAMs: https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...

bombcar8• 4 years ago

Fun but deadly. They’re not wrong about the absolute avalanche of irrelevant data that will be used to show you “you should have known” - hence the proliferation of “interpretation” services.

azherebtsov4• 4 years ago

But it’s up to airline how they present NOTAMS to the pilot in the briefing package. They can (and do) use geolocation present in some (many) NOTAMs to filter them out. For example do not show the NOTAM just because the flight path intersects with the airspace. They can do even better - consider not only location but also altitude range, the best flight planning systems can consider also estimated time of arrival of the plane near the location of the NOTAM and filter out those which are not yet valid or those which are expired (the validity range usually present in each message). This makes the problem of including the NOTAM a 4-dimensional - geographical location 2 dimensions, altitude 1 dimension and a time. Not the easiest to solve, but solvable. Of course for every dimension margins can be applied to accommodate re-routes or departure delays, or ATC commands. There is no written in stone standard of how to present NOTAMS. Newer may be displayed at the beginning, etc Not to say that they can (and do) separate airfield and airspace messages to segment the large set onto meaningful subsets. I mean there ways to make sure important information is not lost and less important available if needed.

maxcan2• 4 years ago

100%. As a GA pilot who mostly flies under IFR, the NOTAM system is borderline useless to me.

For every warning about an approach being altered or a runway closed, there are 100s of "birds invof airport" or "unlit crane 50' AGL 5 miles from the field" NOTAMs. And even for the relevant NOTAMs, Foreflight will highlight them for approaches etc and ATC won't send you to a closed runway or taxiway.

marcosdumay• 4 years ago

Interestingly, that seems to be a US-only problem.

lucasisola• 4 years ago

This is my favorite quote: "In an unintended twist of irony, the agencies seeking to cover their legal ass are party to creating the most criminal of systems – an unending flow of aeronautical sewage rendering the critical few pieces of information unfindable."

eternalban2• 4 years ago

"There are three parts to the problem: the system, the format, and the content. The system is actually quite amazing. The AFTN network connects every country in the world, and Notam information, once added, is immediately available to every user. Coupled with the internet, delivery is immediate.

The format is, at best, forgivable. It’s pretty awful. It’s a trip back in time to when Notams were introduced. You might think that was the 1960’s, or the 50’s. In fact, it’s 1924, when 5-bit ITA2 was introduced. The world shifted to ASCII in 1963, bring ing the Upper and Lower case format that every QWERTY key board uses today, but we didn’t follow – nope, we’ll stick with our 1924 format, thank you."

https://web.archive.org/web/20230104130153/https://www.faa.g...

"The FAA is changing the format for Notice to Air Missions (NOTAM) to align with international standards. The transition to the new format will ensure U.S. NOTAMs are compliant with standards set by the International Civil Aviation Organization (ICAO) and promote further global harmonization among neighboring Air Navigation Service Providers (ANSPs)."

ICAO NOTAM format

https://vat-air.dk/files/ICAO%20NOTAM%20format.pdf

Example! (decoder ring not included)

"Q) EGTT/QMRXX/IV/NBO/A/000/999/5129N00028W005"

mzs• 4 years ago

I want to point out that lower-case is possible with ITA2. Start at 4.6 of ITU-T Recommendation S.2 / 11/1988 in the Blue Book, basically 11111 & 11011 11111 switch case in locking/unlocking mode.

wstuartcl4• 4 years ago

Its even more amazing when you really sit down and think about all of the systems (many highly resistant to change like aircraft navigation systems) that all connect and utilize this service.

What appear like simple changes and proposals in the aerospace realm are extremely hard to execute and many times take decades (multiple) to come into play. Global services/systems are even harder.

xg15• 4 years ago

OpsGroup, the organisation behind that guide is also worth mentioning. It's a self-organised advocacy organization for flight operations personnel. Apparently they have a longstanding campaign to reform the NOTAM system (as explained in the guide).

They also have a summary of today's incident: https://ops.group/blog/us-flights-delayed-after-notam-system...

NelsonMinar2• 4 years ago

This is hilarious. Really appreciate the examples of irrelevant clutter obscuring important information. Any UX designer can learn from this as an example of what not to do.

reaperducer• 4 years ago

Very entertaining to watch the local TV news this morning.

The reporter keeps pulling people aside as they arrive at the airport and asking them what they're going to do with all the planes grounded.

None of them know it's happening.

So far, they have all admitted that they either don't look at the news, or don't look at the local news, except for one woman who defiantly said, "It wasn't on Facebook," as if that meant that since the grounded flights weren't in her social media feed that it wasn't real.

unionemployee2• 4 years ago

The NOTAM system is embarrassing. This just tops it off. Actually, our whole ATC system and the way the FAA operates is embarrassing. It all works kind of like waste management in NYC. If they were to add new tech and efficiency, fewer people would have jobs, so we keep doing it the old way to the detriment of everyone else.

acdha2• 4 years ago

It’s easy to say things are about jobs but do we have any evidence of that? There’s a modernization project underway at the FAA so I’d bet that if you looked there’s someone who’s been trying for years to get attention on it before something like this happened.

kfrzcode4• 4 years ago

Shirley you must be joking.

thesuperbigfrog• 4 years ago

Rumack : I won't deceive you, Mr. Striker. We're running out of time.

Ted Striker : Surely there must be something you can do.

Rumack : I'm doing everything I can... and stop calling me Shirley!

quantified• 4 years ago

What embarrasses you about it, esp compared with more modern systems?

This seems like the kind of critical infrastructure whose uptime requirements can make upgrading challenging once in place.

unionemployee2• 4 years ago

Others have said it better. If you google 'notams are broken', numerous results will show up explaining various problems with the NOTAM system.

Edit: Here's a good one - https://fixingnotams.org/ending-notam-nonsense-in-the-digita...

A general inefficiency example using two airpots I often have in mind: IAH and JFK. In IAH, traffic is spaced out in cruise well before the arrival begins, where they're burning much less fuel at high altitude. They're then instructed to descend via the full STAR (standard terminal arrival), including the transition to the approach for the runway in use. This looks like a hundreds mile long line of planes all following the same path to the same runway. The only further instruction necessary from ATC is speed adjustment, and allows for an almost continuous descent to the runway provided there's no weather in the way.

At JFK, the spacing occurs much later and, even though the airport has some similar arrivals (though none quite as thorough), they instruct every altitude change and speed adjustment for every aircraft individually, along with vectoring them to the approach for the runway in use. It results in an incredibly busy radio environment and numerous inputs from pilots flying planes capable of simply following a STAR and transition, like in IAH (or ORD, SFO, LAX, etc). There's also much more time spent at low altitude burning more fuel. It reminds me of everything else in NYC, which seems to be inefficient on purpose (NYPD, MTA, DSNY, etc.).

JFK and IAH have a similar number of aircraft movements. Yes, NYC has busier airspace with numerous airports, however, I don't see how this prevents the design and implementation of more modern procedures. Flying around at 4000 feet in an airliner getting numerous vectors just seems ridiculous compared to what I experience elsewhere.

dev_daftly• 4 years ago

So basically it's the same as every other government entity?

watersb3• 4 years ago

REALLY IMPORTANT NOTAMS tend to alert pilots to RESTRICTED AIRSPACE or AREAS OF UNUSUAL AIR OPERATIONS. You can't fly anywhere near the President of the United States, the movements of which can change from day to day.

We get NOTAMS about military aircraft operations; often military airspace which is usually permitted may be shut down for a week or so.

Wildfire tanker operations or other natural emergencies can close an airspace.

It's quite dynamic, all on top of the scheduled flight plans and routes.

ru5522• 4 years ago

They are also for notifying pilots of potential caution areas.

I used to operate a 550+ meter tower and when I had a tower light out, I had to file a NOTAM.

watersb3• 4 years ago

Cool, you filed NOTAMs; I just read them.

If you like lots of daily mail, become responsible for the operation of a Cessna 172 (Skyhawk), the type of aircraft with the most units built of any, ever. Endless variations.

Unending stream of paper NOTAMS via US first class post office... at least, when I had a 172, 20 years ago.

These notices are exactly like a recall notice on an automobile: get someone to validate the steering wheel etc. I would suppose that many HN readers have gone through the car hardware remediation process.

I would suppose that this class of NOTAM, although in my particular case the most common, are not immediately relevant to a total Ground Stop Order for the USA. But -- consider that airline and other commercial operators may have their own internal requirements that could ground a fleet in the absence of an ability to state that they have complied with all relevant FAA NOTAMS. Oh, the liability!!

watersb3• 4 years ago

(Holy cow, just did remedial maths: 550m is a Large Number! Would like to know more details.)

jxf• 4 years ago

Q: Is there anything like a systems architecture diagram for how NOTAM works? I'm curious about which parts might have failed and what the moving pieces are.

tyingq2• 4 years ago

Some here, mixed in with other info:

https://www.faa.gov/air_traffic/technology/swim/users_forum/...

https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%...

sumanthvepa• 4 years ago

Was it database failure or did they get hacked?

snake_doc• 4 years ago

For anyone looking for previous FAA outages: In 2014, an U-2 plane’s flight plan crashed LAX’s ERAM system because the system ran out of memory.

https://www.oig.dot.gov/sites/default/files/FAA%20Actions%20...

GalenErso• 4 years ago

Why do secretive spy planes need to file flight plans?

ceejayoz9• 4 years ago

Because they're not always doing secretive stuff, and even their secretive missions may start and end in airspace they have to share with friendly airliners.

Sometimes it's a deliberate "we see you", too. https://twitter.com/Gerjon_/status/1475444704250630148

pmontra• 4 years ago

That was a training flight, it's in the PDF.

controlller• 4 years ago

LAX, an air traffic control tower, does not employ ERAM. Los Angeles Center (ZLA), the overlying air route traffic control center, does.

steponlego2• 4 years ago

They're resuming operations despite "still searching for a cause." Comforting.

ceejayoz9• 4 years ago

You've never rebooted a computer to get it working again, without immediately knowing why it crashed?

virusuy• 4 years ago

I made that my career

steponlego2• 4 years ago

Not since I stopped using Microsoft Windows.

belter• 4 years ago

https://twitter.com/FAANews/status/1613171400801394688

" Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted...We continue to look into the cause of the initial problem"

jerpint• 4 years ago

Looks like it’s been fixed:

https://twitter.com/BBCBreaking/status/1613174866047340545

sandworm1012• 4 years ago

I've got a trip to take in a couple weeks. I have a choice: 4 hours of driving followed by two regional flights. Or just forget flying and drive 16 hours point to point. Between today's issues and all the headaches at YVR over the holidays, driving 16 hours seems more comfortable and less stressful than rolling the dice with regional airline travel.

I would rather be stuck driving my car for a day than spend a day sleeping on the floor of an airport lobby.

outericky• 4 years ago

Depending on the length of the stay and whether you need a car, I'd do the road trip in a heartbeat.

mgkimsal• 4 years ago

Whenever I opt for 'roadtrip', almost inevitably, about 4-5 hours in, I wish I were flying. When I fly, I don't usually say "oh, I wish I'd driven" (unless there's a delay or problem).

As I've driven more, I need to allot myself more stops. If a trip is over 10 hours, I'll probably split it in 2 days, and stay overnight some place. 20 years ago, I would have 'pressed on', but... it's usually not worth it any more. But some of the 'cost savings' of driving go away when you factor in hotels.

toast0• 4 years ago

How many people in the car and how many drivers? What do you expect for the weather? Anywhere you want to stop somewhere in the middle?

Probably less effort and time to fly, but a lot of waiting without agency.

lost_tourist• 4 years ago

99% chance this won't happen to your flight.

sandworm1012• 4 years ago

It happened last time on a nearly identical itinerary. First flight was late. Missed connection. Landed at second airport too late for the bus (military to base where my car was parked). Had to spend night in hotel waiting for bus the following day.

Time before that, I arrived on time but my bag did not. Had to wait at airport for six hours for bag. Missed bus. Hotel. Bus the following day.

buffington• 4 years ago

I used to need to be in a city every other week that was only a 1.5 hour flight.

I decided to see if driving might get me there in comparable time if I compared when I closed my front door to when I opened the door of my destination.

I was surprised to find that the time it took as almost the same. That's because flying requires:

* start drive to airport (45 minutes) * park and get on a shuttle (15 minutes) * get to gate (30 minutes) * wait (60 minutes because I'm early by an hour) * fly (1:45) * taxi to gate (5-45 minutes) * walk to light rail stop (10 minutes) * ride light rail too stop (45 minutes) * walk to destination (15 minutes)

Door to door flying: about 6 hours

Driving:

* start driving (6 hours) * fuel stop (15 minutes) * park (15 minutes) * arrive

If I didn't obey speed limits, I could cut an hour or more off that 6 hours.

That said, I still preferred flying. Hard to take a nap or space out while driving for 6 hours.

ineedasername5• 4 years ago

I'm surprised there's not more activity on this here on HN. I guess info is limited, so I found what I could from ATCSCC notices for today [1]. These are all the notices but portions only w/ redundant bits removed. See link in citation if you want full messages. Times here appear to be GMT:

    004 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 00:47
        THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT
        OUTAGE ISSUES FOR THE UNITED STATES NOTAM SYSTEM.

    013 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 04:18
        THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z.  SINCE THEN NO NEW
        NOTAMS OR AMENDMENTS HAVE BEEN PROCESSED.  TECHNICIANS ARE CURRENTLY
        WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR
        RESTORATION OF SERVICE AT THIS TIME.

    014 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 04:19
        HE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT
        OUTAGE ISSUES...

    015 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 07:42
        THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE
        EQUIPMENT...

    016 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 07:44
        THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z.  TECHNICIANS ARE
        CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR
        RESTORATION OF SERVICE AT THIS TIME. THERE IS CURRENTLY A HOTLINE IN
        EFFECT WHICH HAS NAIMES/FAA FACILITIES/STAKEHOLDERS IN ATTENDANCE.

    020 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 10:58
        THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED...

    030 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 12:37
        THE NOTAM OUTAGE HOTLINE HAS BEEN TERMINATED.

In short, the terminology being used so far is "equipment failure".

[1] 1/11/2023 link to the day's messages w/ index numbering for each one per my above quotes" https://www.fly.faa.gov/adv/adv_list.jsp?WhichAdvisories=ATC...

zitterbewegung• 4 years ago

Probably because it’s not about a package manager , a social network, Cloud Service and a chat application.

eternalban2• 4 years ago

[heads-up: 'r/He\ is/I\ am' in your personal landing page]

ineedasername5• 4 years ago

I wanted to keep my above comment to the data only, so I'll post this here: "Equipment Failure" can cover just about anything. I think the "gut" reaction the public might have here when hearing that phrase is that it was physical equipment, but really it could have been software, network linkages, really anything. I hope the real cause here doesn't get buried somewhere in lack of follow up.

Also I really wish I'd been awake when the hotline was active (# was in the full notices linked) so I could call in and see what they were talking about as a silent observer. I can't tell from the context if that was a general status hotline, a failover service to provide the same data NOTAM would have done (though if that were the case then grounding flights wouldn't have been necessary) [1] or discussion of the actual problem and specific efforts to resolve.

[1] Well, only if it was a more benign & limited cause that wouldn't be expected to spill over into other systems. If the nature of the problem included the possibility of a malign actor then other vectors of attacks would not have been (and maybe still aren't, if that was the source) off the table.

jjav• 4 years ago

I wanted to keep my above comment to the data only, so I'll post this here: "Equipment Failure" can cover just about anything.

Stuxnet was after all an equipment failure. Sure, with an interesting plot to get there.

(Just commenting on the term "equipment failure", don't know anything about the FAA failure of today.)

sidfthec• 4 years ago

The limited activity is probably because it's actually not that exciting from a technical perspective.

The answer is almost certainly something like "lack of funding forces FAA to continue using massively outdated systems and this is what happens when someone trips over the power cord of the mainframe in some basement".

It's much more interesting when companies like Google or Facebook go down, because they properly fund their infrastructure and there might be something for the HN crowd to learn.

eigenform• 4 years ago

Space weather?

adolph• 4 years ago

Where are the blockchain advocates?

This seems like a perfect use case for a distributed ledger.

TecoAndJix• 4 years ago

In case you were like me and did not know what a NOTAM is: https://www.faa.gov/about/initiatives/notam/what_is_a_notam

dreamcompiler• 4 years ago

TIL that NOTAM now stands for "Notice to Air Missions."

Back when I was flying it was "Notice to Airmen." Wondered when they would get around to fixing that. Guess they finally did.

wstuartcl4• 4 years ago

Both are still in use. For instance during your pilots tests either is considered an equivalent answer.

NelsonMinar2• 4 years ago

that's the test that makes you a Temporary Airman.

pookha• 4 years ago

They JUST changed it. Made a point of signifying to the masses about it last month: https://generalaviationnews.com/2021/12/07/notams-are-now-no...

What this tells me is a group of government busy-workers are playing around with complex systems that can dramatically impact millions of people.

anigbrowl• 4 years ago

Wondering if this is just standard aging infrastructure embarrassment or could involve a cyberattack? Cause if I were a state actor with a beef this is exactly the sort of soft-but-expensive injury I'd want to inflict: extremely disruptive, but also hard to assess or explain to the public.

sidcool• 4 years ago

What is the root cause?

allen-munsch• 4 years ago

I was wondering if this might be related to elastic's "several customer facing deployments deleted across multiple regions" issue from earlier today? The timing seemed to line up a bit.

https://status.elastic.co/

throwaway290• 4 years ago

Could it have been a nuclear option used by a federal agency to hold someone's possible escape? Perhaps they were not sure which plane, so they had to hold all. Or it was just the most immediate route

FAA NOTAM System Outage

Related Stories