The Newark airport crisis

https://www.theverge.com/planes/673462/newark-airport-delay-air-traffic-control-tracon-radar

by 01-_- • 1 year ago

137 126 1 year ago

alwa• 1 year ago

bzmrgonz2• 1 year ago

What's stopping us from implementing holo style 3d displays like in the movies? Star-Trek etc. Are we not there yet?? alternatively, what about VR? we could virtually project the traffic controllers out into space like silver-surfer or ironman right(pov of course)? I'm not an expert, but it seems we need a better UX/UI right?

alephnerd5• 1 year ago

but it seems we need a better UX/UI right?

Wrong.

It's the fact that the FAA has $5.2B in outstanding repairs but only $1.7B allocated for repair.

On top of that, the GS pay scale penalizes federal employees in high CoL areas.

Both are very difficult problems to solve, as the former means dramatically increasing the FAA's budget (which is tiny for the scope of responsibility it has across North America), and the latter means completely reforming the General Schedule.

On top of that, Congress constantly meddles with the FAA and DoT in general because it's the easiest way to get some quick wins for constituents.

The FAA has been working on modernizing air traffic control, but that project won't be completed til 2030 at the earliest.

Furthermore, the Northeast is a uniquely congested airspace with the massive number of airports and passengers.

mschuster915• 1 year ago

The budgetary questions are unrelated to UX/UI stagnation though.

It's annoying how many modern web sites change their entire design framework once every two years, yes. But ATC? Aeronautics in general? Most of maritime? Once it's certified, it's practically ossified - and for good reason. Bad UI/UX can literally kill [1].

Nevertheless, I think it's worth having the debate - and that led by actual air traffic controllers, please - if and if yes, how, UX/UI can be improved.

[1] https://uxmovement.com/buttons/how-an-interface-mode-killed-...

dehrmann3• 1 year ago

I'm not an expert, but it seems we need a better UX/UI right?

This isn't where the problem is. It's system reliability, increased air traffic, and increased controller workload.

plorkyeran• 1 year ago

Are you aware that many things depicted in movies do not actually exist?

heraldgeezer4• 1 year ago

but it seems we need a better UX/UI right?

No, you sound like a middle-manager.

Talk to cashiers. They all want terminal DOS based systems where keyboard is king. It's the fastest once you learn it.

ctoth• 1 year ago

To save money, the FAA elected not to build a new STARS server in Philadelphia to support the move. A new server alone would require tens of millions of dollars, as well as installation of new internet and power infrastructure.
Instead, it elected to send a “mirror feed” of telemetry from the STARS servers at N90, traveling over 130 miles of commercial copper telecom lines, with fiber optics to follow by 2030.
The annoyances of traditional cable internet — frequent lag, dropped sessions — are probably familiar to those who stream video or play games online. But for air traffic controllers, even the smallest service disruptions can become dangerous.

So LOL what, they just ... piped it over the Internet? Also can someone make sense of this "new server" costing millions of dollars? Presumably it's not the cost of a server, which is orders and orders of magnitude less than that?

alephnerd5• 1 year ago

Aerospace is hard.

An ATCS like STARS needs to feed from multiple different OT and IT sources like radars, weather stations, other TRACONs, etc and is implemented in it's own airgapped environment.

It can get very pricy very quick. On top of that, the FAA's budget has been sclerotic for decades now after the 1980s era union action and the 1990s era national cost cutting.

And finally, it is a political organization, and NATCA is a fairly prominent union within the AFL-CIO, and could make the lives of NJ representatives hell for pushing reassignment out of Newark.

amluto3• 1 year ago

Aerospace is hard.

I would believe it was hard. And maybe it still is if you’re unwilling or unable to take advantage of modern technology.

Current low-cost equipment can easily send 10 or 100Gbps over long distance fiber links. Depending on how quickly you want to fail over when a link or an entire switch, router or rack fails, there are plenty of options that make various tradeoffs between failover latency and bandwidth, all the way up to completely duplicating all the traffic on redundant routes. I would bet that the entire aggregate traffic needed for air traffic control in a region is well under 10Gbps. And 10Gbps dedicated links or leases or (effective) purchases of dark fiber are not expensive on the scale of the FAA. Air traffic should use a network with a lot of redundancy, so maybe multiple those low costs by something like 5.

SoftTalker2• 1 year ago

Why not pipe it over the internet? A few redundant ISPs at the endpoints should be as reliable as a private run of fiber or T1 lines or whatever bespoke solution they think they need?

mschuster915• 1 year ago

As soon as it's on the general Internet, any ill-meaning enemy can just force the VPN endpoints offline by blasting them with junk traffic.

K0balt2• 1 year ago

Doing so sticks billions of transistors between A and B, all of which have to work perfectly billions of times a second. If you just run fiber, you need to put an amplifier in every 50km or so, other than that it’s all just buried glass.

The fiber is already there, and it gives you guaranteed capacity, even during a major power outage. And it guarantees no one is changing out a switch or a router at a bad time, etc.

It’s just a lot less to control and to go wrong.

edoceo2• 1 year ago

When they say "server" it means the (cheap) hardware and the (very expensive) software to drive these complex and critical systems.

mschuster915• 1 year ago

If it's mainframe, and there is no reason to believe it's not given the strict requirements, the hardware is expensive as fuck and the people who can keep these beasts alive are just as expensive.

yusyusyus• 1 year ago

Nah, probably on leased lines like t1s/t3s. Airport telecom infra isnt always the best.

wkat4242• 1 year ago

A "Server" is just layman's speak for that big room with all the noise and blinky lights. Including all the hardware, networking and software that goes with it. Of course you're not going to run critical infra on a single server :)

Though I have to admit I have seen government operations where the "server" was an old dell optiplex desktop lying on its side in a broom closet without ventilation, a post-it with "IT SERVER DON'T TURN OFF", a spiderweb of cables running through the closet and the "server" fans screaming for air trying to keep everything cool in the enclosed space. I'm not kidding.

I mean, I know, government. Small local welfare-related org. Shoestring budget. Sure, that sucks. But at least you can make sure it's tidy and the cabling doesn't look like shit. Jeez. I didn't imagine I'd still see that in this century. Do people no longer take pride in their job? They hadn't even activated the "AC Power on" in the BIOS so after electrical maintenance they had to wait for the "engineer" to press the on button again.

gopher_space• 1 year ago

There's no budget to fix whatever you messed up when redoing the cabling, which we also didn't really have the budget for. Everyone who knows what the box actually does has retired or died, so nobody wants you in the broom closet shifting dust motes near a working system we can't replace.

amluto3• 1 year ago

I doubt that 130 miles of copper is the public Internet. It’s presumably some legacy telecom system that depends on a bunch of generally fairly well made but thoroughly obsolete hardware.

Keep in mind that Ethernet over copper is only specified to ~100 meters. Long distance copper networks have been obsolete for a few decades.

izacus2• 1 year ago

Here in my EU state, air traffic control still used leased copper twisted pair for server link over ~150km as of at least 2015 or so. It's pretty reliable and not "public internet" at all - just two modems with known performance and known latency taking to each other.

Might be something like that in this case as well.

K0balt2• 1 year ago

It’s called a dry loop. It might be real with repeaters at the various COs, or it might be partially virtualized and sent over a fiber trunk. Usually somebody leading a dry loop does so for the relatively extreme reliability, even though the speed is not very high.

khazhoux• 1 year ago

pp. lag
hold all traffic
rdy?
gogogo

heraldgeezer4• 1 year ago

What are leased lines?

SoftTalker2• 1 year ago

Old tech. You’d literally lease a line from ATT or whatever Bell company covers your area. A reserved private connection. Modem at each end. They ran at about 56Kb/s IIRC.

moomin• 1 year ago

It's amazing, really. You hear about "government overspending" all the time. You actually look into something in any detail and what you discover is a consistent pattern of underspending. Call it mismanagement if you want, but it is consistently what we, as votes, and the executive, ask them to do.

alephnerd5• 1 year ago

Most people see a number $6 Trillion and something breaks inside them.

What they don't realize is maintaining infrastructure is expensive.

Sure there are a lot of inefficiencies with need to be fixed, but you can't throw the baby out with the bathwater.

Sadly, such is society, and this is a problem that happens everywhere - be they democracies or authoritarian states.

bombcar3• 1 year ago

With 3 million employees and about one cent a flush, the US government spends 100,000 or so a day just to flush. Assuming three bathroom breaks a working day.

juujian• 1 year ago

When people think wastage, they think bureaucrats wasting away in offices. Social security admin, air traffic control, and teachers are more like Amazon warehouses at this time in terms of utilization. Less than 1% of social security admin funding is overhead at this time. Any charity would be exhilarated to hit numbers like that.

pj_mukh• 1 year ago

"Most people see a number $6 Trillion and something breaks inside them. What they don't realize is maintaining infrastructure is expensive."

It breaks all our brains too, because that $6 Trillion has very little to do with maintaining infrastructure. The bulk of it is just direct payouts (Social Security, Medicare and defense contracts).

Pretty frustrating that this big number means the government is politically forced to do drastic austerity for things like keeping planes flying safely.

Also, makes DOGE starting at USAID (<1% of budget) look especially incompetent.

RyJones• 1 year ago

Meanwhile, back at the ranch, I was a contractor for HHS for a year or so. When I started, there was a server farm built for a contract that was powered up and maintained; never used, it was broken up and sold for surplus. Just before I left, HHS thought they would like that program again, so us US taxpayers bought another tranche of computers to power up. I don't know the end of the story.

I could type more, but it would be a long and boring story.

newsclues• 1 year ago

There is overspending. It’s a scam.

There is mismanagement.

There is also a misallocation and underfunding of essential services and infrastructure. This is the excuse for ever more funding.

Modified3019• 1 year ago

Came across this post, which perfectly encapsulates a sort of pathological inversion of sunk-cost fallacy, where some people I encounter would happily let infrastructure rot and countless people suffer if it saves them pennies on their taxes, and are incapable of re-evaluating their position because “taxes bad” has somehow become such an ingrained/religious value that anything else is unconsciously and immediately rejected no matter the consequences.

https://old.reddit.com/r/goodnews/comments/1kuaasx/i_voted_f...

>I posted this somewhere else ages ago, feel it's relevant
I remember having a conversation with my ex's sister and their mum a few years ago, around election time. I try to not talk politics with people because it's a fast way to lose friends, but the topic came up between my ex and them over dinner and I just listened in.
I remember them saying that the only thing they're interested in is tax cuts. More money for them. I had to chime in and ask what about the NHS, what about funding schools? They said they didn't care because they had private healthcare through their jobs (finance), so they don't need the NHS. The mum said her kids are through school so she doesn't care about funding schools, and the sister said she'll be sending her kids to private school one day. I was pretty gobsmacked at the brazen selfishness of it, and asked what if they lose their jobs - and therefore their private health care - or become unable to work, what if when you have kids you can't afford private school? Neither of them could grasp this hypothetical... it was as if I was speaking another language to them. They were just like 'but we do have jobs.' And what if you didn't? 'But we do.' It was just circular and they couldn't see themselves in any situation other than the one they were currently in.
I can quite see why empathy is a hard concept for right wingers to grasp, it was like they just simply couldn't understand the concept. They weren't stupid either, and nor were they rich - the mum worked in admin for a finance company in the city, and the sister was being paid by the same company (mum got her in the door) to train as an accountant.
I think about those two every now and then when I can't understand how the other side thinks. Because it seems we do literally think very differently.

J37T3R• 1 year ago

It's like tech debt. It's an ongoing cost in a one and done environment, it's hard to see problems from the outside until there's catastrophic failure, and if there's a slow niggling annoyance of things getting worse over time the point where people notice enough to care is usually past the point of needing a refactor. So we get underspending where it matters, overspending where it doesn't, and the solution is always a redo.

nxm• 1 year ago

Don't generalize. There was 0 underspending at the Department of Education or USAID.

cperciva4• 1 year ago

The general pattern I see is that operating spending is too high while capital spending is too low.

jagger272• 1 year ago

Newark’s overworked controllers might argue differently. Of course in this forum the general suggestion will be to replace tired controllers with sleepless machines, and the technologists here have strong incentives to advocate for such solutions.

anon7000• 1 year ago

So true. The state of Washington has a great example: the outgoing Secretary of Transportation was very clear that if the state doesn’t change the budget to be primarily about maintenance, and much less about new highways or lane expansion, then our infrastructure will quickly begin crumbling.

Even the people in charge of our highways want us to switch to operations & maintenance oriented projects, but the representatives have not done so.

The incentives in government are really fucked — you get visibility and wins through cool projects, not by keeping the lights on and things running smoothly. Honestly true in big companies as well.

jjallen2• 1 year ago

The US overspends on military and underspends on most other things. Not sure when this will ever change. Only when it has to I guess.

sour-taste• 1 year ago

In a pretty remarkable statement a month or so ago Trump actually said we were overspending on the military and suggested cuts. Then his budget comes out and proposes military expansion and Medicare cuts so like usual he was just bullshitting I guess.

readthenotes1• 1 year ago

Have been many billions of dollars spent over the years to "modernize" air traffic control. If my recollection is correct, most of it ended up being wasted.

The current administration is asking for a lot of money to try to fix it again

ars• 1 year ago

you discover is a consistent pattern of underspending

No, what you discover is a pattern of wasted spending. Then they ask for more money to actually get the job done despite all the waste.

People try to solve this by privatizing certain things, figuring that competition will help efficiency. Sometimes it works, sometimes it doesn't.

Maybe we need competing governments, and whichever government is more efficient gets to rule. Seriously: Add a second FAA at some test airports, see if they can do better, with the understanding that if they can't, they get shut down.

cycomanic3• 1 year ago

Maybe we need competing governments, and whichever government is more efficient gets to rule. Seriously: Add a second FAA at some test airports, see if they can do better, with the understanding that if they can't, they get shut down.

And you would be willing to be personally responsible if people die in this experiment?

It's funny how people here always complain that any money government spends is wasted, but if you look at big companies they are "wasting" money as well. Just look at the number of projects that google killed. It's simply a function of large (and small) organizations that they don't get it right all the time, it's difficult to predict the future.

bombcar3• 1 year ago

This is a classic system at 99.9% capacity - there’s no slack to take up issues or do anything but run at, well, 99.9%.

And without something like a major disaster, it’ll likely continue to get worse and worse.

KennyBlanken• 1 year ago

Air travel is far too subsidized by the public and I can think of few worse applications of public funds save maybe sports stadiums. It's a huge waste of energy, pollution, human labor...

Rail is incredibly efficient, and there's a reason China has been building high speed rail as fast as it can.

To all the "it would never work here" people: we used to be a nation of rail travel, where you could walk or bike or take a taxi to the local trolley/train/bus station, take a train to where you needed to go.

All that was systematically ripped apart by the auto industry either directly or indirectly. There is no reason whatsoever we can't work our way back, especially given how much faster and easier construction of a railway line is now.

alephnerd5• 1 year ago

Rail is incredibly efficient

If you have the density to justify it.

There is a case to be made for enhancing rail transit in the eastern seaboard and maybe parts of the Midwest, but America is too large and sparse to justify rail transit at scale.

It makes more sense to concentrate on rail infra for freight transit and work on revamping our existing rail freight infra.

there's a reason China has been building high speed rail as fast as it can

China stopped subsidizing HSR during the COVID recession. It costs the exact same as a flight ticket now [0] due to high debt [1] (excluding the Beijing-Shanghai track, which actually can justify usage).

Most Chinese use normal rail for intercity transit, but this is easier to justify given the density and ease of land acquisition.

But even then, China began slowing down railway investment and construction since 2018 [2][3], and started calibrating towards air transit [4] as part of a commercial aviation push [5]

[0] https://www.nytimes.com/2024/05/13/business/china-bullet-tra...

[1] https://www.caixinglobal.com/2019-01-29/zhao-jian-whats-not-...

[2] https://www.caixinglobal.com/2018-01-02/china-railway-corp-s...

[3] https://www.caixinglobal.com/2021-03-30/china-looks-to-slow-...

[4] https://www.caixinglobal.com/2021-03-25/smaller-cities-reach...

[5] https://www.reuters.com/business/aerospace-defense/comac-jet...

tuna74• 1 year ago

Going from Qingdao to Beijing is much cheaper on train vs airplane. Takes roughly the same time as well.

Also, China does both new airports and new rail lines.

dehrmann3• 1 year ago

Air travel is far too subsidized by the public

Don't most of FAA's funds come from taxes on air travel? And around half of Americans travel by air every year, so it's not a niche service.

everybodyknows• 1 year ago

I can think of few worse applications of public funds save maybe sports stadiums.

Californians can.

Subsidies to the movie industry, plumbing run to waterless urinals, bullet trains between farm towns, ...

sofixa2• 1 year ago

bullet trains between farm towns,

This is disingenuous and you know it.

I think it was the wrong choice for a number of reasons, but the farm towns in question are just first lot of the whole network, starting with the (supposedly) easiest part. Instead of building in the densest parts which would be even more complicated and expensive.

zamadatix• 1 year ago

I'd LOVE some proper high speed rail across the US but EWR is also huge for international flights. A good rail network would at least help getting to/from their for it but that still leaves a good amount of flights that make no sense to dump.

anal_reactor• 1 year ago

Rail is incredibly efficient

As a European, I don't understand the fetishization of railway. These are two different means of transportation with two different use cases, with tiny overlap between them.

wnevets• 1 year ago

It turns out trying to run an airport like twitter is a bad idea, who knew

dehrmann3• 1 year ago

its implementation of a “NextGen” air traffic control system to replace the current version may not be completed until 2034, even though the project was started in 2003.

Governments (and a lot of businesses) like to look at software as a one-time purchase, but it's really better too look at it as a liability and an ongoing cost. It'd be better to have a team make continuous, incremental improvements to the system than have "NextGen" last-gen replacement vaporware.

throwaway48476• 1 year ago

There are examples of both. Firmware and embedded software usually is a one time purchase. Tax software requires ongoing updates. FAA software would fall somewhere in between.

pphysch• 1 year ago

It really is a binary though. Either you need to think about and deal with deploying (system-wide) updates, or you don't.

Even infrequent ~5y update lifecycles tend to be extremely painful unless there is substantial investment in treating it as an essential business process. This leads to a "kick the can" mentality that translates to show-stopping amounts of tech debt.

yencabulator• 1 year ago

Lots of firmware is upgradable these days. Some of of it even should be updated regularly, or you end up with weird novelty problems like your bluetooth earbuds being vulnerable to drive-by attacks.

yoaviram• 1 year ago

Software is a liability, a product is an asset.

ferguess_k• 1 year ago

They like to give contracts to connected outsiders so that they can take care of their friends and take the blame away. 30 years? quoting the venerable Sir Humphrey Appleby:

"Precisely. Months of fruitful work. Leading to a mature and responsible conclusion."

Since it takes 30 years it must be a responsible piece of software, polished to the bone.

siliconc0w• 1 year ago

+1, the "strangler" pattern is the best (and often only) way to replace/update systems.

0) no new logic in the old system (stop the bleeding)

1) incrementally wrap old system interfaces with new system

2) proxy to old system and dark launch new system

3) monitor / confirm correctness

4) cut-over

ethbr12• 1 year ago

I can say that at least one extremely large US company successfully retired their transaction processing mainframe system using exactly this pattern.

It wasn't simple, but running old+new in parallel on the same production data stream (then verifying behavior against each other) is a surprisingly powerful approach.

Most critically, because it trades temporary maintenance and infra costs in exchange for decreasing risk and flexibility around cutover date.

Kind of like black box TDD, which helps the new system team be much more aggressive with their delivery timelines.

ck2• 1 year ago

We had a 9/11's worth of death EVERY DAY for the first TWO YEARS of covid

Then a 9/11's worth of death EVERY WEEK for the next TWO YEARS of covid

We still have a 9/11's worth of death EVERY MONTH in 2025

Personally I don't want anyone to die in a plane crash

But apparently a hundred million other people do not care anymore about others dying needlessly

So factor that into your next flight if you are taking your life into your own hands?

coderatlarge5• 1 year ago

i flew delta out of ewr last week. the pilot announced over speaker that air traffic control was serializing take offs on one runway to avoid problems. we sat on the tarmac for two hours waiting our turn. that was on top of another hour delay for our flight arriving due to this policy.

delta doesn’t reimburse for missed connections claiming air traffic control policies are outside of their control.

it reminds me of a dev team i worked with once which used single threaded memcache as a way to serialize inbound requests to a server with improper locking logic inside.

pixl972• 1 year ago

Sounds like it's time to drop Delta

elijaht• 1 year ago

What do you mean delta doesn’t reimburse connections? I believe they would still be responsible for getting you to your final destination

coderatlarge5• 1 year ago

they still booked me on next day flight but refused any compensation for hotel or meals or ground transport for the overnight stay due to the missed connection.

hbsbsbsndk2• 1 year ago

delta doesn’t reimburse for missed connections claiming air traffic control policies are outside of their control.

AFAIK if it's one booking on the same airline or a codeshare they are required to rebook you. If you planned a "connection" which is two single flights with different airlines you don't get any legal protections. This isn't just Delta, no airline will reimburse you for missing a flight you didn't book through them

coderatlarge5• 1 year ago

they did book me on a flight the next day. i was scheduled to fly two legs on delta. but see my other replies below on lack of accommodation and rep behavior.

IAmGraydon• 1 year ago

delta doesn’t reimburse for missed connections claiming air traffic control policies are outside of their control.

Reimburse? They'll put you on another connecting flight of your choice as any airline does. I did this on a Delta flight with a missed connection due to delay just last week. They also automatically upgraded me to Comfort+ at no extra charge for the inconvenience.

coderatlarge5• 1 year ago

no upgrades for me; not even an apology. imo the ground team leader was practically gleeful at not having to provide any compensation for the delay.

autobodie• 1 year ago

Need more trains.

stn8188• 1 year ago

I'm in no way qualified to comment on the details of the issue outlined here (though I did get stuck babysitting a friend's kids for many extra hours due to a 6hr delay on said friend's flight into EWR a few weeks ago). For anyone who lives far enough north and west of EWR though, I highly recommend trying either the Allentown or Scranton/Wilkes-Barre airports. I've moved my business travel to Scranton 100% and have been loving it. There are more restrictions on flight times, and basically everything needs a connection, but it more than makes up for any time I would have lost due to sinkhole traffic on I-80 and the EWR parking shuttles. It's amazing to park and be at the security line basically 3 minutes later.

mathgeek• 1 year ago

Flew out of ABE most of my life, still an excellent airport, but it’s best to be aware that any connections through EWR or PHL will be by bus.

cycomanic3• 1 year ago

I don't understand this sentence:

Instead, it elected to send a “mirror feed” of telemetry from the STARS servers at N90, traveling over 130 miles of commercial copper telecom lines, with fiber optics to follow by 2030.

This does not make any sense. If they would really transmit data over a 130 miles copper line (which I doubt even still exist, especially not commercial ones), we would be talking rates in the low Mbit/s. I suspect the situation is that the "last mile" of the center is served by copper connections, not good either but by far not as bad as a 130 miles copper connection.

EDIT: I should add if they really would have a link running on copper lines it would have repeaters, which would be sitting in datacenters. In New Jersey there would by 1000s of km of dark fiber floating around, so it would be trivial to convert at least the majority of the link to fiber.

quesera2• 1 year ago

Telephone providers lease "dry loop" lines for point-to-point signalling.

I've used them for telemetry systems with acoustic modems on both ends.

Also for sending audio between broadcast studios. I recall that it was priced by bandwidth (in the literal, analog, sense), e.g.: 5kHz (~AM radio) was less expensive than 15kHz (~FM radio). For comparison: A normal phone call is 3kHz.

So yes, copper and repeaters. But very inexpensive and quick to provision. :)

cycomanic3• 1 year ago

Not sure I understand your point, using a 15 kHz carrier gives you nowhere near enough bandwidth to transmit even just 100 Mb/s.

gosub100• 1 year ago

An interim solution would be to force carriers to upsize their jets to reduce total flights per day. Force any carrier who has wide-body aircraft to use them on domestic routes. heavily limit or ban the use of smaller commuter jets (replace them with 737s with fewer total flights). I'm sure they won't be happy about it, but it's a solution that could be implemented overnight and the larger carriers would have no problem adjusting. smaller ones could sell their slots or lease larger aircraft.

vermooten• 1 year ago

It would have been easier and cheaper to raise the salaries of the Long Island employees.

The Newark airport crisis

Related Stories