The Problem That Built an Industry

https://ajitem.com/blog/iron-core-part-1-the-problem-that-built-an-industry/

by ShaggyHotDog • 3 months ago

152 48 3 months ago

paulnpace• 3 months ago

It...handles 50,000 transactions per second with sub-100ms latency on hardware that costs a fraction of an equivalent cloud footprint. It has been doing this for 60 years.

Eat that, Bitcoin.

bombcar2• 3 months ago

50,000 transactions a second is a bunch for humans.

It’s nothing for even an ancient CPU - let alone our modern marvels that make a Cray 1 cry.

The key is an extremely well-thought and tested design.

ForOldHack2• 3 months ago

It was mostly written in COBOL. This is how I got into XENIX/UNIX. A machine went down that was a training machine: It required XENIX on a PC, because it ran on RM/COBOL on XENIX, because all the screens, and the encoding, already written were in COBOL. RM/COBOL had an ancient compatibility, but the code was extremely simple, having been ironed out many decades earlier. ( I got it in 1985, but all the creation dates were 1982. The original files must have been from the mid 1960s. I pointed this out, and someone called me on it. I found a training manual from 1966, and the screens were exactly The same, except of course for the 3270 status line.

It was fast on an 4.77Mhz IBM PC, and much faster on a 10Mhz V20.

50,000 transactions was pretty standard for a IBM Mainframe, now? The z/ Series is still about the same, but it scales up to 32 processors. ( excuse me, billions. per day )

buckle8017• 3 months ago

Ah yes a completely centralized system that scales, who would have thought.

(For the pedantic, it's not exactly centralized nor federated since each airline treats their view of the world as absolutely correct)

arter452• 3 months ago

It’s not decentralized either, at least not in the Bitcoin sense of the word. Interactions between participants may be automated but they can ultimately rely on legal contracts and people. IATA is one of those participants, but everyone has to trust IATA in the airline industry because of their role. A decentralized airline system built to avoid trust in a central authority would be pretty different (actually the booking part may be the least of their problems there).

It probably doesn’t require consensus among all participants (pairwise consensus at every step should be fine), so there is very likely no voting.

It’s not even permissionless. It’s not like a random company could join this “chain” simply because they can generate a keypair.

It’s a fundamentally different problem, and it makes sense that the architecture is different.

andersmurphy2• 3 months ago

I mean you can easily do 100K TPS on a M1 with sqlite and a dynamic language. With sub 100ms latency.

People don't do it because it's not fashionable (the cool kids are all on AWS with hundreds of containers, hosting thousands micro services, because that's web scale).

arter452• 3 months ago

Well, transactions in this context are business transactions, which may involve 1 or N remote calls. Imagine checks against no fly lists, fraud detection, flight delay and so on. Speed of light is also another concern. So it’s not as simple as doing 35k TPS on a local SQL database.

But yes, you don’t always need cool technologies.

contraposit2• 3 months ago

There is a parody youtube video in which one delusional pro-blockchain speaker is talking about how he wants his food to be put on blockchain which was the audio part, the video part was showing how a no-name China restaurant is using oil from sewers to cook food.

https://en.wikipedia.org/wiki/Gutter_oil

outside1234• 3 months ago

It is interesting to think how AI will potentially change the dynamics back to this from general purpose software.

In a world where implementation is free, will we see a return to built for purpose systems like this where we define the inputs and outputs desired and AI builds it from the ground up, completely for purpose?

DanielVZ• 3 months ago

I was thinking the same sans AI. What other industries require low latency high throughput transactions that haven’t been served yet?

conductr2• 3 months ago

Probably. It’s already happening with SaaS as an example. I’ve mentioned this on HN a lot in past but my (established) company has been rolling its own CRM and some other tools with AI.

It seems we can build a product ourselves in the same time it would take us to talk to saas vendors and draft the RFP/requirements. We can build it and iterate as the requirements are being forged, so can essentially have completed software with just the features we care about, with full ability to add features in future (something saas doesn’t promise) often before an implementation would even kick off. We’re searching through all our SaaS products and i expect we’ll cut 50% of them in 1-2 years. The ones that are sufficiently complex or regulated have some protection (like accounting systems).

contraposit2• 3 months ago

A glove (software) that fits the hand (company processes) perfectly.

arrsingh2• 3 months ago

Interesting to note right at the start of the article that they sat on a plane next to each other in 1953 but the formal partnership between AA and IBM was not till 1959 - 6 years later! The article makes it look like all this happened magically fast but in reality a reminder that things take time!

> is almost mythological. In 1953, C.R. Smith, president of American Airlines, was seated next to R. Blair Smith, an IBM salesman, on a cross-country flight. By the time they landed, the outline of a solution had been sketched. IBM and American Airlines entered a formal development partnership in 1959.

edit: oh and then the actual system didn't actually go live another 5 years later - in 1964. Over a decade after the two of them sat next to each other.

Reminder to myself when my potential customers don't sign the deal 5 minutes after my pitch!

throwup238• 3 months ago

Reminder to myself when my potential customers don't sign the deal 5 minutes after my pitch!

The classic "the decision makers can take longer to buy than you can stay solvent" problem of enterprise sales.

uxhacker2• 3 months ago

So during the intermittent period IBM and American Airlines were focused on research of the problem.

The system was a based on a military messaging system.

What is important to note is before SABRE the system used was a sell at will until a stop message was issued. Then sales would be on request. This method is still used between different airline systems today.

Before the implementation of SABRE airlines used teleprinters as a way of communicating. Some of the commands SABRE and other IBM 360 systems come directly from this period. For example AJFkSFO9MAR was a way of economizing on characters sent. It means what is the available seats from JFK to San Francisco on the 9th of March. This predates SABRE.

There is several reasons that the System 360 (the reservation systems used by airlines like SABRE) is one that it is written in Assembler, and also the logic is very tied into its role of reservation. For example it was designed in the days of punchcards, which have a totally different method of matching than a relational Database. The logic is still used on matching a seat to a fare.

On the pure speed much of it is gained by clever engineering tricks. An example would be the passenger record. This is 9 alphanumeric id of the passenger reservation. It is the hash of virtual memory location of the reservation. It takes 4 cpu cycles to retrieve it.

rawgabbit• 3 months ago

The technology for SABRE came from SAGE. Which was built for NORAD.

https://www.ibm.com/history/sage

qazxcvbnmlp• 3 months ago

Ive been trying to find a SABRE reference manual somewhere. Do you know if theres any that exist online?

WillAdams• 3 months ago

Well, there is the old joke...

Three women are sitting in a bar discussing their lovers, the first says, "My lover is a wrestler, he's so energetic, it's _wonderful_!".

The second responds, "My lover is a poet, he's so romantic and thoughtful and sensitive, it's like something out of a fairy tale!".

The third is silent, and the other two women look at her expectantly until she finally sighs and says, "My lover is an IBM salesman, he just sits on the edge of the bed and tells me how good it will be when we eventually make love."

StilesCrisis• 3 months ago

"The key insight is [...]. No daemons. No background threads. No connection state persisted in memory between transactions."

Closed the tab.

arrsingh2• 3 months ago

I noticed that too and did roll my eyes as well but I'm glad I kept reading - its actually quite a good article. Maybe the author used an LLM to help do some copy editing but should have probably given it less editorial agency.

Either way I'm glad I read it and waiting for the other parts of the series. Really curious how to get access to this airline booking data so I can write my own bot to book my flights and deal with all the permutations and combinations to find the best deal.

alwa• 3 months ago

Same here—I tolerated the linguistic tics, I found enough meat to keep going, but it’s the conclusions that confuse me. And that feels like the intellectually dangerous part with this sort of LLM writing. For example:

Convergent evolution is real. Every major GDS independently arrived at the same underlying platform. That is not coincidence — it is the market discovering the optimal solution to a specific problem.

I struggle to understand the claim that GDSes “arrived independently” at interoperability standards through “convergent evolution” and market discovery. Isn’t it something closer to a Schelling point, or a network effect, or using the word “platform” to mean “protocol” or “standard”?

Isn’t it like saying “HTML arose from web browsers’ independent, convergent evolution”? Like—I guess, in that if you diverge from the common standard then you lose the cooperative benefits—see IE6. And I guess, in that in the beginning there was Mosaic, and Mosaic spoke HTML, that all who came after might speak HTML too. But that’s not convergent evolution, that’s helping yourself to the cooperative benefits of a standard.

“The market” was highly regulated when the first GDSes were born in the US. Fares, carriers, and schedules were fixed between given points; interlining was a business advantage; the relationships between airlines and with travel agents were well-defined; and so on [0]. IATA extended standards across the world; you didn’t have to do it the IATA way, but you’d be leaving business on the table.

If anything, it seems like direct-booking PSSes (he mentioned Navitaire [1]) demonstrate the opposite of the LLM’s claim. As the market opened up and the business space changed, new and divergent models found purchase, and new software paradigms emerged to describe them. It took a decade or two (and upheaval in the global industry) before the direct-booking LCC world saw value in integrating with legacy GDSes, right?

…the LLM also seems bizarrely impressed that identifiers identify things:

One PNR, two airlines, the same underlying platform.
Two tickets, two currencies of denomination, one underlying NUC arithmetic tying them together.
One 9-character string, sitting in a PNR field, threading across four organisations' financial systems.

[0] https://airandspace.si.edu/stories/editorial/airline-deregul...

[1] https://www.phocuswire.com/Jilted-by-JetBlue-for-Sabre-Navit...

cr125rider2• 3 months ago

Can you explain why that’s wrong?

defen• 3 months ago

It's the LLM-generated-text signature.

croisillon• 3 months ago

ironically...

  "That is not coincidence — it is the market discovering the optimal solution to a specific problem. When you see that pattern in your own domain, pay attention to it."

vachina• 3 months ago

The entire thing (and probably the entire series) is LLM generated. Bro didnt even bother to redo the generated flowchart.

Site is likely SEO slop for future product placement.

cr125rider2• 3 months ago

Can you add RSS to your site? I’d love to follow but can’t.

neilv• 3 months ago

ITA Software integrated with the mainframe network, and was acquired by Google.

An exec made a public quote that they couldn't have done it if they hadn't used Lisp.

(Today, the programming language landscape is somewhat more powerful. Rust got some metaprogramming features informed by Lisps, for example, and the team might've been able to slog through that.)

zellyn• 3 months ago

I was so sad when Google shelved their Sabre replacement. Could have run the whole industry.

zer00eyz2• 3 months ago

SABRE, is a reminder that things that are well designed just work.

How many banks and ERP's, how many accounting systems are still running COBOL scripts? (A lot).

Think about modern web infrastructure and how we deploy...

cpu -> hypervisor -> vm -> container -> run time -> library code -> your code

Do we really need to stack all these turtles (abstractions) just to get instructions to a CPU?

Every one of those layers has offshoots to other abstractions, tools and functionality that only adds to the complexity and convolution. Languages like Rust and Go compiling down to an executable are a step, revisiting how we deploy (the container layer) is probably on the table next... The use case for "serverless" is there (and edge compute), but the costs are still backwards because the software hasn't caught up yet.

01HNNWZ0MV43FF• 3 months ago

Library code - This is necessary because some things are best done correctly, just once, and then reused. I am not going to write my own date/time handling code. Or crypto. Or image codecs.

Run time - This makes development faster. Python, Lua, and Node.js projects can typically test out small changes locally faster than Rust and C++ can recompile. (I say this as a pro Rust user - The link step is so damned slow.)

Container - This gives you a virtual instance of "apt-get". System package managers can't change, so we abstract over them and reuse working code to fit a new need. I am this very second building something in Docker that would trash my host system if I tried to install the dependencies. It's software that worked great on Ubuntu 22.04, but now I'm on Debian from 2026. Here I am reusing code that works, right?

VM - Containers aren't a security sandbox. VMs allow multiple tenants to share hardware with relative safety. I didn't panic when the Spectre hacks came out - The cloud hosts handled it at their level. Without VMs, everyone would have to run their own dedicated hardware? Would I be buying a dedicated CPU core for my proof-of-concept app? VMs are the software equivalent of the electrical grid - Instead of everyone over-provisioning with the biggest generator they might ever need, everyone shares every power station. When a transmission line drops, the lights flicker and stay on. It's awe-inspiring once you realize how much work goes into, and how much convenience comes out of, that half-second blip when you _almost_ lose power but don't.

Hypervisor - A hypervisor just manages the VMs, right?

Come on. Don't walk gaily up to fences. Most of it's here for a reason.

zer00eyz2• 3 months ago

Most of it's here for a reason.

Your argument for host os, virtual os, container is the very point im making. Rather than solve for security and installablity, we built more tooling, more layers of abstraction. Each have overhead, security surface and complexity.

Rather than solve Rusts performance (at build time), switch to a language that is faster but has more overhead, more security surface, more complexity.

You have broken down the stack of turtles that we have built to avoid solving the problem, at the base level...

SABRE, what the article is discussing, is the polar opposite of this, it gives us a hint that more layers of abstraction arent always the path to solutions.

Muromec• 3 months ago

I used to work on a project that interfaced with both SABRE and Amadeus and "just works" isn't how I would describe it. The thing is also quite slow and annoying, as it's interface is optimized for the trained operator to use it in a terminal setting and not for us poor shcmucks calling it through some weird API bolted on top.

Also, try to retrieve a PNR on an airline website or do like anything on the airline's own website -- the UX is usually pretty bad and the data loading takes forever. For that too the GDS is to blame.

andai• 3 months ago

TPF is not modern. It would fail every architectural review a contemporary engineering team would apply to it. It also handles 50,000 transactions per second with sub-100ms latency on hardware that costs a fraction of an equivalent cloud footprint. It has been doing this for 60 years.

What kind of review would it fail? Sounds like it's pretty well designed to me.

jakub_g• 3 months ago

FWIW, IBM has been increasing TPF costs dramatically since mid 2000s, which prompted the GDSes to go through decade-plus efforts to migrate away from it, to the cloud.

conductr2• 3 months ago

Interesting they went from 90 minute manual booking time to microseconds. I’m unsure of what the landscape was really like before Unix and such, maybe this was just how software was and everything was a bespoke ordeal. But, it makes me wonder if something less fast could have been “better”. As in, faster to build or easier to maintain/improve, something where we still weren’t talking about wrestling with legacy software that runs the world, those kinds of things.

Even 1 second transaction speed sounds slow today but if it’s replacing a 90 minute manual process I’d rather have that solution now than a microsecond fast solution that takes 5-10 years.

Animats• 3 months ago

This wasn't the first electronic reservation system. It was preceded by the Magnetronic Reservisor, custom built for American Airlines by Teleregister. It didn't use general-purpose computers. It had magnetic drums and remote terminals, but it was not general purpose.

Teleregister built a range of such special-purpose systems, for several airlines, railroads, and air traffic control. General purpose CPUs were both too expensive and too slow for these jobs in the early 1950s. Using a general purpose computer for everything didn't really happen until the minicomputer era in the mid-1970s. SABRE had to wait until general-purpose computers got better.

It's interesting that transaction processing operating systems died out. Tandem's OS worked that way. But the run transaction program once and flush it approach is almost dead. Except, amusingly, for CGI programs, which are true use-once transaction programs, with an inefficient implementation.

[1] https://www.youtube.com/watch?v=F4d-OFDs1hY

[2] https://s3data.computerhistory.org/brochures/teleregister.sp...

kuzivaai• 3 months ago

Fitness for purpose beats fashionable architecture" — this matches something I've seen in automotive. Systems that would fail every modern review but have been running correctly for longer than most of the team has been alive. Looking forward to Part 2 on the PNR.

The Problem That Built an Industry

Related Stories