Understanding Kafka with Factorio (2019)

https://ruurtjan.medium.com/understanding-kafka-with-factorio-74e8fc9bf181

by pul • 3 years ago

196 104 3 years ago

ruurtjan.medium.com

morelisp5• 3 years ago

Cute, but over years of explaining it I think any explanation of Kafka that presents it as a queue is bound to leave the reader with more misaligned expectations than when they started (while also making them think they learned something, which can be even more dangerous). To keep the Factorio-esque framing, move the consumers, not the messages.

kentm• 3 years ago

Agreed. There's an important difference between things like Kafka & Kinesis vs RabbitMQ & SQS. The latter are conceptually queues, and the former are conceptually logs. Logs and queues can both be used in many of the same use cases, but it's important to understand how they are different.

Terr_2• 3 years ago

i.e.: Items are frequently removed from queues which have a shared "next item", while logs usually just get longer and each consumer is responsible for keeping track of their own progress or positions.

It's harder to think of factory-game analogies for logs, since they involve copying without altering the original sequence. It would have to involve some kind of moving non-destructive sensor or object-cloner mechanic.

marginalia_nu6• 3 years ago

Vertical scaling — a bigger, exponentially more expensive server

This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once.

There are instances where you can get nearly 10x the machine for 2x the cost.

geodel2• 3 years ago

The idea is don't let the logic come in the way of promoting "web scale" software.

morelisp5• 3 years ago

The kind of server you'd run Kafka on tends to already be pretty far up the curve. I don't think I can get 10x our default broker for 20x the cost. Maybe 100x the cost. (I could probably get 2x it for 2x the cost but once you value HA the practical inflection point starts below the actual cost intersection.)

teawrecks4• 3 years ago

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

Sohcahtoa82• 3 years ago

Think cores instead of clock speeds.

In the case of cloud instances, doubling cores is frequently less than 100% more expensive.

marginalia_nu6• 3 years ago

You can go from a 8T/16C Epyc 7xxx series CPU to a 32T/64C CPU and not even double the cost.

dekhn4• 3 years ago

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

morelisp5• 3 years ago

Kafka is also a system that can make pretty good general use of more CPUs and more storage, but doesn't have much need for RAM. Tying the CPU and RAM together whether by CPU model or cloud vendor offerings is annoying if you're trying to scale only vertically.

vegabook• 3 years ago

If they charge these big numbers more it's precisely because they're trying to capture some of the embarrassingly better value you get from vertical scaling. It's a testament to vertical scaling's effectiveness that they _can_ do so.

defendBanana2• 3 years ago

With clouds this is not true anymore. They are exactly linear. If you ask for a smaller node they are simply propositioning a chunk of a larger machine anyway.

There is a point where the exponential pricing starts, but that point is way out there than most people expect. Probably ~100CPU, ~1TB RAM, >50Gbps network etc.

KRAKRISMOTT• 3 years ago

Also beyond a certain point, it makes sense to go straight to dedicated bare metal. The AWS tax is not worth paying if your workload is mostly fixed, somewhat fault tolerant (i.e. failed hardware on the weekends can be replaced on Monday without major interruption to business operations), and CPU bound. Get a high end machine on Hetzner and put everything behind a VPN or API auth and you will save more than 50% in spending.

RhodesianHunter• 3 years ago

I haven't found this to be true generally unless your workloads are truly completely static, which I've never actually experienced.

Given what engineers at this level cost, their costs per hour dealing with all of the nonsense clouds handle for you (networking, storage, elastic scaling, instant replacement of faulty servers, load balancing, yadda yadda) end up being higher than whatever tax you're paying for using the cloud.

Economies of scale are real.

BSEdlMMldESB2• 3 years ago

what about understanding our own imperialistic civilization?

crash in planet (or new continent?) and proceed to expand while decimating native life?

in any case, good game.

golergka3• 3 years ago

That's what all life forms do — consume everything they can and reproduce as much as they can. So, doing is _is_ life.

hutzlibu2• 3 years ago

That's an oversimplification. If you look closer, you might find that most living things are mostly living in symbiosis with their surroundings. Your human body is an example of cooperation of simple life forms. And when those living cells of your body consume everthing they can and reproduce as much as they can - than this is called cancer.

Dylan168073• 3 years ago

The native life already has massive coverage and is expanding into the remaining space.

In a typical game the player doesn't decimate them, if anything they make the native life flourish.

jrm42• 3 years ago

I was 100% expecting something about the writer. :)

geodel2• 3 years ago

Bad software is way more popular than a good writer.

jrm42• 3 years ago

Why are you downvoting him, hes right? :)

lukasLansky• 3 years ago

Both Franz Kafka and Factorio come from Prague after all.

golergka3• 3 years ago

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed into a piece of iron ore on a moving conveyor belt.

tobbe2064• 3 years ago

That one mad me laugh so much i annoyed my wife

dang2• 3 years ago

Understanding Kafka with Factorio (2019) - https://news.ycombinator.com/item?id=29304414 - Nov 2021 (72 comments)

Understanding Kafka with Factorio - https://news.ycombinator.com/item?id=20362179 - July 2019 (84 comments)

(Reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)

cubefox2• 3 years ago

Semi related: Are other people also annoyed by how many projects are using names of completely unrelated famous things? I expected to read some wild association between a game and Franz Kafka, but no, it's about a streaming platform which happens to be named "Kafka". This is getting seriously annoying when you google for some, e.g. historical, term and then your search results are littered by some completely unrelated software/IT project which reuses the name for no reason in particular. "Factorio" is actually an example of how to do better: Just make up your own word!

the_af2• 3 years ago

Even worse, Kafka and Apache Kafka have almost no meaningful connection. According to Wikipedia, the author was reading (Franz) Kafka, who was a writer, and the software system is "optimized for writing". (Franz) Kafka wasn't "optimized" for anything, so this is just whimsical naming. It could just as well have been named Apache Hemingway, or Apache Tolstoy.

Whimsical naming is ok, but can also be confusing and annoying.

cubefox2• 3 years ago

I think many such names are mostly chosen because they kind of sound nice and they have a prestigious association.

titanomachy• 3 years ago

Wow, that's so unrelated. I think I'd tacitly assumed that it was named that because adopting an event-driven architecture results in byzantine and overly complicated software, like the bureaucracies in Kafka's novels.

lbarrett• 3 years ago

"Kafkaesque" suggests people standing in very long queues. It doesn't seem unreasonable as a name for very long queue software.

lionkor2• 3 years ago

I was at one point thinking of making a Kafka (the software) competitor called Tuberculosis, but maybe that was only funny in my head.

radiator• 3 years ago

Yes, it should have been Apache Kafka in this case.

thomastjeffery• 3 years ago

It would help tremendously if the title was changed to "Understanding Apache Kafka with Factorio".

AceJohnny2• 3 years ago

On the one hand, Apache Kafka is a pretty well-known topic on this hacker/web/computing-centric message board, so for most people it doesn't need elucidating.

On the other hand, Factorio arguably has some Kafka-esque aspects!

CptFribble• 3 years ago

Isn't it because of the beneficial SEO? I'm sure we all well understand the benefits of landing a product on the first results page of a well-known historical figure.

ooterness• 3 years ago

I've read a few of Franz Kafka's short stories, but most of them went over my head. As someone who loves Factorio, I was legitimately excited to get more insight.

Now: My disappointment is immeassurable and my day is ruined.

lionkor2• 3 years ago

I never thought how weird it would be to read Kafka in anything but German - I bet it's like reading Dostojewski in anything but Russian, but somehow worse?

not_your_mentat• 3 years ago

And now I think of everything I do in Factorio models.

AbraKdabra• 3 years ago

It's always good to se Cracktorio on the frontpage of HN, hopefully someone will make a similar article showing the similarities between Factorio and drugs.

mistermann• 3 years ago

Or human culture as a video game in general, and humans as semi-rational, semi-aware characters in the video game, except some of the characters are special in that their job is to deceive and exploit other characters, including constructing illusions indistinguishable from reality of who is good and who is bad, what is true and what is not, what we should do and should not do, etc. And to make it all even more exciting, not all of the illusionists are aware of their actual, often hybrid role in the big scheme of things.

Maybe Shakespeare or some famous philosophers would have seen this angle, were video games to exist in their era. Unfortunate timing I guess.

mavu• 3 years ago

I clicked this and was immeasurably disapointed that the article talks about Apache Kafka, and not the author Kafka and how to understand his work with Factorio.

That would be a much much better article.

haswell• 3 years ago

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do.

As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the program, but rarely can the user make sense of what's going on by seeing stuff moving around as the thing executes.

And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain.

gjulianm• 3 years ago

And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain.

I feel the same. It scratches so many itches. I made the mistake of installing Space Exploration mod after the first run… so many hours invested now.

lsaferite2• 3 years ago

SE is such a deep hole to fall into. I feel for you.

throitallaway• 3 years ago

SE is such a slog but I must keep going!

keithnz2• 3 years ago

well, at least you didn't install sea block!

lsaferite2• 3 years ago

it's built in the shape of the reward center of my brain.

This is a very accurate description of how I feel about factorio. Thanks, I'm going to use this going forward.

ineedasername2• 3 years ago

it's built in the shape of the reward center of my brain

Yes, it's like a distillation of the feeling I get from the most enjoyable parts of my job, risking productivity loss from real world responsibilities until the complexity rises high enough to require project organization and advanced planning of tasks that are the least enjoyable parts of my job, along the lines of:

"Crap, I have a sudden urgent need to deal with enemies creeping out from beyond my radar range that will push back operationalization of my proof-of-concept production pipeline. I'd estimate 3 man hours are required to perform a one-off fix on the enemies & radar expansion, maybe 5 to automate long-term... damnit I need a break, let me VPN into work to decompress from Factorio stress."

jay_kyburz• 3 years ago

Yes, but it also embodies the worst parts of my job, which is that there is always more to do and the work is never ending.

cortesoft• 3 years ago

I wonder who the real world equivalent of biters are… sales?

reportgunner• 3 years ago

You should try Foxhole facility maintenance. Makes Factorio stress feel like a tickle.

samsquire• 3 years ago

I am interested in this topic, so thank you for mentioning it.

I haven't played Factorio but this article GIFs helped me understand it, at least from a high level simplistic perspective!

I think most of programming is just logistics: moving data from one place to another, picking and choosing what fields to use for a given purpose and then calling a function with those selected parameters or talking to another system.

I am working on a number of experiments in this area. I'm working on a programming environment which is unlike programming languages where you specify instructions and the state is implied but you work directly with state and instructions are implied.

The problem with nodal editors is that they're not very information dense.

lakomen• 3 years ago

Idk... maybe it's because I'm self taught and have been coding since the age of 11, but I don't find the indirect approach helpful, the opposite.

I believe that's why OO is so popular, people who only know the object way of thinking, who have difficulties with the virtual and abstract like OO and condemn the pragmatic approach.

alexleeds• 3 years ago

OK. 3rd repost of the same material. Really?

keithnz2• 3 years ago

reposts are completely acceptable here.

dang2• 3 years ago

After a year or so: https://news.ycombinator.com/newsfaq.html

aledalgrande• 3 years ago

If you don’t have a lot of time to spare, don’t download Factorio.

Wish I read this months ago. Factorio sucked my life in until I managed to launch that damn ship

henrydark• 3 years ago

Who else was disappointed to find it's Apache Kafka and not Franz Kafka?

Understanding Kafka with Factorio (2019)

Related Stories