Your computer is a distributed system

by carlesfe • 4 years ago

228 149 4 years ago

rbanffy5• 4 years ago

This was true for several home computers since the late 70's. Atari 8-bit computers had all peripherals connecting via a serial bus, each one with its own little processor, ROM, RAM and IO (the only exception, IIRC, was the cassete drive). Commodores also had a similar design for their disk drives. A couple months back a 1541 drive was demoed running standalone with custom software and generating a valid NTSC signal.

Frenchgeek• 4 years ago

( https://youtu.be/zprSxCMlECA )

catern7• 4 years ago

Wow! Reminds me of https://www.rifters.com/crawl/?p=6116

A hydrocephalic demo!

dkersten• 4 years ago

Wow, that is cool!

YZF2• 4 years ago

well, it's been true since a wire has been connecting any two bits. The processor, ROM, RAM are all "distributed" systems internally.

rbanffy5• 4 years ago

That's not what "distributed system" means.

sesuximo• 4 years ago

I think there’s a big difference which is that your computer is allowed to crash when one component breaks whereas a distributed system is typically more fault tolerant.

uvdn72• 4 years ago

This is actually what makes handling the distributed system in a single computer easier – everything crashing together makes it an easier problem.

E.g. you have multiple CPU cachelines, caching different values of a main memory location. And there are different cache coherence protocols to keep them sane. But cache coherence protocols never need to worry about the failure mode when one cacheline is temporarily unavailable but the others are.

So yes, there's a distributed system in each multi-core computer, but it's a distributed system with an easier failure mode.

If you like more analogies between CPU caches and distributed systems, https://blog.the-pans.com/cpp-memory-model-as-a-distributed-... :p

harperlee• 4 years ago

Ideally a peripheral crashing should not crash the whole system.

catern7• 4 years ago

And indeed it does not: Modern operating systems like Linux can perfectly well deal with all kinds of devices crashing or disappearing at runtime. Just like in larger distributed systems.

zerohp• 4 years ago

That's not entirely true. There's usually some level of fault recovery built in but it doesn't extend to the level of allowing any component to fail at any time.

throwaway7875443• 4 years ago

The thing we are missing still is the distributed OS. Kubernetes only exists because of the missing abstractions in Linux to be able to do computation, discovery, message passing/IO, instrumentation over multiple nodes. If you could do ps -A and see all processes on all nodes, or run a program and have it automatically execute on a random node, or if (grumble grumble) Systemd unit files would schedule a minimum of X processes on N nodes, most of the K8s ecosystem would become redundant. A lot of other components like unified AuthZ for linux already exist, as well as networking (WireGuard anyone?).

zozbot23412• 4 years ago

The abstractions are there in Linux, largely imported from plan 9. And work is ongoing to support further abstractions, such as easy checkpoint/restore of whole containers. Kubernetes is a very new framework intended to support large-scale orchestration and deployment in a mostly automated way, driven by 'declarative' configuration; at some point, these features will be rewritten in a way that's easier to understand and perhaps extend further.

MisterTea3• 4 years ago

The abstractions are there in Linux, largely imported from plan 9.

Which abstractions are those?

NavinF4• 4 years ago

Eh I can’t see Linux getting a built-in distributed kv store (etcd) any time soon. Same goes for distributed filesystems. All you have out of the box is nfs which gives you the worst of both worlds: Every nfs server is a SPOF yet these servers don’t take advantage of their position to guarantee even basic consistency (atomic appends) that you get for free everywhere else.

And besides how would you even implement all those features you listed without recreating k8s? A distributed “ps -A” that just runs “for s in servers; ssh user@$s ps; done” and sorts the output would be trivial, but anything more complex (e.g. keeping at least 5 instances of an app running as machines die) requires distributed and consistent state.

zozbot23412• 4 years ago

requires distributed and consistent state

Distributed yes, but not necessarily consistent. You can use CRDTs to manage "partial, flexible" consistency requirements. This might mean, e.g. sometimes having more than 5 instances running, but should come with increased flexibility overall.

throwaway7875443• 4 years ago

Fwiw those features existed in Mosix (a Linux SSI patch) 2 decades ago... I feel like we could probably do it again

In terms of CAP, yeah it might not have been technically as reliable. But there's different levels of reliability for different applications; we could implement a lot of it in userland and tailor as needed

wwalexander• 4 years ago

Plan 9 was designed in this way, but never took off.

Rob Pike:

This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and calling it revolutionary. I sorely miss the unified system view of the world we had at Bell Labs, and the way things are going that seems unlikely to come back any time soon.

nautilus12• 4 years ago

Glad to see Plan 9 getting some love in the comments even if it didn't make it into the article.

jlpom2• 4 years ago

This describe more a Single System Image [0] to me (WPD includes Plan 9 as one but considering it does not does not supports process migration I find it moot). LinuxPMI [1] seems to be a good idea but they seems to be based on Linux 2.6, so you would have to heavily patch newer kernel. The only thing that seems to support process migration with current software / still active are CRIU [2] (which doesn't support graphical/wayland programs) and DragonflyBSD [3] (in their own words very basic).

[0] https://en.wikipedia.org/wiki/Single_system_image [1]: http://linuxpmi.org [2]: criu.org [3]: https://man.dragonflybsd.org/?command=sys_checkpoint&section...

gnufx7• 4 years ago

If you yearn for Plan 9 -- I'm not sure I do -- Minnich's current incarnation of the inspiration seems to be https://github.com/u-root/cpu

monocasa• 4 years ago

Are there any good walkthroughs of what a good, distributed plan 9 setup looks like from either a development or a administration perspective? Particularly an emphasis on many distributed compute nodes (or cpu servers in plan 9 parlance).

NavinF4• 4 years ago

Meh. Every time a 9p server dies, every client dies. Plan9 is not comparable to k8s.

jasonwatkinspdx2• 4 years ago

I think Rob is right to call out the problem, but is being a bit rose colored about Plan 9.

Plan 9 was definitely ahead of its time, but it's also a far cry from the sort of distributed OS we need today. "Everything is a remote posix file" ends up being a really bad abstraction for distributed computing. What people are doing today with warehouse scale clusters indeed has a ton of layers of crap in there, and I think it's obvious to yern for sweeping that away. But there's no chance you could do that with P9 as it was designed.

ff317• 4 years ago

There were older attempts at this stuff, in the 90s with "Beowulf" clusters that had cross-machine process management and whatnot. It's a lot harder than it seems to make this approach make sense in the real world, as the abstraction hides important operational details. The explicit container + orchestration abstraction is probably closer to the ideal than trying to stretch linux/systemd/cgroups across the network "seamlessly". It's clearer what's going on and what the operational trade-offs are.

mnd999• 4 years ago

Imagine a Beowulf cluster of hot grits in soviet Russia with CowboyNeal.

gnufx7• 4 years ago

in the 90s with "Beowulf" clusters

In case of any confusion, that sort of thing wasn't a generic Beowulf feature, but it sounds like Bproc. I don't know if it's still used. (The Sourceforge version is ancient.)

https://updates.penguincomputing.com/clusterware/6/docs/clus... https://sourceforge.net/projects/bproc/

Containers actually only make it harder to "orchestrate" your distributed processes in an HPC system.

oceanplexian• 4 years ago

Am I the one who doesn’t want this?

The entire point of UNIX philosophy (Which seems to be something they aren’t teaching in software development these days) is to do one thing and do it well. We don’t need Linux operating operating as a big declarative distributed system with a distributed scheduling systems and a million half-baked APIs to interact with it, the way K8s works. If you want that you should build something to your specific requirements, not shove more things into the kernel.

goodpoint• 4 years ago

Linux/UNIX does not have to turn into a mess like k8s to be natively distributed. Plan9 was doing it with a tiny codebase in comparison.

Karrot_Kream4• 4 years ago

The UNIX philosophy made more sense as an abstraction for a computer when computers were simpler. Computers nowadays (well at least since 2006-ish) have multiple cores executing simultaneously with complicated amounts of background logic, interrupt-driven logic, shared caches, etc. The UNIX philosophy doesn't map to this reality at all. Right now there's no set of abstractions except machine code that exposes the machine's distributed systems' in a coherent abstraction. Nothing is stopping someone else from writing a UNIX abstraction atop this though.

random3144• 4 years ago

The Unix philosophy was a reasonably good model decades ago. But I think it is over romanticized.

It's binary blob design is no good for security, as opposed to a byte code design like Forth. Its user security model was poor and doesn't help with modern devices like phones. Its multiprocess model was ham fisted into a multithreading model to compete with windows NT. Its asynchronous i/o model has always been a train wreck even compared to NT. Its design creates performance issues, especially in multiproc networking code with needless amount of memcopys. Now folks are rewriting the networking stack in user space. Its software abstraction layer was some simple scheme from the 70s which has fragmented into crazy number of implementations now. Open source developers still complain about how much easier it is to build a package for windows, as opposed to linux. It was never meant to be a distributed system either. Modern enterprise compute cannot scale by treating and managing each individual VM as it's own thing with clusters held together by some sysadmins batch scripts.

aseipp3• 4 years ago

All of this was possible with QNX literally decades ago, and it didn't need whatever strawman argument you're making up in your head in order to accomplish it. QNX was small, fast, lean, real-time, distributed, and very powerful for the time. Don't worry, it even had POSIX support. A modern QNX would be very well received, I think, precisely because taking a distributed-first approach would dramatically simplify the whole system design versus tacking on a distributed layer on top of one designed for single computers.

Which seems to be something they aren’t teaching in software development these days

This is funny. Perhaps the thing you should have been taught instead is history, my friend.

pjmlp6• 4 years ago

The philosophy that is cargo culted and was never taken seriously by any commercial UNIX.

evandrofisico• 4 years ago

Actually, at some point in the 2.4 kernel it was possibile to do that, with single image systems, such as openmosix, that handled process discovery, computation and much more, but underneath the simple user interface it was complex, kinda insecure and so, was never abandoned and never ported to newer versions.

icedchai• 4 years ago

I think OpenVMS did this... in the 80's.

Karrot_Kream4• 4 years ago

I think you're looking at the wrong abstraction level. You're thinking on a node (computer) basis. Even on a single computer, many of the things that happen are distributed. DMA controllers, input interrupts, kernel-forced context switches, there's a lot going on there but we still pretend that our computers are just executing sequential code. I agree with the OP and think it's high time we treat the computer as the distributed system it is. Fuschia and GenodeOS are both making developments in this direction.

f0e4c2f7• 4 years ago

I very much agree with this and while Kubernetes is better than a poke in the eye, I look forward to the day when there is a true distributed OS available in the way you describe. It's possible Kubernetes could even grow into that somehow.

als0• 4 years ago

I remember the Barrelfish OS was trying to tackle this problem head on https://barrelfish.org/

gnufx7• 4 years ago

Distributed computation with message passing (and RDMA) is the essence of HPC systems. SGI systems supported multi-node Linux single system images up to ~1024 cores a fair few years ago, but they depend on a coherent interconnect (NUMAlink, originally from the MPIS-based systems under Irix).

However, you don't ignore the distributed nature of even single HPC nodes unless you want to risk perhaps an order of magnitude performance loss. SMP these days doesn't stand for Symmetric Multi-Processing.

zozbot23412• 4 years ago

Distributed shared memory is feasible in theory even via being provided in-software by the OS. You're right that this would not change the physical reality of message passing, but it would allow a single multi-processor application code to operate seamlessly using either shared memory on a single node, or distributed memory on a large cluster.

uvdn72• 4 years ago

Abstract a fleet of machines as single super computer sounds nice. But how about partial failures? It's something that a real stateful distributed system would have to deal with all the time but a single host machine almost never deals with (do you worry about a single cacheline failure when writing a program?).

marcosdumay2• 4 years ago

There is a huge amount of research about distributed OSes (really, they were very fashionable at the 90's and early 00's). Plenty of people worked on this problem, and it's basically solved (as in, we don't have any optimal solution, but it won't be a problem on a real system).

cultofmetatron2• 4 years ago

I built my startup in elixir and the erlang VM provides all of these. its kind of amazing. Things we've been able to have out of the box

* a metrics dashaboard that gives you a ps -A for all our nodes

* intermachine pubsub without having to setup a third party message queue

* auto failover for all our microservices

* spinning up a microservice takes about as much effort as adding a controller in rails

* cronjobs where one node can trigger a job on another node in the network. Hell, we have crons scheduled where we don't even know which machine it'll run on. it just gets done

sethammons• 4 years ago

I was thinking elixir/erlang too. I've only been using it for a couple of months but I've quickly come to the conclusion that their claims of robustness are not in line with what I want/need. For example, the pubsub lacks persistence and if the node that carries that info dies, you lose that data. There is no built in consensus that tries to maintain state in the face of failure, so you bring in Oban. I've yet to experience the advantages of elixir. Sure, I can hot reload a module, but I spend probably an hour a day waiting for things to compile. I prefer k8s and Go but figure that may be because I'm still new to the ecosystem.

pjmlp6• 4 years ago

Kubernetes only exists because people wanted to do Application Servers in any language, and now they are rediscovering them trying to sell us on Kubernetes + WebAssembly, the irony.

taeric2• 4 years ago

So much of programming languages is to hide the distributed nature of what the computer is doing on a regular basis. This is somewhat obvious for thread abstractions where you can get two things happening. It is blatant for CUDA style programming.

As this link points out, it gets a bit more difficult with some of the larger machines we have to keep the abstractions useful. That said, it does mostly work. Despite being able to find and harp on the areas that it fails, it is amazing how well so many of the abstractions have held up.

Would be neat to see explicit handling of what features are basically completely hiding distributed nature of the computer.

jayd162• 4 years ago

The abstractions aren't just for simplicity. In many cases, ensuring that the distributed nature is unknown or unobserved means the system can make different decisions without affecting the program. This leaves room for flexibility in the system design.

zozbot23412• 4 years ago

The distributed nature can never be unobserved, by definition. What a well-designed distributed system can do is offer facilities to enable useful constraints on its operation, that might then be used as necessary via a programming language.

Karrot_Kream4• 4 years ago

Maybe? Alternatively by bringing the distributed nature up front-and-center you can have more flexible designs. If I could timeout my drawing routine when the screen has already refreshed (or context has been stolen from the OS) then I have a lot more flexibility in how to recover instead of pretending to do my best and ending up with a lot of screen tearing when I miss my frame budget.

taeric2• 4 years ago

Distributed problems that are largely timing are easy to see in this nature. In large, the whole synchronize on a clock idea is invisible to programmers.

That said, there are times when it isn't hidden, but only taken out of your control. I guess the question is mainly in how to move them to first class objects to reason about?

simne3• 4 years ago

Unfortunately, this idea fights vs idea of least responsibility.

Because, user level programs are all at one level of abstraction, and this distribution is distributed over many levels of abstraction.

So in desktop systems, mean mostly successors of business micro machines, access to other levels of abstraction intentionally hardened for measures of security and reliability. The same thing applied to crowd computing - there also vps's are isolated from hardware and from other vps's.

These measures usually avoided in game systems and in embedded systems, but they are not allowed to run multiple programs from independent developers (for security and reliability), and their programming magnitudes more expensive than desktops and even server side (yes, you may surprised, but game consoles software in many case more reliable than military, and usually far surpass business software).

To solve this contradiction, need some totally new paradigms and technologies, may be some revolutionary, like usage of GAI to write code.

andrey_utkin• 4 years ago

Your body is a distributed system. Your brain is a distributed system. A live cell is a distributed system. A molecule is a distributed system. In other news, water is wet.

pkilgore• 4 years ago

What is the kernal and the bus for the cloud?

simne3• 4 years ago

These all now are virtual state machines, which store some state and convert all kernel/bus behavior to interaction with connected via network devices.

At the moment there are lot of such devices - exists for sure many full featured, like Raspberry; but also there are network connected ATA drives, network connected sensors, RAM, ROM (Flash); BTW IEEE 1394 FireWire is serial interface, could been used as networking bus; exists adapters ethernet-usb (and many commodity devices work well with such connection), so virtually anywhere could been considered as connected via network bus. Even exists USB 3.0 to PCIe adapter, to use PCIe device throw USB connection.

And in reality exists problem, that FireWire so distributed, that it where possible on Macs with FireWire interface, to read memory via this interface.

So hardware and software exists, but need some steps to make it's usage safe.

Koshkin• 4 years ago

Yes, and concurrency is, in fact, an implementation detail. Which is why I think that in most applied scenarios it should be hidden, and taken care of, by the compiler.

amelius• 4 years ago

Yes, and your computer is a ball of interconnected microservices too.

JL-Akrasia• 4 years ago

You are also a distributed system.

__turbobrew__• 4 years ago

Some more distributed than others

dredmorbius• 4 years ago

The people are already here, they're just not very evenly distributed.

hsn915• 4 years ago

Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing. Sure, storage devices can come and go, but that's been the case since forever, and most programs are not written to handle this edge case gracefully. Except for the OS kernel.

The links between the components of your computer are solid and cannot fail like actual computer network connections.

In terms of "CAP" theorom, the system has no Partition tolerance. If one of the the links connecting CPUs/GPUs/RAM breaks, all hell breaks loose. If a single instruction is not processed correctly, all hell might break loose.

So I find the analogy misleading.

aidenn02• 4 years ago

I think that TFA gets it exactly backwards. It's not that we will be able to treat multi-node systems as non-distributed it's that single-nodes will have to start being treated like distributed systems.

The links between the components of your computer are solid and cannot fail like actual computer network connections.

I've personally had this disproven to me on multiple occasions.

catern7• 4 years ago

I've personally had this disproven to me on multiple occasions.

That sounds like interesting stories! Can you elaborate?

StillBored2• 4 years ago

There have been machines tolerant to CPU and Mem failures, and to a certain extent this sorta works on some of the higher end machines that support ram/cpu hotplug. (historically see hp/tandem/nonstop, sunos/imp, etc).

The problem is linux's monolithic model, doesn't work well for kernel checkpoint/restore despite it actually supporting hotplug cpu/ram it they have to be gracefully removed.

So, this is less about the machine being distributed, and more about the fact that linux is the opposite of a microkernel/etc that can isolate and restart its subsystems in the face of failure. Its also sorta funny that while these types of operations tend to need to be designed into the system, the last major OS's designed this way were done in the 1980's.

dwohnitmok2• 4 years ago

I know of no OSes that are resilient to CPU cores producing wrong results (or incorrect mem results: I consider ECC a lower level concern that is not part of the OS), whereas a lot of distributed consensus algorithms have this built into their requirements. EDIT: I have heard through the grapevine that something like this might be done for aerospace, but I have no personal experience with that.

I agree with parent. The major reason why programming on a single computer is easier than a distributed system is that we assume total resilience of various components that we cannot for a distributed system.

From the article:

This offers hope that it is possible to some day abstract away the distributed nature of larger-scale systems.

To do this is not a question of software abstractions, but hardware resilience. If we have a network which we can reasonably assume to have 100% uptime and absolutely no corruption between all its components then we can program distributed systems as single computers.

gnufx7• 4 years ago

It doesn't count as resilient in the mainframe sense, but in an effort to encourage system management, I ran the Node Health Check system on our "commodity" HPC cluster and found multiple failed DIMMs and a failed socket no-one had noticed. (I'd had enough alerts from that on a cluster I managed.)

catern7• 4 years ago

Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing

That's incorrect.

There are plenty of machines/OSs which are (or can be) resilient to a CPU failing; Linux, for example. From the OS point of view, you just kill the process that was running on the CPU at the time and move on.

Resilience to spontaneous RAM failures is rarer but possible.

bee_rider2• 4 years ago

Killing the processes running on the compute element seems not very graceful, right? I'd expect a gracefully handled failure to have some state staved from which the computation can be continued.

Which would be overkill on a single node, given that CPUs don't really fail all that often.

imtringued• 4 years ago

The article also ignores that e.g. the CUDA API looks nothing like a local function call. People are explicitly aware when they are launching GPU kernels.

bee_rider2• 4 years ago

You can 'disable' a core in Linux pretty easily, although I'm not sure to what extent you'd consider this graceful (in the sense that you write to a system file and then some magic, which may be arbitrarily complicated I guess, happens in the background. So it doesn't seem equivalent to just yanking a core from the package, if that were possible).

ilaksh• 4 years ago

This proves that conventional wisdom (such as the idea that abstracting distributed computation is unworkable) is often wrong.

What happens is enough people try to do something and can't quite get it to work quite right that it eventually becomes assumed that anyone trying that approach is naive. Then people actively avoid trying because they don't want others to think they don't know "best practices".

Remember the post from the other day about magnetic amplifiers? Engineers in the US gave up on them. But for the Russians, mag amps never became "unworkable" and uncool to try, and they eventually solved the hard problems and made them extremely useful.

Technology is much more about trends and psychology than people realize. In some ways, so is the whole world. It seems to me that at some level, most humans never _really_ progress beyond middle-school level.

The starting point for analyzing most things should probably be from the context of teenage primates.

it• 4 years ago

The Erlang VM (BEAM) can be viewed as a distributed operating system, or at least the beginnings of one.

simne3• 4 years ago

Agree, and could add, that ALL Erlang flavors (exists at least 4 independent implementations for different environments and for different targets) are distributed.

And Erlang is based on relatively new syntax from Prolog, which also have cool ideas.

WestCoastJustin• 4 years ago

Great post called "Achieving 11M IOPS & 66 GB/s IO on a Single ThreadRipper Workstation" [1, 2] that basically walks through step-by-step that your computer is just a bunch of interconnected networks.

Highly recommend the post if you're into this and also sort of amazing how far single systems have come. You can basically do "big data" type things on this single box.

[1] https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...

[2] https://news.ycombinator.com/item?id=25956670

AdamH121132• 4 years ago

[The fact that computers are made of many components separated by communication buses] suggests that it may be possible to abstract away the distributed nature of larger-scale systems.

This is a neat line of thought, but I don't think it can go very far. There is a huge difference in reliability and predictability between small-scale and large-scale systems. One way to see this is to look at power supplies. Two ICs on the same board can be running off of the same 3.3V supply, and will almost certainly have a single upstream AC connection to the mains. When thinking about communications between the ICs, you don't have to consider power failure because a power failure will take down both ICs. Compare this to a WiFi network where two devices could be on separate parts of the power grid!

Other kinds of failures are rare enough to be ignored completely for most applications. An Ethernet cable can be unplugged. A PCB trace can't.

I used to work with a low-level digital communication protocol called I²C. It's designed for communication between two chips on the same board. There is no defined timeout for communication. A single malfunctioning slave device can hang the entire bus. According to the official protocol spec, the recommended way of dealing with this is to reset every device on the bus (which may mean resetting the entire board). If a hardware reset is not available, the recommendation is to power-cycle the system! [1]

Now I²C is a particularly sloppy protocol, and higher-level versions (SMBus and PMBus) do fix these problems, so this is a bit of an extreme example. But the fact that I²C is still commonly used today shows how reliable a small-scale electronic system can be. Even at the PC level, low-level hardware faults are rare enough that they're often indicated only by weird behavior ("My system hangs when the GPU gets hot"), and the solution is often for the user to guess which component is broken and replace it.

[1] Section 3.1.16 of https://www.nxp.com/docs/en/user-guide/UM10204.pdf

charcircuit2• 4 years ago

A PCB trace can't.

Sure, but physical damage can disconnect it.

anyfoo• 4 years ago

But in that case we accept the device as "broken" and need to replace or repair it. If you are lucky the PCB trace was only relevant to a single feature, say an LED. If you are unlucky, it's part of the memory bus and everything is toast. But no significant engineering went into making one more resilient than the other.

Whereas in a distributed system, a single broken communication line, even or especially if it's extremely important, still means that the distributed system has to recover somehow, preferably gracefully.

littleJeck• 4 years ago

Now I²C is a particularly sloppy protocol

I know this pain as an embedded software monkey. So many edge cases unthought of and no method to gracefully fail. Not to mention you always find some fault with the I²C implementation on the SOC or slave device you want to talk to.

I’ve spent weeks debugging bus lockup problems only to find erratas for the silicon, or find this particular chip needs an extra 100ms to startup than others.

It’s the one communication protocol that o always have problems with. Nothing else has caused me as many headaches.

a9h74j2• 4 years ago

This is something which worries me with Oxide using I2C (according to their board bring-up podcast), and using an auto-direction-switching level-shifter on the I2C bus. I hope and I am sure they can make it work, but to me that comes closer to violating a (core?) value of determinism and observability. Of course if it is only one well-studied component or two, perhaps out of design/supply necessity, this might be a very small risk in the larger scope.

benreesman• 4 years ago

Eric Brewer thinks this is a good point of view on such things:

https://codahale.com/you-cant-sacrifice-partition-tolerance/

L1-blockchain entrepreneurs and people who got locked into MongoDB aside, I think most agree.

syngrog66• 4 years ago

once you learn to bias to thinking in terms of message passing between actors, and, bias to having immutable shared state, then,a lot of problems become easier to decompose and solve elegantly, esp at scale

jmull• 4 years ago

This is something unique: an abstraction that hides the distributed nature of a system and actually succeeds.

That's not even remotely unique.

OP is grappling with "the map is not the territory" vs. maps have many valid uses.

Abstractions can be both not accurate in every context and 100% useful in many, many common contexts.

Also (before you get too excited), abstractions have quality: there are good abstractions -- which are useful in many common contexts -- and bad abstractions -- which overpromise and turn out to be misleading in some or many common contexts.

I'll put it this way: the idea that The Truth exists is a rough (and not particularly useful) abstraction. If you have a problem with that, it just means you have something to learn to engage reality more fruitfully.

tonymet• 4 years ago

i recommend people model their apps this way. spin up more threads than needed, one each for api , DB , LB, async, pipelines etc. you can model an entire stack in one memory space. It's a great way to prototype your complete data model before scaling to the proper solutions. Lots of design constraints are found this way . everything looks great on paper but then falls apart when integrating layers.

alexisread• 4 years ago

There are lots of good resources in this area: The programming language of the transputer https://en.m.wikipedia.org/wiki/Occam_(programming_language)

Bluebottle active objects https://www.research-collection.ethz.ch/bitstream/handle/20.... with some discussion of DMA

Composita components http://concurrency.ch/Content/publications/Blaeser_Component...

Mobile Maude (only a spec) http://maude.sip.ucm.es/mobilemaude/mobile-maude.maude

Kali scheme (atop Scheme48 secure capability OS) https://dl.acm.org/doi/pdf/10.1145/213978.213986

Kali is probably the closest to a distributed OS, supporting secure thread and process migration across local and remote systems (and makes that explicit), distributed profiling and monitoring tools, etc. It is basically an OS based on the actor model. It doesn't scale massively as routing nodes was out of scope (it connects all nodes on a bus), but that can easily be added.

Extremely small (running in 2mb ram), it covers all of R5rs, and the VM has been adapted to bare metal.

I feel that there is more to do, but a combination of those is probably the right direction.

saltcured• 4 years ago

I think that someone newly interested in this should consider the longer history of distributed OS concepts too. What are you trying to do differently? What set of tradeoffs give you a well-defined solution space which is a gap in current approaches?

https://en.wikipedia.org/wiki/Sprite_(operating_system) was a project at UC Berkeley that ran building-scale networks of workstations as a distributed OS. It ended in 1992 but produced innovations such as the log-structured filesystem along the way.

Various commercial products like Domain/OS or SGI Irix also were distributed operating systems at different scales. It seems like these lost out to the more typical HPC solutions with distributed/parallel OS instances limited to each node. Or on the other extreme, mainframe systems continue to be a kind of distributed OS built for high availability and scaling in a very controlled environment.

kmod• 4 years ago

I worked on this for my masters thesis! The thesis was for a specific part but the group worked on the problem as a whole, see https://dspace.mit.edu/handle/1721.1/49844

IMO there are two things that make the current abstraction of a computer as a unit make sense:

- You (mostly) don't have to try to handle partial failures within a computer. Partial failures are what make distributed systems hard.

- The difference in communication costs between two cores in a single machine is several orders of magnitude lower than communicating with a separate machine using commodity technologies. So while yes, "it's all just distributed" and you can use a common abstraction, a large enough constant factor difference means that you still will have to look through the abstraction to build a performant system.

zozbot23412• 4 years ago

The latency involved in communicating with a separate machine is comparable to loading data from disk. So yes, it is slow, but not so slow that you couldn't reuse many of the abstractions involved in programming a single machine.

Your computer is a distributed system

Related Stories