Using Java's Project Loom to build more reliable distributed systems

https://jbaker.io/2022/05/09/project-loom-for-distributed-systems/

by CHY872 • 4 years ago

205 91 4 years ago

_benedict2• 4 years ago

Cassandra already does this[1] (and a whole lot more besides to introduce chaos to the simulation), and just accepts that there's a huge penalty of pausing and scheduling threads.

I'm really looking forward to being able to use Loom to run much more efficient cluster simulations.

[1] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-10...

CHY8726• 4 years ago

What sort of performance characteristics does Cassandra see from its implementation? I was fairly impressed with the 2.5M context switches per second (a fairly random unit, I know) I saw from virtual threads.

Have you found that the approach has made it easier for you to ship improvements to Cassandra?

_benedict2• 4 years ago

It's very slow, particularly since introducing byte weaving to introduce chaos in concurrency-sensitive places. Presently it takes a single thread in the region of twenty minutes to simulate a cluster from startup to shutdown, and to run a thousand linearisable operations (down from a couple of minutes prior to those changes). I haven't got any numbers on the rate of context switches per second, but it is certainly a great deal less than 2.5M/s.

This kind of approach is in my opinion a _requirement_ for work on distributed consensus protocols, which the Cassandra community is improving at the moment: the upcoming 4.1 release has some major improvements to its Paxos implementation that were validated by this simulation, and the following release is expected to contain a novel distributed consensus protocol we have named Accord, for supporting multi-key transactions.

So, I'm not sure it has affected the speed or ease of shipping improvements, rather than enabled a class of work that would have previously been impossible to do safely (IMO).

exabrial• 4 years ago

This was linked in the article and is a fascinating read by itself: https://shipilev.net/blog/2016/close-encounters-of-jmm-kind

dang• 4 years ago

Discussed at the time:

Close Encounters of the Java Memory Model Kind - https://news.ycombinator.com/item?id=11955392 - June 2016 (65 comments)

vips7L4• 4 years ago

Shipilev’s entire blog is worth reading. He’ll even email you back if you have questions.

I really enjoy his anatomy quarks series:

https://shipilev.net/jvm/anatomy-quarks/

tadfisher• 4 years ago

So Loom is just green threads? You have to implement a CPS transformation on top? If so, great decision!

cogman103• 4 years ago

Loom is just green threads (referred to as virtual threads).

heartofgold• 4 years ago

For any older java programmers that may have a bad taste in their mouth when they think of green threads: when Java first came out, it had green threads, and all N threads that were spawned were tied to a single kernel thread. It also required cooperative multithreading, where one green thread may need to yield.

Java upgraded to native threads, and then you could have N java threads bound to N kernel threads. This was way better, but had downsides: You're limited on how many threads you can spawn, you need a threadpool to help manage, and any long-running tasks could effectively deplete your thread-pool.

With Loom, now you have M green threads mapped to N kernel threads. These green threads are way cheaper to spawn, so you could have thousands (millions even?) of green threads. Blocking calls won't tie up a kernel thread. So if you have many long-running IO tasks, they aren't going to waste a kernel thread and have it sit around idle waiting on IO. This is similar to async libraries, but without the mental overhead. You should be able to just code synchronously and the JVM will take care of the rest.

shellac• 4 years ago

It's more accurate to say Loom is [a particular type of] continuations in the JVM, which can be used to implement green threads / fibers.

You can see the implementation of VirtualThread here: https://github.com/openjdk/loom/blob/fibers/src/java.base/sh...

This uses the internal 'one-shot delimited continuation': https://github.com/openjdk/loom/blob/fibers/src/java.base/sh...

So, at least in principle, there is scope for other styles of concurrency to be implemented over this.

electricshampo1• 4 years ago

Unlike goroutines, seems here you have control over the execution schedule for the virtual threads if you provide an executor. This is pretty great.

Think this will obsolete go over the next few decades.

cogman103• 4 years ago

Not on the first go around. AFAIK, they are looking at exposing more of the internal scheduling API but that's likely not going to be a part of the initial release.

The executor services referred to in the blog are for the order of execution of tasks on the virtual thread pool. For a "virtualThreadExecutor" service, every task will get a virtual thread and scheduling will happen internally.

You can still use a fixed thread pool with a custom task scheduler if you like, but probably not exactly what you are after.

CHY8726• 4 years ago

This blogpost does rely on plugging in an executor. While the API was removed, it’s one private variable away (documented in footnote). As you say, it seems like it’s an ‘on the way, but later’ thing - the last Loom preview I used (a while ago) actually had the API so when I started drafting this post I was unhappily surprised!

jjice• 4 years ago

I don't think it will. If everyone was clamoring for Java and settled on Go only because of goroutines, then sure, but I think Go was liked for a lot of reasons aside from that. I also don't often see people complain about wanting more control over the scheduler for Go (could be that I just miss those).

I'd be surprised if Go adoption plummeted because of this, but who knows, I sure don't have a crystal ball.

lostcolony2• 4 years ago

Sane concurrency is -one- of the reasons people reach for Go, and sure, that may no longer be a differentiator. But it's definitely not the only one I've heard people toss around (and, agreed, I've never heard anyone bemoan the lack of control of the scheduler). In fact, the introduction of virtual threads and no new memory semantics I think means it still fails one of the main benefits of goroutines (channels and default copying semantics); everything in JVM land by default is still going to use shared memory and default pass by reference semantics.

I think it's all a moot point though, as it basically just demonstrates the next iteration of Paul Graham's Blub Paradox. With every iteration of new improvements for the JVM it reinforces the belief of many that the JVM is the best tool for every job (after all, it now just got cool feature y they just now learned about and can use and OMG Blub-er-Java is so cool, who needs anything else?!), and reinforces the belief of many others that the JVM is playing catchup with other languages (it only just -now- got feature y) and there are often better tools out there.

didip• 4 years ago

Java moves a lot slower so I don't think it will obsoletes Go.

If anything, if Loom is great, then it will keep Go on its toes and hopefully Go will also evolved due to external pressure.

slantedview• 4 years ago

I beg to differ. Go has been moving at a glacial pace. Generics took forever to implement, aren't even feature complete (no type parameters on methods), and have no integration with the standard lib. Meanwhile Java is adding lots of new features and versions at a quick pace.

kjeetgill• 4 years ago

One of the unsung heroes of go is how goroutines sit on top of channels + select. Blocking and waiting on one queue is easy, blocking and waiting on a set of queues waiting for any to get an element is a good deal trickier. Having that baked into the language and the default channel data-structures really does pay dividends over a library in a case like this.

You can kinda do this with futures but I suspect it'll be wildly inefficient. I really hope Java get's something to fill this niche. We already have a menagerie of Queue, BlockingQueue, and TransferQueue implementations. What's a few more?

ackfoobar6• 4 years ago

Having that baked into the language and the default channel data-structures really does pay dividends over a library in a case like this.

Kotlin coroutines have the bare minimum in the language, and implement the rest (e.g. channel, select, `go`/`launch`) in libraries. Could you explain what the dividends for Go are?

MrBuddyCasino2• 4 years ago

Channels on the JVM would be sweet. You can do the same with Futures, and its probably not even slower, but it is a lot more clunky. I suspect its never gonna happen, too big a change. Maybe Kotlin will do it.

sudarshnachakra• 4 years ago

I guess the Structured Concurrency JEP below addresses the problems you'll get to solve. It'll enable things like AND / OR combinations of virtual threads which IMHO looks like a better way to solve this rather than having a special syntax for select.

https://openjdk.java.net/jeps/8277129

But frankly I'm afraid of how these changes affect garbage collection since more and more vthread stacks are going to be in the heap (I hope they are contemplating some form of deterministic stack destruction along with the above JEP).

Thaxll6• 4 years ago

Go does not need 128MB of memory to run hello world in a container.

People don't pick up Go over Java because of goroutines, Java is still and will forever be an "enterprise" language behind many layers of abstractions.

kaba06• 4 years ago

Low memory consumption also has a price in this case. On a 1 TB server machine guess which platform will have better throughput by far? Go’s GC will die under that load.

Writing a “hello world”-scoped microservice is a tiny niche.

CHY8726• 4 years ago

The JVM is a master of gradually closing the gap. Over the last decade:

- Garbage collectors have required far less tuning; with G1, Shenandoah, ZGC it's likely that your application will need little tuning on normal sized heaps.

- modules, jlink etc allow one to build a much smaller Java application by including only those parts of the JVM that one needs.

- Graal native images are real. These boast a far lower startup overhead and much lower steady state memory usage for simpler applications.

Probably my counterexample of choice is this: https://github.com/dainiusjocas/lucene-grep - it uses Lucene, one of the best search libraries (core of Elasticsearch, Solr, most websites), which is notoriously not simple code, to implement grep-like functionality. In simple cases, they demonstrate a 30ms whole process runtime with no more than 32MB of RAM used (which looks suspiciously like a default).

The JVM is fast becoming a bit like Postgres... one of those 'second best at everything' pieces of tech.

native_samples2• 4 years ago

FWIW, this criticism no longer applies to people using AOT compilation. From my macOS laptop:

     /usr/bin/time -l ./hello-world
    Hello World!
        0.00 real         0.00 user         0.00 sys
             3231744  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
                 841  page reclaims
                   1  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   2  voluntary context switches
                   4  involuntary context switches
            22395110  instructions retired
            18507246  cycles elapsed
             1294336  peak memory footprint

So "peak memory footprint" for hello world is 1.2 MB and it starts instantly.

Now, not everyone can/will use AOT compilation. It's slow to compile and peak performance is lower unless you set up PGO, plus it may need a bit of work in cases where apps assume the ability to do things like generate code on the fly. But Go can't do runtime code generation easily at all, and if you are OK with those constraints, you get C-like results.

MobiusHorizons2• 4 years ago

I'm curious why you think a language feature would obsolete an entire programming language. Do you imagine that go programmers secretly wish they were writing Java syntax, in my experience this is very much not true.

EdwardDiego5• 4 years ago

I'm tempted to make cheap shots about sets that do actual set operations without a for loop, but I'm hoping Go >=1.19 will start introducing some nice generic collections in the stdlib.

d3nj4l3• 4 years ago

I've heard some variation of "$java_feature will make $language obsolete" for years now, most recently wrt kotlin/scala, and it's never held true. It's great for the people who use Java, but there are tons of reasons why other people use other languages.

EdwardDiego5• 4 years ago

Java record vs. Kotlin dataclass?

gorjusborg• 4 years ago

I've noticed the same behavior.

Imitation can only get you so far. Java is changing, and in many cases for the better, by absorbing features from other languages. However, I still think several other languages do a better job curating features to fit a niche.

That said, Loom appears to be a serious upgrade for JVM languages. Now, if startup could get an order of magnitude faster...

theptip• 4 years ago

This is an interesting idea. I've been idly thinking about how to make a framework for detecting DB anomalies in Django applications (say, missing transactions, race conditions, deadlocks etc.) which can be hard to detect. You can strap your application to Jepsen but this is a more white-box approach and probably harder to grok for the average Python developer.

I like the OP's idea of using a virtual thread implementation to parallelize the application-layer, while having the test code implement a custom executor to control the interleaving of the units-of-work being scheduled. I can see a few ways this approach could be used more widely; for example in Django you could write a DB driver wrapper and have this as the "test executor". Or for remote API requests, run the test code in green threads and then have a test executor that intercepts and then then chooses how to interleave the requests/responses.

zmj• 4 years ago

I once did something similar in C# to deterministically test some concurrency-heavy code (injecting custom TaskScheduler instances). It gave me a lot of confidence that it actually worked.

rhplus• 4 years ago

If you're looking for similar concurrency testing in the dotnet world, checkout Coyote: https://microsoft.github.io/coyote/

Explainer here: https://innovation.microsoft.com/en-us/exploring-project-coy...

kendallgclark• 4 years ago

The FoundationDb team is working on distributed systems simulation testing via a new startup... Antithesis. See https://antithesis.com/

Stardog, enterprise knowledge graph platform, (stardog.com), is an early adopter of Antithesis and we've found it to be very helpful in HA Cluster, distributed consensus, etc.

Not a stakeholder, just a satisfied early adopter.

ryanworl• 4 years ago

This is the first reference to Antithesis being used I've seen since it was announced.

Can you describe what their tools are doing for you and how you're integrating with them?

no_wizard2• 4 years ago

I feel like the unsung winner of Project Loom is going to be Clojure. Its already immutable first data structures, it should be relatively straightforward for the Clojure project to expose the benefits of Project Loom to their ecosystem, as a language its designed to fit well its execution model.

pvorb• 4 years ago

Why should Clojure benefit more from Loom than other languages? I think it simplifies reactive/concurrent programming in any JVM language.

no_wizard2• 4 years ago

Same reason I always felt extremely comfortable with concurrent programming in F#. The immutable first principle makes it easier to reason about what the program is doing during concurrent execution.

Of course all projects will benefit from Loom. I am merely positing that Clojure in particular could leverage this both most quickly and to very deep positive effect, due to the nature of the language itself putting immutability first, which if I recall correctly they built up a pattern of concurrent execution around already.

Kotlin too will be be able to optimize their concurrency story.

I simply wanted to posit an observation I hadn't seen elsewhere on the web about Project Loom yet.

eyelidlessness• 4 years ago

Clojure was explicitly designed to make reasoning about concurrency simpler (most data is immutable and can be referenced in multiple routines without concern; mutable reference types use the same data semantics, wrapped in transactions) with minimal performance impact (immutable data types are implemented as persistent data structures, sharing memory rather than naive strategies like copy-on-write). I don’t know that it’ll necessarily benefit more from Loom than other JVM languages, so much as it’s well positioned to benefit from concurrency improvements generally because its underlying design encourages code well suited for taking advantage of concurrency generally.

Edit to add: I hadn’t initially thought to speculate on the language itself taking advantage, but it occurred to me immediately after posting this comment. Another potential benefit is that many of its fundamental abstractions (most collection APIs are declarative) lend really well to making those operations concurrent, which is a common FP/lisp hypothetical but becomes more tangible if coroutines are less expensive. This isn’t dissimilar to how RDBMS query planners can provide wild performance improvements without any semantic changes, because SQL queries express “what” now “how”.

chrisseaton3• 4 years ago

Loom threads have the same basic semantics as normal threads. There's nothing special about them that relates to immutability.

lgessler• 4 years ago

I think GP's point is that immutable objects (which Clojure uses "by default") can be easily used in concurrent settings. Immutable objects can be shared without concern between threads, whereas mutable objects may need to be copied or have their operations synchronized somehow.

whalesalad• 4 years ago

I think a Clojure with first class legitimate actor semantics would be unreal. Clojure core will espouse async and tell you why actors are bad but man I love the actor model.

weatherlight• 4 years ago

https://www.clojerl.org

Clojure for the Erlang VM

klysm• 4 years ago

I’d think it would also make a huge difference for Akka.

SemanticStrengh• 4 years ago

Will loom get support for Blockhound? https://github.com/reactor/BlockHound Blockhound supports both reactive streams and Kotlin coroutines currently.

delusional• 4 years ago

Project loom is about deprecating "non-blocking" and "reactive" as a concept. The whole point is to just let the code block.

isbvhodnvemrwvn• 4 years ago

How does it replace the backpressure mechanism from reactive solutions?

rr808• 4 years ago

I spent the last few years really working hard to understand async frameworks/libraries like VertX, Mutiny etc I've gone from "I hate it" to "I can live with it". Now I dont have to. :) Yay.

qlm• 4 years ago

Any (relatively easy) way we can try virtual threads out yet?

vips7L4• 4 years ago

It should be available in the 19 EA builds:

https://jdk.java.net/19/

YawningAngel• 4 years ago

You can download JDKs from https://jdk.java.net/loom/ that support it

Using Java's Project Loom to build more reliable distributed systems

Related Stories