Virtual Threads: New Foundations for High-Scale Java Applications

https://www.infoq.com/articles/java-virtual-threads/

by axelfontaine • 4 years ago

213 174 4 years ago

samsquire5• 4 years ago

This is good.

I implemented a userspace 1:M:N timeslicing thread, kernel thread to lightweight thread multiplexer in Java, Rust and C.

I preempt hot for and while loops by setting the looping variable to the limit from the kernel multiplexing thread.

It means threads cannot have resource starvation.

https://github.com/samsquire/preemptible-thread

The design is simple. But having native support as in Loom is really useful.

mattgreenrocks2• 4 years ago

I like it! Do you have any sense for what the perf hit is for making those loops less hot to enable pre-emption?

samsquire5• 4 years ago

There is no if statement in the hot loop or in the kernel thread so there is no performance cost there.

The multiplexing thread is separate to the kernel thread so you could say it's 1:M:N thread scheduling. I should have been clearer in my comment. There is 3 types of threads.

The multiplexing thread timeslices the preemption of the lightweight threads and kernel threads every 10 milliseconds. That is it stops all the loops in the lightweight thread and in the lightweight thread and causes the next lightweight thread to execute.

So there is no overhead except for a structure variable retrieval in the loop body

Rather than

For(int I = 0; I < 1000000; I++) {

}

We have

Register_loop(thread_loops, 0, 0, 1000000);

For(; thread_loops[0].index < thread_loops[0].limit; thread_loops[0].index++) {

}

Handle_virtual_interrupt()

And in the thread multiplexer scheduler, we do this

thread_loops[0].index = thread_loops[0].limit

ccooffee• 4 years ago

This is a great writeup, and reignites my interest in Java. (I've long considered "Java Concurrency in Practice" to be the _best_ Java book ever written.)

I haven't been able to figure out how the "unmount" of a virtual thread works. As stated in this article:

Nearly all blocking points in the JDK have been adapted so that when encountering a blocking operation on a virtual thread, the virtual thread is unmounted from its carrier instead of blocking.

How would I implement this logic in my own libraries? The underlying JEP 425[0] doesn't seem to list any explicit APIs for that, but it does give other details not in the OP writeup.

[0] https://openjdk.org/jeps/425

geodel2• 4 years ago

Well one way is to replace "synchronized" blocks with ReentrantLocks where ever you can.

cvoss• 4 years ago

I would guess that LockSupport.park() and friends have also been adapted to support virtual thread unmounting.

ecshafer• 4 years ago

Java Concurrency in Practice is a fantastic book. I had DL as a professor for about a half dozen courses in undergrad, including Concurrent and Parallel Programming. Absolutely fantastic professor, with a lot of insight into how parallel programming really works at the language level. One of the best courses I've taken.

anonymousDan4• 4 years ago

Yeah Java gets a lot of grief, but I learned a lot about concurrent programming from making sure I really understood every line of code in this book.

vips7L• 4 years ago

I’m honestly so envious you had him as a professor.

Nzen• 4 years ago

I don't know how they did it, but you could use that jep id as a query in the jdk issue tracker [0], and then use the issue tracker id to find the corresponding github issue [1]. (I had hoped for commits with that prefix, but there don't seem to be any for that issue.)

[0] https://bugs.openjdk.org/browse/JDK-8277131?jql=issuetype%20...

[1] https://github.com/openjdk/jdk/pull/8787

chrisseaton• 4 years ago

I haven't been able to figure out how the "unmount" of a virtual thread works.

The native stack is just memory like any other, pointed to by the stack pointer. You can unmount one stack and mount another by changing the stack pointer. You can also do it by copying the stack out to a backing store, and copying the new thread's stack back in. I think the JVM does the latter, but not an expert.

galaxyLogic4• 4 years ago

Seems like a good development. I've been doing Node.js for last few years after letting go of Java. But there's something uneasy about async/await. For one thing it's difficult to debug how the async functions interact.

lolive• 4 years ago

Debugging asynchronicity is complex in any language, no? Blocking endpoints are mostly irrelevent because it alters the temporal flow of things, which makes your code executed in debug mode not 100% « isomorphic » with your code executed in run mode.

cypressious• 4 years ago

Does you library use any of the JDK's blocking APIs like Thread.sleep, Socket or FileInputStream directly or transitively? If so, it is already compatible. The only thing you should check is if you're using monitors for synchronization which are currently causing the carrier thread to get pinned. The recommendation is to use locks instead.

woooooo• 4 years ago

Monitors as in the synchronized keyword? Because that is kind of a big one.

pron16• 4 years ago

How would I implement this logic in my own libraries?

There's no need to if your code is in Java. We had to change low-level I/O in the JDK because it drops down to native.

That's not to say every Java library is virtual-thread-friendly. For one, there's the issue of pinning (see the JEP) that might require small changes (right now the problem is most common in JDBC drivers, but they're already working on addressing it). The bigger issue, mostly in low-level frameworks, is implicit assumptions about a small number of shared threads, whereas virtual threads are plentiful and are never pooled, so they're never shared. An example of such an issue is in Netty, where they allocate very large native buffers and cache them in ThreadLocals, which assumes that the number of threads is low, and that they're reused by lots of tasks.

_benedict3• 4 years ago

Conversely, some applications would like a leaky abstraction they have some control over. Some caching will likely remain beneficial to link to a carrier thread.

As a member of the Cassandra community I’m super excited to get my hands on virtual threads come the next LTS (and Cassandra’s upgrade cycle), as it will permit us to solve many outstanding problems much more cheaply.

I hope by then we’ll also have facilities for controlling the scheduling of virtual threads on carrier threads. I would rather not wait another LTS cycle to be able to make proper use of them.

fibersfan• 4 years ago

An example of such an issue is in Netty, where they allocate very large native buffers and cache them in ThreadLocals, which assumes that the number of threads is low, and that they're reused by lots of tasks.

Fixing Netty is very high yield. Every modern Java server application I'm aware of uses Netty. Quarkus, Vertx, Micronaut, Java-GRPC, ...

Then Graal? Virtual threads in Graal with a Netty that isn't 60MB would be superb.

What about just shimming Netty? Is that in Oracle's scope? There are already selectable backends for Netty. Why not have "virtualthread-graalcompatible" that uses your already fixed Java IO? It would reduce so much pain, and make Java competitive with golang for the first time ever.

rlawson• 4 years ago

Agree - Java Concurrency in Practice was a revelation when it came out. Well the whole concurrent api by Dr. Lea made it all so much saner. Very excited for virtual threads!

geodel2• 4 years ago

I think it is really important development in Java space. One reason I plan to use it soon is because it does not bring in complex programing model of "reactive world" and hence dependency on tons of reactive libraries.

I tried moving plain old Tomcat based service to scalable netty based reactive stack but it turned out to be too much work and an alien programing model. With Loom/Virtual thread, the only thing I will be looking for server supporting Virtual threads natively. Helidon Nima would fit the bill here as all other frameworks/app servers have so far just slapping virtual threads on their thread pool based system. And unsurprisingly it is not leading to great perf expected from Virtual thread based system.

RcouF1uZ4gsC• 4 years ago

How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code?

Windows had has Fibers[0] for decades (IIRC since 1996 with Windows NT 4.0)

0. https://learn.microsoft.com/en-us/windows/win32/procthread/f...

gpderetta5• 4 years ago

Fibers are just one part of it. You also need a scheduler and a way to hook into existing blocking APIs.

jeffbee2• 4 years ago

" Operating systems typically allocate thread stacks as monolithic blocks of memory at thread creation time that cannot be resized later. This means that threads carry with them megabyte-scale chunks of memory to manage the native and Java call stacks."

This extremely common misconception is not true of Linux or Windows. Both Windows and Linux have demand-paged thread stacks whose real size ("committed memory" in Windows) is minimal initially and grows when needed.

uluyol• 4 years ago

Do they shrink too? How many threads can be created before address space is exhausted (even if the memory isn't backed by pages, the address space is still reserved)?

tedunangst2• 4 years ago

The stack for any thread other than the first is just memory like any other allocation. You can free it, resize it, copy it elsewhere, whatever you want to do. Literally just a pointer in a register. People work up weird mythologies about it, but the stack can be anything you want if you're willing to write code to manage it.

jeffbee2• 4 years ago

You'll run out of physical memory for the first page of the stack long before you run out of room in the virtual address space.

mikece• 4 years ago

How does this compare to Processes in Elixir/Erlang -- is Java now as lightweight and performant?

stefs2• 4 years ago

according to the article:

Virtual threads have more in common with the user-mode threads found in other languages, such as goroutines in Go or processes in Erlang -- but have the advantage of being semantically identical to the threads we already have.

anonymousDan4• 4 years ago

And the disadvantages of global shared memory, stop the world pauses etc. For sure it is a step in the right direction but it depends on where you stand wrt having shared mutable state in your programming model. It's also not clear to me whether virtual threads can lock up carrier threads (e.g. due to an infinite loop) or are somehow preemptible (c.f. erlang reduction counts).

thom• 4 years ago

So right now it seems like you can replace the thread pool Clojure uses for futures etc with virtual threads and go ham. You could even write an alternative go macro to replace the bits of core.async where you’re not supposed to block. Feels like Clojure could be poised to benefit the most here, and what a delight it is to have such a language on a modern runtime that still gets shiny new features!

shaunxcode• 4 years ago

Yep core async will work as is just use <!! and friends.

KingMob• 4 years ago

You wouldn’t need the go macro for anything with vthreads. If you continued to use core.async, you’d switch to the blocking variants, and ignore go entirely.

mgraczyk14• 4 years ago

The section "What about async/await?", which compares these virtual threads to async/await is very weak. After reading this article, I came away with the impression that this is a dramatically worse way to solve this problem than async/await. The only benefit I see is that this will be simpler to use for the (increasingly rare) programmers who are not used to async programming.

The first objection in the article is that with async/await you to may forget to use an async operation and could instead use a synchronous operation. This is not a real problem. Languages like JavaScript do not have any synchronous operations so you can't use them by mistake. Languages like python and C# solve this with simple lint rules that tell you if you make this mistake.

The second objection is that you have to reimplement all library functions to support await. This is a bad objection because you also have to do this for virtual threads. Based on how long it took to add virtual threada to Java vs adding async/await to other languages, it seems like virtual threads were much more complicated to implement.

The programming model here sounds analogous to using gevent with python vs python async/await. My opinion is that the gevent approach will die out completely as async/await becomes better supported and programmers become more familiar.

EDIT: Looking more at the "Related Work" section at the bottom. I think I understand the problem here. The "Structured Concurrency" examples are unergonomical versions of async/await. I'm not sure what I'm missing but this seems like a strictly worse way to write structured concurrent code.

Java example:

    Response handle() throws ExecutionException, InterruptedException {
        try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
            Future<String>  user  = scope.fork(() -> findUser());
            Future<Integer> order = scope.fork(() -> fetchOrder());

            scope.join();           // Join both forks
            scope.throwIfFailed();  // ... and propagate errors

            // Here, both forks have succeeded, so compose their results
            return new Response(user.resultNow(), order.resultNow());
        }
    }

Python equivalent

    async def handle() -> Response:
      # scope is implicit, throwing on failure is implicit.
      user, order = await asyncio.gather(findUser(), findOrder())

      return Response(user, order)

You could probably implement a similar abstraction in Java, but you would need to pass around and manage the the scope object, which seems cumbersome.

wtetzner3• 4 years ago

I can see you having objections to their arguments against async/await, but what makes you say async/await is somehow the better solution?

mgraczyk14• 4 years ago

There are a few reasons.

async/await allows you to do multiple things in parallel. I don't see how you can do that in the virtual threading model, although I haven't used it and only read this article. You would have to spin up threads and wait for them to finish, which IMO is much more complicated and hard to read.

javascript

    async function doTwoThings() {
      await Promise.all([
        doThingOne(),
        doThingTwo(),
      ]);
    }

python

    async def do_two_things() {
      await asyncio.gather(
        do_thing_one(),
        do_thing_two(),
      );
    }

Another issue is building abstractions on top of this. For example how do you implement "debounce" using virtual threads? You end up unnaturally reimplementing async/await anyway.

Finally it's generally much easier implement new libraries with a promise/future based async/await system than with a system based on threads, but I'm not familiar enough with Java to know whether this is actually a good objection. It's possible they make it really easy.

pron16• 4 years ago

async/await require yet another world that's parallel to the "thread" world but requires its own "colour" and set of APIs. So now you have two kinds of threads, to kinds of respective APIs, and two kinds of the same concept that has to be known by all of your tools (debuggers, profilers, stacktraces).

This is a bad objection because you also have to do this for virtual threads

No. We had to change a bit of the implementation -- at the very bottom -- but none of the APIs, as there is no viral async colour that requires doubling all the APIs.

You're right that implementing user-mode threads is much more work than async/await, which could be done in the frontend compiler if you don't care about tool support (although we very much do), but the result dominates async/await in languages that already have threads (there are different considerations in JS) as you keep all your APIs and don't need a duplicate set, and a lot of existing code tools just work (with relatively easy changes to accommodate for a very high number of threads).

The "Structured Concurrency" examples are unergonomical versions of async/await.

They're very similar, actually.

We've made the Java example very explicit, but that code would normally be written as:

    Response handle() throws ExecutionException, InterruptedException {
        try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
            var user  = scope.fork(() -> findUser());
            var order = scope.fork(() -> fetchOrder());

            scope.join().throwIfFailed();
            return new Response(user.resultNow(), order.resultNow());
        }
    }

But when the operations are homogenous, i.e. all of the same type rather than different types as in the example above (Java is typed), you'll do something like:

    try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
        var fs  = myTasks.stream().map(scope::fork).toList();
        scope.join().throwIfFailed();
        return fs.map(Future:resultNow).toList();
    }

Of course, you can wrap this in a higher level `gather` operation, but we wanted to supply the basic building blocks in the JDK. You're comparing a high-level library to built-in JDK primitives.

Work is underway to simplify the simple cases further so that you can just use the stream API without an explicit scope.

mgraczyk14• 4 years ago

This makes sense, especially the bit about tooling. I'm unfamiliar with the state of Java tooling besides very simple tasks.

On the other hand using things like debuggers and reading stack traces in python/js "just work" for me. Maybe because the tooling and the language have evolved together over a longer period of time.

I also feel like the reimplementation of all functions to support async is not a big deal because the actual pattern is generally very simple. You can start by awaiting every async function at the call site. New libraries can be async only.

Eduard• 4 years ago

Languages like JavaScript do not have any synchronous operations so you can't use them by mistake.

Can you explain what you mean by this? Isn't it the opposite - Javascript has a synchronous execution model?

mgraczyk14• 4 years ago

I mean what the article calls a "synchronous blocking method", which javascript (mostly) does not have.

vlovich123• 4 years ago

Aside from NodeJS-specific APIs, JS as a whole does not generally have any synchronous I/O, locks, threads etc. SharedArrayBuffer is probably the notable exception as it can be used to build synchronous APIs that implement that functionality if I’m not mistaken.

Unless by synchronous you meant single threaded in which case JS is indeed single threaded normally (unless you’re using things like Web Workers).

gigatexal3• 4 years ago

Reading through the source code examples has me rethinking my dislike for Java. It sure seems far less verbose and kinda nice actually.

marginalia_nu3• 4 years ago

Modern Java is a lot less boilerplaty than old enterprise Java.

dboreham3• 4 years ago

Modern Java people are a lot less boilerplaty than old enterprise Java people.

kaba05• 4 years ago

Definitely reconsider it, while Java is still often used in an “enterprise” style, modern Java brought in plenty of advancements and it is a quite expressive, while still readable language.

Which is kind of strange, I never understood people being so hyped about Go, while straight up refusing to touch Java for “verbosity”, when in fact the latter is much more terse. We sure are not too rational when it comes to tooling :D

gigatexal3• 4 years ago

I only shudder really learning it that if i got a job doing java i'd be doing horrible java < 8 codebase work

Blackthorn• 4 years ago

So happy this is finally coming out! After years of using the library that inspired this (fibers), I'm so stoked this is coming to the wide outside world of Java. There's just no comparison in how understandable and easy to program and debug this is compared to callback and event based programming.

smasher1649• 4 years ago

After all the hoopla surrounding concurrency models, it seems that languages are conceding that green threads are more ergonomic to work with. Go and Java have it, and now .NET is even experimenting with it.

How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code?

Jtsummers5• 4 years ago

Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code?

Green threads are, definitionally, not OS threads, they are user space threads. So you will never see OS-native green threads as it's an oxymoron. The way many green thread systems work is to either lie to you (you really only have one OS thread, the green threads exist to write concurrent code, which can be much simpler, but not parallel code using Pike's distinction), or to introduce multiple OS threads ("carrier threads" in the terms of this article) which green threads are distributed across (this is what Java is doing here, Go has done for a long time, BEAM languages for a long time, and many others).

EDIT:

To extend this, many people think of "green threads" as lightweight threading mechanisms. That's kind of accurate for many systems, but not always true. If that's the sense that's meant, then OS-native lightweight threads are certainly possible in the future. But there's probably not much reason to add them when user space lightweight concurrency mechanisms already exist, and there's no consensus on which ones are "best" (by whatever metric).

Sakos• 4 years ago

To extend this, many people think of "green threads" as lightweight threading mechanisms. That's kind of accurate for many systems, but not always true. If that's the sense that's meant, then OS-native lightweight threads are certainly possible in the future. But there's probably not much reason to add them when user space lightweight concurrency mechanisms already exist, and there's no consensus on which ones are "best" (by whatever metric).

Wouldn't it make sense to implement them kernel-side when looking at how every programming language seems to have to reinvent the wheel regarding green threads?

smasher1649• 4 years ago

If that's the sense that's meant

Yeah that's what I meant, a lightweight threading mechanism provided by the OS.

there's probably not much reason to add them when user space lightweight concurrency mechanisms already exist

Yeah... I don't think there's consensus on that. It seems that many people find OS threads to be an understandable concurrency model, but find them too heavyweight. So the languages end up introducing other abstractions at either the type-level (which has other benefits mind you!) or runtime to compensate.

gpderetta5• 4 years ago

Kernel support for green threads (or better, M:N) was a big topic 20 years ago. Look for "scheduler activation" for example.

It kind of petered off as kernel only threads were fast enough with significantly less complexity. In might be worth revisiting again.

zozbot234• 4 years ago

How long until OS vendors introduce abstractions to make this easier?

The OS-level abstraction is called M:N threads. It has always been supported by Java on Solaris. But it's not really popular elsewhere.

MarkSweep• 4 years ago

Maybe ironically given all the green thread experiments in industry, but Microsoft just removed their user-mode scheduling library in Windows 11.

https://learn.microsoft.com/en-us/windows/win32/procthread/u...

lmm3• 4 years ago

After all the hoopla surrounding concurrency models, it seems that languages are conceding that green threads are more ergonomic to work with. Go and Java have it, and now .NET is even experimenting with it.

Green threads were always better, the issue was compatibility with native (C) libraries. 20 years ago a language whose C FFI was as cumbersome as Go's would be laughed at.

How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code?

Most of the point of green threads is to avoid the OS getting in the way. Rather than OSes adding green threading, the future looks like sidestepping more and more of the traditional OS, ending up with unikernels or something that looks like them. The likes of graal native image are already a step in this direction.

smasher1649• 4 years ago

Green threads were always better

There’s a lot of people who advocate otherwise, but I don’t think we have the mechanisms for effect polymorphism in current languages good enough to make effects ergonomic.

Most of the point of green threads is to avoid the OS getting in the way.

I see the OS as an environment to help me run programs. Language runtimes have been taking over that role over the last few decades, sure. However, whether or not unikernels are the way forward remains to be seen. I still see it worthwhile to improve the OS, however.

halayli• 4 years ago

bsd variants and sun os used to have n:m APIs but they got deprecated and removed long time ago. It's hard to provide a consistent behavior that works for everyone.

smasher1649• 4 years ago

I said in a sibling comment that scheduler activations may have been a flawed idea, but I don’t think the space of user-space scheduling APIs is fully explored. Google’s ghOSt work is an example of that. If io-uring is proof of anything, it’s that there’s still fundamental changes we can make in how we schedule work with the kernel.

dboreham3• 4 years ago

There are : Fibers in NT for example.

lenkite• 4 years ago

Really hope this makes it to Android. (probably need to wait for a decade or two though)

anonymousDan4• 4 years ago

Copying virtual stacks on a context switch sounds kind of expensive. Any performance numbers available? Maybe for very deep stacks there are optimizations whereby you only copy in deeper frames lazily under the assumption they won't be used yet? Also, what is the story with preemption - if a virtual thread spins in an infinite loop, will it effectively hog the carrier thread or can it be descheduled? Finally, I would be really interested to see the impact on debugability. I did some related work where we were trying to get the JVM to run on top of a library operating system and a libc that contained a user level threading library. Debugging anything concurrency related became a complete nightmare since all the gdb tooling only really understood the underlying carrier threads.

Having said all that, this sounds super cool and I think is 100% the way to go for Java. Would be interesting to revisit the implementation of something like Akka in light of this.

aardvark1792• 4 years ago

Yes, lazy copying tricks are employed, and some work round stack frames delayed until the stack is moved by the GC on the assumption most will not live that long.

There was a lot of work done on debugging so standard Java debuggers work well.

anonymousDan4• 4 years ago

That's pretty cool! What about the blocking issue? Presumably also if you are using JNI all bets are off?

hawk_• 4 years ago

Stacks in idiomatic Java aren't usually that deep (pointers instead of in place values) so this isn't that big of a deal, unlike say in C++.

grishka• 4 years ago

Also, what is the story with preemption - if a virtual thread spins in an infinite loop, will it effectively hog the carrier thread or can it be descheduled?

As I understood it, there's no preemption. You're supposed to not do busy waiting on virtual threads (and better not do it at all, use wait-notify or a barrier or whatever). Virtual threads are for I/O-bound tasks. For CPU-bound tasks you'd want an OS thread per CPU core anyway.

mike_hearn3• 4 years ago

They did implement pre-emption at some point and there are parts of that still in the code. It's intended to let you more easily manage load e.g. by de-scheduling complicated batch jobs that are low priority and then re-scheduling when the service is under lighter load. But it won't ship as public API as a part of Loom, maybe it will never ship (generally once OpenJDK finishes a project it gets de-staffed and they don't go back to it).

jsyolo2• 4 years ago

Keeping familiarity aside, why would one use Java/JVM instead of nodejs for the server of a web app? I need to call SOAP services.

anonymoushn3• 4 years ago

What are the benefits of using nodejs over Java?

jsyolo2• 4 years ago

SSR React, using the same language in the server as in the browser and not wrangling between to completely different ecosystems.

pjmlp5• 4 years ago

When one cares about performance.

kaba05• 4 years ago

And maintainability. And quality of libraries.

stefs2• 4 years ago

for the record, i really don't know much about threads, so the following questions are probably kinda stupid.

first question: so, as the article states, the ONLY performance upside of virtual threads (versus os threads) is the number of inactive threads, thanks due to lower per-thread memory overhead.

for some reason i was expecting to read something about context switching cost too.

as far as i understand, virtual thread context switches are most likely between a lot cheaper and roughly as expensive than their carrier thread context switches, depending on how much memory has to be copied around and how to find the next thread to execute.

the problem here is that virtual context switches may be cheaper, but have to be executed in addition to the os thread context switches, so the overall efficiency is actually lower because more work is spent scheduling (os vs. os+virtual).

to minimize this it might be possible for privileged applications to disable os thread context switching for the carrier threads as long as there are active virtual threads. that way, the context switching and scheduling overhead is reduced from "os vs. os+virt" to "os vs. virt". i.e. as soon as there are active virtual threads the carrier thread is excluded for os scheduler until there aren't any active virtual threads anymore (or, alternatively, the virtual thread pool is empty).

is this a thing? does this make sense? would it be worth it? do operating systems even support "manual" (i.e. by the app) thread scheduling hints? or are the carrier threads only rarely taken out of schedule because they're not really put to sleep as long as there are active virtual threads anyway, making this a non-issue?

second question: as far as i understand blocking os threads, the scheduler stores which thread is waiting on which io resource and the appropriate thread gets woken up once a waited-on io resouce is available. this is not much of a problem with with a few hundred or thousand os threads, but now with virtual threads, the io resource must now be linked to the os thread for the virtual thread executor's scheduler by the os and then to the virtual thread waiting on the resource by the virtual thread scheduler. so for example if there are 100.000 inactive virtual threads waiting for a network response and one arrives, the os scheduler has to match it to an os thread first (the one the vt scheduler runs on) and then the vt scheduler has to match it to one of the virtual threads. i.e. two lookups in hashtables with 100.000 entries each (one io to os threads, the other io to vt). is this how it works or do i misunderstand this? as async models have the same issue but work fine i guess this isn't really a problem in practice. also, as far as i understand, the os thread woken up is given a kind of resouce id it's been woken up for, instead of "well, you went to sleep for a certain resource id so it's obvious which one you've been woken up for" in blocking IO).

gpderetta5• 4 years ago

You do not have to context switch the kernel thread when context switching the multiplexed user thread. So the context switch should in principle be much faster. There are second order effects of course, for example the new user thread might touch cold cache lines so the context switch speed up might not make much of a difference.

Normally on an M:N setup the kernel threads are pinned one per phisical hardware thread (i.e. core or SMT thread), so as long a they are the only program running in that core, they are never preempted.

lmm3• 4 years ago

to minimize this it might be possible for privileged applications to disable os thread context switching for the carrier threads as long as there are active virtual threads. that way, the context switching and scheduling overhead is reduced from "os vs. os+virt" to "os vs. virt". i.e. as soon as there are active virtual threads the carrier thread is excluded for os scheduler until there aren't any active virtual threads anymore (or, alternatively, the virtual thread pool is empty).
is this a thing? does this make sense? would it be worth it? do operating systems even support "manual" (i.e. by the app) thread scheduling hints? or are the carrier threads only rarely taken out of schedule because they're not really put to sleep as long as there are active virtual threads anyway, making this a non-issue?

It's normally recommended to run with as many "physical" threads as you have CPU cores, and then the OS scheduler can generally just do the right thing (assuming nothing else is running on the machine) - you don't need to do any context switches if you have as many processors as there are OS threads wanting to run. Most OSes do offer a way to "pin" a thread to a processor (at varying levels of hint/requirement) but I've only seen them used when doing fairly extreme performance tuning.

rr808• 4 years ago

I spent the last 5 years learning reactive programming. I hate it. I'm looking forward to going back to something solid.

s-macke• 4 years ago

Java Reactive works around the shortcomings of the platform by basically inventing its own language inside Java. A high cost for the developer.

Under the hood, the virtual thread feature does more or less the same as reactive. Async calls via epoll, a scheduler in user space, .....

bheadmaster4• 4 years ago

It's funny how it took Java almost a decade to finally implement goroutines.

kaba05• 4 years ago

As opposed to what platform?

Also, they did that in a completely backwards compatible way, so that programs that were written before Go was even a thing will also benefit, so if you will, Java has out of the doors orders of more “goroutine”-aware code as is, than Go will likely ever will.

bheadmaster4• 4 years ago

As opposed to what platform?

Go.

programs that were written before Go was even a thing will also benefit

As far as I understand virtual threads, they have a new API for creating them, so older programs will at the very least need some source change and a recompilation in order to use them. Of course, if there's a JVM flag to automatically use virtual threads in place of OS threads, please correct me :)

Java has out of the doors orders of more “goroutine”-aware code as is, than Go will likely ever will.

I honestly doubt that older code is "goroutine"-aware, as virtual threads are used in a different way than ordinary OS threads. I don't think there's any code that spawned hundreds of thousands of OS threads before, which would be the main benefit of using virtual threads.

pjmlp5• 4 years ago

You mean the co-routines that Modula-2 already had in 1978, or Erlang in 1986, not such big achievement on Go's part.

By the way, JVM green threads were available in 1996, however eventually red threads became the default implementation across all major JVM vendors.

bheadmaster4• 4 years ago

You mean the co-routines that Modula-2 already had in 1978, or Erlang in 1986, not such big achievement on Go's part.

What are you implying, that Go should've implemented goroutines before Go even existed?

polskibus• 4 years ago

Is this like async/await machinery in .NET?

sebazzz• 4 years ago

Yes but transparent

Virtual Threads: New Foundations for High-Scale Java Applications

Related Stories