JVM statistics cause garbage collection pauses (2015)

https://www.evanjones.ca/jvm-mmap-pause.html

by tosh • 2 years ago

79 40 2 years ago

ta988• 2 years ago

Is it still the case?

lbalazscs• 2 years ago

In 2015 there was no ZGC. Today ZGC (an optional garbage collector optimized for latency) guarantees that there will be no GC pauses longer than a millisecond.

survivedurcode2• 2 years ago

I would check your answer. These are pauses due to time spent writing to diagnostic outputs. These are not traditional collection pauses. This affects both jstat as well as writes of GC logs. (I.e. GC log writes will block the app just the same way)

hawk_• 2 years ago

ZGC doesn't remove safepoint requests on threads which is the root cause. "Guarantees" here are with very heavy quotes.

kanzenryu2• 2 years ago

Sadly in many cases no; it's not magic. This nirvana is restricted to cases where there is CPU bandwidth available (e.g. some cores idle) and plenty of free RAM. When either CPU or RAM are less plentiful... hello pauses my old friend.

esaym• 2 years ago

These modern garbage collectors are not simply free though. I got bored last year and went on a deep dive with GC params for Minecraft. For my needs I ended up with: -XX:+UseParallelGC -XX:MaxGCPauseMillis=300 -Xmx2G -Xms768M

When flying around in spectator mode, you'd see 3 to 4 processes using 100%. Changing to more modern collectors just added more load to the system. ZGC was the worst, with 16+ processes all using 100% cpu. With the ParallelGC, yes you'll get the occasional pause but at least my laptop is not burning hot fire.

hinkley3• 2 years ago

The cost of statistics gathering on a GC implementation that avoids ineffective GC activity is less affected by the cost of telemetry (no news is good news), but it is still affected.

ackfoobar3• 2 years ago

Probably yes.

https://bugs.openjdk.org/browse/JDK-8076103

Closed with "Won't Fix".

flykespice• 2 years ago

...With no reasoning at all?

ahoka• 2 years ago

Not with Linux 5.x AFAIK.

smrtinsert• 2 years ago

Is this account a submission bot of some sort?

throwaway04324• 2 years ago

The account seems to be connected to a real person, but it has a high number of submissions (over 350 submissions the past 30 days)

geodel• 2 years ago

Also spending a lot cause higher credit card bills.

cogman10• 2 years ago

in /tmp

Why is `/tmp` on disk and not a tmpfs mount?

sltkr2• 2 years ago

There is no law that says /tmp must be on tmpfs, and historically this wasn't done, because tmpfs is limited in size to some faction of the kernel's memory, while /tmp may be used to store much larger files.

For example, GNU sort can sort arbitrarily large input files, which is implemented by splitting the input into sorted chunks that are written to a temporary directory, /tmp by default. But this is based on the assumption that /tmp can store significantly larger files than fit in memory, otherwise the point is moot. So using tmpfs makes /tmp useless for this type of operation.

In the end, it's a trade-off between performance and disk space. I also prefer to mount /tmp on tmpfs for performance reasons, but you should not assume that this is the case on all systems.

aidenn02• 2 years ago

While I run /tmp on disk, I should point out that tmpfs is not limited to the size of RAM; contents of tmpfs can be swapped out just like any other memory allocation.

aidenn02• 2 years ago

Why would I want it on tmpfs? Only advantage I see is slightly improved boot times (/tmp is typically cleared on boot, which is obviously not necessary for tmpfs).

hinkley3• 2 years ago

Slightly simpler handling for docker containers. Particularly if you run multiple copies of the same image on one box (blue-green deploys, process-per-cpu programming languages, etc)

TacticalCoder• 2 years ago

Why would I want it on tmpfs?

It's now there in several distros by default. Not that it answers your question.

sltkr2• 2 years ago

The pauses occur even [..] if you call mlock

I wonder how this is even possible. The only scenario I can think of involves a page fault on the page table itself (i.e., the page is locked into memory, but a page fault occurs during virtual-to-physical address translation). Does anyone know the real reason?

survivedurcode2• 2 years ago

Probably because pages mapped, even if they are locked into memory are not allowed to stay dirty forever. Does this help? https://stackoverflow.com/a/11024388 (In contrast, if you mlocked but never wrote to the pages, you probably would not encounter read pauses)

pjmlp2• 2 years ago

For proper statistics use Visual VM or Flight Recorder, if using an OpenJDK derived JVM implementation.

Also note that not all JVMs are made alike, and there are plenty to chose from.

hashmash• 2 years ago

When using the `-XX:+PerfDisableSharedMem` workaround, VisualVM cannot attach to the running process anymore.

jakewins• 2 years ago

Man I remember being bit by this in migrating to AWS - this had like snuck through on fast on-prem disks, but as soon as that /tmp was on RDS oh boy, it was a dozy.

hinkley3• 2 years ago

Stuff like this is why back when I still wrote Java we only wanted to turn on JVM telemetry on production boxes if they were canaries. Slower you can work around by deploying more copies. But jitter is not something you can do much about.

opentokix• 2 years ago

Using ebpf, perf and flamegraphs would let him find this in a couple of hours. That was not available for him in 2015 tho.

JVM statistics cause garbage collection pauses (2015)

Related Stories