3 billion items in Java Map with 16 GB RAM

http://kotek.net/blog/3G_map

by qwerta • 14 years ago

105 37 14 years ago

kotek.net

dahlia1• 14 years ago

"Near the end insertion rate slowed down thanks to excesive GC activity"

Can someone explain why this happens?

kevingadd• 14 years ago

I expect that as memory usage increases, the resulting increase in GC pressure causes the GC to run more often. Most GCs that I know of are designed to have some memory free for the young generation (for fast, cheap allocations) and some memory free in the old generation (so objects can be moved from the young generation to the old generation). Maybe the JVM detects that it's approaching the limits of physical memory and starts GCing more often to avoid running out of space?

EDIT: From rereading the original post it seems more likely that it was approaching the heap size limit set when running the JVM and not a physical memory limit. I would expect the same principle applies, though.

kt9• 14 years ago

The Java GC uses a mark and sweep algorithm that has to stop the world to collect the garbage. As the number of items on the heap goes up, there are a higher number of valid pointers (less garbage) and the GC takes a long time to find even a little bit of garbage to clean up. So each GC run takes longer and there are more of them as the heap fills up. Given that the world is stopped during each GC run things take longer and hence the insertion rate is slower.

qwerta3• 14 years ago

Exactly. If there is too many pointers GC becomes very slow.

spartango4• 14 years ago

Java's garbage collector has advanced from pure stop-the-world mark-and-sweep collection over the years. In Java 7, a new generational garbage collector became the default, working concurrently with application threads to minimize GC pauses and collection time on multicore systems. If you're curious, check out these resources about how it works:

* http://www.drdobbs.com/jvm/g1-javas-garbage-first-garbage-co...

* http://www.oracle.com/technetwork/java/javase/tech/g1-intro-...

Note also that the JRE ships with multiple garbage collectors that you can select manually, and that some of the collectors tune themselves to match the application.

andrewvc2• 14 years ago

What? In java there are multiple GCs available: ConcMarkSwep, G1GC, ParallelGC, and I believe a couple more. All you have to do is set the right CLI flag.

From the concurrent mark and sweep docs: "The concurrent mark sweep collector, also known as the concurrent collector or CMS, is targeted at applications that are sensitive to garbage collection pauses. It performs most garbage collection activity concurrently, i.e., while the application threads are running, to keep garbage collection-induced pauses short. The key performance enhancements made to the CMS collector in JDK 6 are outlined below. See the documents referenced below for more detailed information on these changes, the CMS collector, and garbage collection in HotSpot."

It does have to stop the world sometimes but those times should be quite rare!

Tharkun• 14 years ago

With each consecutive insert in the collection, the available memory shrinks, the GC detects this, and attempts to free some of it. As long as more memory (the 15GB) can be allocated, this isn't much of a problem. However, at some point there is hardly any memory left: the 15GB is all but allocated, and very little memory can be freed with each run. So the GC runs ever more often, to the point that it starts to slow down the system.

Before you say "well that's stupid", realise that the alternative would throwing an OutOfMemoryError, in which case the JVM would terminate. A slow JVM is better than no JVM.

fcoury• 14 years ago

I disagree. A slow JVM can go on and on and the perception may be that the system is up. I think in this scenario a fail fast is much more interesting, since in a normal system, the fact that the JVM is down would trigger some manual intervention which seems like is needed anyways.

miahi• 14 years ago

The article doesn't say what the -Xms was. If you set it to less than Xmx, the VM will increase the memory size only when it's really necessary - it will issue a GC before trying to increase it. A GC on a heap of 100M objects takes a while.

Even when you reach the Xmx the VM will try to GC, and sometimes it can free a bit of memory so it doesn't throw an OOM, insert another 10 items and run the GC again, and so on - as long as you allocate small objects, it can take a while until you see the OOM.

cfieber• 14 years ago

It is kind of a dumb test to do 'new HashMap' with the default initialCapacity then cram stuff into it in a tight loop.

It is basically testing constant rehashes which is obviously going to generate a ton of garbage

I suspect the default collection implementations would perform significantly better if they were used correctly

zorked• 14 years ago

They still suck even if you set a large initialCapacity.

plant422• 14 years ago

Interesting, though I'm curious about what kind of results you get when something other than an empty String object is used?

qwerta3• 14 years ago

It uses more memory and result number of entries will be smaller. It is database engine, physical laws still applies.

plant422• 14 years ago

I was more interested in knowing whether you have any real world examples of MapDB?

_delirium2• 14 years ago

The MapDB mentioned in this article looks pretty interesting: https://github.com/jankotek/MapDB. Anyone have experience with it?

spartango4• 14 years ago

I've been using the previous version of the library (JDBM3) for some time now, although I haven't tried MapDB just yet. They have very similar APIs, however.

I've found JDBM3 a pleasure to use; it's fast and stable, with an excellent API. It's really quite easy to start using (it exposes Java collection interfaces), but you can configure it to do some powerful things under the hood. I'm not using the library in a really high-performance setting (just need a persistent key-value store), so I can't quite comment on the extreme scaling qualities.

I'm excited to switch to MapDB as it matures.

hncommenter13• 14 years ago

I've used JDBM3 to build a high-speed geocoder from US Census Tiger data. Because the data set is static and read-only, I use a minimal perfect hash function to map zip|street pairs to ids and then JDBM to map those ids to compressed strings of data. I have < 25M entries in JDBM, so not an envelope-pushing use case.

I'm on an old XP machine with very little memory and a slow disk, and I get quite good performance out of it, despite not being able to effectively use mapped or off-heap memory. I expect that moving to MapDB on JDK7 (with its concurrent collections and better memory allocation) will improve performance quite a bit.

thrownaway24242• 14 years ago

Can someone please explain how this is possible, when all the java collections have "int size()" and the max value of a java int is 2,147,483,647?

_delirium2• 14 years ago

Collections are allowed to grow larger than size() can handle. The documentation for size() says:

> If this collection contains more than Integer.MAX_VALUE elements, returns Integer.MAX_VALUE.

thrownaway24242• 14 years ago

I see, thanks for pointing it out. I always considered it a fundamental limitation of collections, because all or almost all of them are array-backed, and array indices in java are int, therefore the collections are generally limited to 2^31 items.

greenyoda• 14 years ago

Does anyone know which license MapDB is distributed under (GPL, BSD, etc.)? I couldn't find this information on the MapDB site.

biobot• 14 years ago

Apache Software License, Version 2.0

https://oss.sonatype.org/content/repositories/snapshots/org/...

clumsysmurf• 14 years ago

From Github:

"What you should know: MapDB relies on mapped memory heavily. NIO implementation in JDK6 seems to be failing randomly under heavy load. MapDB works best with JDK7"

The original JDBM2 apparently worked fine under Android; I wonder how the new implementation (JDBM4 renamed MapDB) fairs.

I do worry about Java as implemented on Android diverging from the official stuff in general. For example, Android does not have NIO.2, and Harmony NIO bugs would be different than OpenJDK NIO bugs.

lhaussknecht• 14 years ago

Does anyone know of a .net implementation?

3 billion items in Java Map with 16 GB RAM

Related Stories