An alternative to fat JARs

http://product.hubspot.com/blog/the-fault-in-our-jars-why-we-stopped-building-uber-jars

by javinpaul • 10 years ago

73 73 10 years ago

product.hubspot.com

pcl4• 10 years ago

Cool! We've been doing something similar with public Maven repositories as the backing store.

Jonathan: if you're listening, have you guys looked at Maven as a source instead of an S3 upload? What percentage of your dependencies are internal vs. external? We get 10-20x space savings from public-Maven-only elision.

EDIT: we've been working on a similar technique for executable JARs as well, which shares the same general optimization numbers and approach. It's a bit trickier, since we want to preserve the same 'java -jar' syntax, so requires a bit of classloader tomfoolery, and corresponding classloader awareness in code.

LoneWolf2• 10 years ago

I really like your approach, is there any code you can share on that?

pcl4• 10 years ago

Getting it open-sourced has been a bit of a back-burner thing; I'll see if I can stoke the fires a bit.

jpollock2• 10 years ago

I would recommend against depending on public repositories (of any form) as a backing store for repeatable builds. We've repeatedly seen public repositories remove packages, disappear entirely or otherwise make third party projects a release depends on unavailable. If bringing up a production service requires the presence of Maven, it will suffer an outage.

Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.

niftich3• 10 years ago

You can run Nexus or Archiva or some other Maven artifact mirror within your network to get the best of both worlds -- maven builds and none of your stuff will break if central is down.

makmanalp• 10 years ago

The iffy thing here (if I understand you correctly) is creating a dependency in your deployments that requires the maven servers to be alive, which isn't an uncommon kind of thing, but just saying - larger companies usually run their own package manager mirrors.

HiJon898• 10 years ago

We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc.

We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.

pron2• 10 years ago

Capsule.io supports Maven as a source, and requires no classloader tomfoolery. It also supports native libraries, either embedded or in a Maven repo, java and native agents, and automatically applies JVM flags embedded in the manifest. It's got Maven, Gradle and Leiningen plugins to assist in capsule creation.

virmundi• 10 years ago

It's neat. But is it worth it? I thought inbound for S3 was free so the 150 Mb doesn't cost much. You're paying $0.25 a month to store up to a gig of fat jars so really nkthing. It's 20-ish seconds faster. Assuming a CI build, that doesn't matter. It is a more complex solution. This makes it more brittle. Which makes it more bad. Finally you're losing the single deployable. When something goes bad, you can't just download the jar. You have to fish through the maven dependencies.

swsieber• 10 years ago

I will respectfully disagree and say it's worth it in this case.

Here's the reasons I see that makes it worth it:

* It reduces the build time to less than 50%

* They are doing this for 1000+ java applications

* The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.

brianwawok• 10 years ago

So the real question is.. do they NEED 1000-2000 java applications?

Or could it be much more sanely organized with say 15 or 20 applications?

I know nothing about the codebase but it sounds like they may have nanoservices...

jacques_chester3• 10 years ago

It's neat. But is it worth it?

Saving traffic is good in general, insofar as it speeds deploys and allows higher container density.

But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect.

I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it.

Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.

HiJon898• 10 years ago

Agreed, we might not have done it if we had to resort to having the application download its own dependencies at startup. In our case, our Mesos scheduler, Singularity, handles S3 artifacts at deploy time for us. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. By the time the app starts up, all of its dependencies are guaranteed to be present (if an S3 download failed, the deploy would have failed) so it's totally transparent to the application.

HiJon898• 10 years ago

20-30 seconds ends up being 50% for most of our Java builds, so it was a huge speedup. It also saves 20-30 seconds at deploy time when we need to download this file again. And with lots of concurrent deploys we were occasionally seeing these large downloads saturate the NIC on our application servers.

Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.

When something goes wrong we rarely downloaded the JAR, but if you really wanted to you could download the thin JAR and use the SlimFast metadata bundled with it to download the exact dependencies it would get when deployed.

matt_wulfeck2• 10 years ago

It's hard for me to understand why in 2016 200MB files are still any kind of real problem.

pjc50• 10 years ago

From the article:

"Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."

matt_wulfeck2• 10 years ago

I wasn't being critical of the OPs build process. I agree that uploading 200 MB to S3 is slow, but it shouldn't be.

devsatish• 10 years ago

Back in the day , used to have all common dependency jars in server/lib folder and only deploy slim application war to the server . Reminds me of this

sytringy055• 10 years ago

and remember how bad it was trying to get application server X to deploy your war file from a command line without exploding for some bizarre reason? It still actually makes me sad to think of the hours of my life wasted writing WLST jython scripts trying to get WebLogic to behave properly.

HiJon898• 10 years ago

What was the process for adding or removing a dependency? Changing a version? If two apps needed different versions?

niftich3• 10 years ago

You added the dependency to server/lib. You standardized on versions, and applied only security fixes, testing them in a separate environment first. You only upgraded versions of libs if you had a critical feature you couldn't live without, of if your existing libs were about to be deprecated and out-of-support.

I'm not saying it wasn't a dark time, but it had its merits and drawbacks just like everything else.

_pmf_3• 10 years ago

Reinventing OSGi configurations.

mindcrime2• 10 years ago

We're using ServiceMix as a container for services (micro or otherwise) and while OSGI isn't without its own annoyances, it really is nice once you get everything working.

The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. Recently I specifically had an issue with Spring in this regard (yeah, yeah, don't use Spring, I know... but there was a specific reason in that case, at least initially) that would have been easy to resolve if the Spring guys still distributed OSGI'ified versions of their jars.

Still, all in all, I'm pretty happy with this approach.

_pmf_3• 10 years ago

The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information.

It's a mess, and I fear Java 9/10's modularization approach will even make it worse (so that instead of P2 (Eclipse's Equinox' OSGi bundles) and Maven, we'll have another set ... sigh. I'm still looking for a solution that transparently translates Maven metainformation to OSGi bundle information at some higher level (for example by hooking into the OSGi container's dependency resolution logic).

Edit: apparently, such a mechanism exists (ResolverHook)

sytringy055• 10 years ago

I've been working with Fuse for ages and I'm still not convinced OSGi is worth the trouble. ServiceMix, Felix, Karaf etc can be a total lottery some times as to whether or not it will do what you asked. Dropwizard or SpringBoot ftw!

chadmaughan• 10 years ago

First thought (to the article title "Why we stopped building fatjars") was that the build person had "graduated" and took their newly "acquired superpowers elsewhere" [0]

Joking aside, fantastic idea =)

Have you received a cease and desist from SlimFast yet for violating their trademark?

[0] http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyo...

gurubavan• 10 years ago

Don't believe everything you read -- especially when there's a single source.

sytringy055• 10 years ago

I did my first fat jar project in 2010 with embedded jersey and it was about 10mb. We use dropwizard now and most of our microservices sit at around 20mb which I think is still slim enough to avoid the extra overhead of managing the dependencies on deployment. IMHO Its hard to overstate how much benefit you get from java -jar

pron2• 10 years ago

You can have both. Capsule.io (I'm a developer) gives you the option of either embedding all dependencies in the JAR (including native libraries, if you need them), or just listing Maven dependencies and having them downloaded, cached, and shared on the first launch. It also lets you supply all the JVM flags directly in the JAR manifest.

sytringy055• 10 years ago

I'll have to check it out, thanks!

HiJon898• 10 years ago

We still use java -jar with this setup. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR with no runtime or ClassLoader funny business.

bradleybuda• 10 years ago

We have over 1,000 microservices constantly being built and deployed.

Wait until they find out how much faster in-process function calls are than RPCs.

heartsucker4• 10 years ago

Wait until they realized microservices can be deployed independently and have strict APIs without people using classes they shouldn't or creating spaghetti code because they have 800k LOC.

msbarnett• 10 years ago

without people using classes they shouldn't or creating spaghetti code because they have 800k LOC.

Honestly, if you can't trust your devs not to make a mess in a monolith, you can't trust them not to make a distributed mess with microservices.

Both architectural patterns have their uses. Neither will save you from bad developers.

aleyan• 10 years ago

Wait until they realized that solving their spaghetti problem with microservices resulted in them operating an Italian restaurant.

Pushing complexity onto interprocess/machine communication will just result in higher levels of spaghetti. Do you really think that same people who would construct a monolith poorly would do a better job building a distributed system instead?

I don't mean to be too dogmatic; there are legit use cases for services. I am just highly skeptical of microservices as a solution for complexity.

lmm• 10 years ago

You don't need microservices for that. This is Java, we have working dependency management and multi-module projects via maven, and the system can actually enforce things like separation of interface and implementation.

Retr0spectrum• 10 years ago

Faster hardware is usually cheaper than hiring more developers.

vidarh• 10 years ago

That changes very quickly if you create distributed systems that tend to either have underdimensioned external IO (networks) and/or be excessively expensive. It's not unusual for systems to balloon up to several times the original size the moment you have to cross the boundary to communicating over the network. Even if it's a LAN.

GrumpyYoungMan• 10 years ago

Faster hardware is usually cheaper than hiring more developers.

Now that Moore's law is running out of steam, we can reasonably expect that to become false within the next decade or so. The division between those who can merely write code and those who can properly engineer software systems will become all too apparent then.

_pmf_3• 10 years ago

Wait until they find out how much faster in-process function calls are than RPCs.

2017 will be the year of the return of the monolith: "How We Replaced 946 Microservices With a Statically Linked 10kB Rust Binary and How It Ulimately Did Not Save Us From Our Complete Lack of Anything Resembling a Coherent Business Model".

QuercusMax• 10 years ago

Nobody said that calls between microservices have to be done via RPCs.

I use a framework at Google that is microservice-based, and allows for composition of microservices into an assembly. Within an assembly, calls between services are done in-process; if you are calling a service in a different assembly, it has to make an RPC.

The mechanics of whether it's a local call or an RPC are all handled at the framework leve. This makes splitting a service into its own assembly a simple task (don't have to update any code, just configuration; this can be managed by SREs).

wcummings• 10 years ago

I use a framework at Google that is microservice-based, and allows for composition of microservices into an assembly. Within an assembly, calls between services are done in-process; if you are calling a service in a different assembly, it has to make an RPC.

Sounds a lot like OTP.

In Java-land, you could also just have your client and service use the same interface. Just inject different implementations for local and remote.

pacoverdi• 10 years ago

To deploy a process in my current project, I use Apache Ivy to resolve its dependencies (starting from a small set of top-level modules). Then I cache the results because Ivy resolution tends to take a while when you have a lot of dependencies and a lot of repositories.

Then I send a message to a remote agent with the command to run (classpath, main class, jvm args, args, configuration, etc). The agent then downloads the jars that are not already present in its cache with a valid SHA-1 checksum then starts.

Works quite well.

exelius• 10 years ago

Everyone uses fat JARs because it removes a lot of things that can go wrong operationally (especially when standing up new servers).

The approach described in this article is the way Java was designed to work; but then everyone said you needed a "self-executing" service binary for portability. Enter the fat JAR, containers, orchestration layers, etc.

JVM application servers are certainly the way to go if you run a bunch of Java. But then you're committing to maintaining a bunch of dependencies on your application servers, so really you're pushing the tech debt down the line to your ops folks who now have to ensure dependencies are synced, maven servers running and updated properly, proxies punched for new external dependencies, etc.

1138• 10 years ago

On limited information, this sounds like a work around on a root problem of runaway dependencies.

Maybe something like proguard could reduce deployable jar size to only the classes used.

I assume these are daily dev deploys and not production deploys.

HiJon898• 10 years ago

We definitely have some dependency cruft that could be trimmed (lots of relocated copies of Guava due to incompatibilities, for example).

We do frequent production deploys (of individual services, there's no such thing as deploying our entire application). To give an idea, it's a little before 1pm here and across our team there have been 180 production deploys already today.

setheron• 10 years ago

Here is something I mentioned on the reddit post:

You can go just as far with http://www.capsule.io/ and the capulet to import dependencies with Maven (from a Nexus proxy that also fronts S3)

You are now deploying thin Jars and having the artifacts cached on the local host through maven~!

mjt0229• 10 years ago

It doesn't use S3, but you can do something similar with Coursier:

https://github.com/alexarchambault/coursier#launch

twic4• 10 years ago

This sounds pretty sensible. I wonder if there's a simple way to reproduce it. Maybe:

  I. Build your app as a thin JAR, with:
    a. a Main-Class entry in the manifest
    b. Class-Path entries in the manifest
    c. a file containing a list of dependency coordinates (group, artifact, version) in META-INF
  II. Write a shell script (for whatever value of 'shell' you like) which:
    1. takes a Maven repo URL and a Maven coordinate (group, artifact, version)
    2. downloads the JAR from the repo, extracts its dependency list, then pulls its dependencies too
    3. (optionally) somehow records which JAR is the main one, say by writing a tiny shell script or a symlink pointing at it

You might be able to use a standard embedded POM instead of a dependency list, which might reduce the work a bit, but that would then require doing transitive dependency resolution at deploy time, which is probably a bad idea.

Phase I is something like 10 - 20 lines of Gradle, tops. You could pack it up into a plugin easily enough. Phase II is a similar amount of shell script, maybe more.

For extra safety, add SHA3 hashes to the dependency list file, and check them when you download the dependencies.

I've described this as using a Maven repo, but that doesn't mean it has to hit Nexus or whatever; you can just put JARs in S3 in the right layout. A while ago, i was doing this by maintaining a repository locally, and just pushing the whole thing to a Bitbucket website:

https://bitbucket.org/twic/twic.bitbucket.org/src/14ac48d4c4...

You could do something similar, perhaps going from Nexus to S3.

HiJon898• 10 years ago

This similar to our setup with the slimfast-plugin. We use the maven-jar-plugin to add the Main-Class and Class-Path entries to the manifest. At build time, we use the upload goal of the slimfast-plugin to upload the dependencies. It automatically reads the configuration of the maven-jar-plugin to make sure the paths it generates match the classpath in the manifest. Then it spits out a JSON file that has info on each dependency, including its location in S3, file size, checksum, and relative path where it needs to be copied to match the Class-Path in the manifest (the JSON entries look like this https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e...).

On deploy, the simplest way to get up and running is to use the download goal of the slimfast-plugin. It reads this JSON file, downloads each dependency (using an optional cache folder), verifies the file size and checksum, and copies it to the correct relative path. The application will then start up happily with java -jar.

At HubSpot, we instead integrated this download step more transparently into our deploy process. At build time we read this JSON file and store the dependency information in our build database. Our deploy infrastructure already accepts S3 URLs and handles downloading, caching, verifying checksums, and copying to arbitrary directories so we just piggybacked on this existing functionality to have it download the application plus all of its dependencies for us.

twic4• 10 years ago

Here's my very rough attempt in Gradle:

https://bitbucket.org/twic/ensure

Turned out to be more than 20 lines, but then i did it in Java rather than Groovy.

I do the downloads from the Maven repo (which should be your internal Maven repo!), so there's no need for an upload step. Deployment is done with a shell script with a few undemanding dependencies - unzip, curl, and openssl to verify the digests. I use a TSV file rather than JSON because it's easier for the shell script to read!

My next trick will be to figure out how to get this into a Cloud Foundry buildpack ...

Oh, the name: https://ensure.com/nutrition-products

cmcginty• 10 years ago

Sounds like an interesting idea. I would be even more excited to see someone leverage Gradle for this. Assuming you publish your custom jar artifacts from your build server, your "deployment artifact" could be a single Gradle file that acts a your JAR dependency definition and execution wrapper. For deployment, sync the Gradle file and execute the wrapper to launch the service with Gradle handling all dependency updates and caching.

An alternative to fat JARs

Related Stories