Removal of Unsafe in Java 9 – A disaster in the making

http://blog.dripstat.com/removal-of-sun-misc-unsafe-a-disaster-in-the-making/

by pdeva1 • 11 years ago

92 133 11 years ago

blog.dripstat.com

ante_annum2• 11 years ago

So the explanation is:

  Let me be blunt -- sun.misc.Unsafe must die in a fire.
  It is -- wait for it -- Unsafe. It must go. Ignore any kind of theoretical rope and 
  start the path to righteousness

And what the author takes away is:

  This engineer hates the Unsafe class for no reason at all.

The reason the engineer hates it is that it's unsafe. Disagree with it, sure. But it's a reason, and it's a good reason (imho). I understand that it's commonly used by apps I use, but it's hard to keep reading after this kind of knee-jerk.

snuxoll5• 11 years ago

In addition, it's in the sun.misc package, if it's not in java.* or javax.* it is ripe for pruning, you'd think people would have learned by now to stop using internal classes since pretty much every JRE/JDK release known to mankind has changed them or removed them.

ante_annum2• 11 years ago

It's sort of amazing that the diatribe here is against Oracle for wanting to remove an undocumented internal API because of typesafety, and not against the open source projects that use a specifically unsafe, undocumented internal API. I mean, nothing against these projects -- I understand that type safety can carry a performance penalty -- but the anger of this article is amazing.

kasey_junk7• 11 years ago

The problem with that, is unlike most of the other internal classes, there is zero other way to do the things people are doing with unsafe without it.

It's not a matter of expedience, it's a matter of necessity.

taeric4• 11 years ago

That is a pretty crappy explanation, on the engineer's side.

To be clear, any explanation that is not immediately followed with examples to how current valid uses of Unsafe can be accomplished without it, are simply witch hunts by an engineer that hasn't had to meet the same requirements.

danudey• 11 years ago

It's a) unsafe, and b) an internal, undocumented API.

When you start importing from sun.misc.* you've made the conscious (or ignorant) decision to write non-portable code which may well break due to any JVM release. It sucks that these people are going to have to update their code (or that end users won't be able to upgrade to Java 9 without adding special command-line flags), but it's not due to fickle Oracle developers, it's due to technical debt that's finally catching up to them.

jevinskie• 11 years ago

I think the author's point about being able to make safe Unsafe replacements and keeping Unsafe around (for the time being) to enable better compatibility was thoughtful.

carsongross3• 11 years ago

Once again, The Platonic Engineer gazes hatefully at the gronky, ugly and awkward things that make things work in real life, rub his hands together and says...

"Burn it to the ground."

xg152• 11 years ago

Or you could actually stop for a second and find out why your current model needs so many ugly and awkward workarounds and try to adjust it accordingly - which seems to be what Oracle is trying to do in the long term.

taeric4• 11 years ago

This is an odd one, though. I can not remember seeing/hearing any real issues with Unsafe. For the most part, people don't use it. There are a few that do, to ridiculous benefit. As things stand right now, without it, you can not get anything close to the performance people are seeing with it.

So, what many of us are seeing here is someone is upset with basically the shape of an API. They have no evidence that I have seen of trouble caused by it. Only a desire that such things not exist. This feels almost puritanical in how it is being exercised. Which is just plain silly for a technical debate.

carsongross3• 11 years ago

But a new model should be that: a new model, not a suicide pact. Wise engineers keep the old API working while offering the new API (which, soon enough, will acquire its own gronkiness.) See the x86 and OS9/OSX evolutions, for example.

Thinking about the domain problem in the abstract is smart. Thinking about the current system in practical terms is wise.

ScottBurson• 11 years ago

Yeah, I don't get it either. Even the Lisp Machines had unsafe subprimitives you could call if you needed to and knew what you were doing.

ihuman• 11 years ago

What is "Unsafe?" Is it unsafe, as its name implies? If it is unsafe, why to so many programs use it?

dikaiosune3• 11 years ago

It's part of the OpenJDK/Oracle VM implementation, not an official part of the Java standard library. It allows for all sorts of fun stuff like native compare and swap operations (used for concurrent collections iirc), object allocation without calling the constructor, accessing blocks of memory directly without object references, and other things. The reason it's called unsafe is because it allows developers to bypass a lot of the safety guarantees that java is supposed to provide as a memory managed language. But it's useful for getting more performance out of the VM.

There's currently an effort underway to push through a variety of Java enhancement proposals that would add a lot of the unsafe functionality to the official standard library. This would enshrine the unsafe functionality as official, remove access to some of the more dangerous and less useful parts of unsafe, and would also mean that the functionality would be properly documented, which it is currently not.

Daneel_2• 11 years ago

Thanks for the thorough explanation - you summarised the issue well, as well as explaining the impact and flow on effects.

I have to say, I agree with the engineer.. I'm cringing just knowing this sort of practice is seen as acceptable, and it reinforces my distaste for Java. If you have to resort to insecure practices to achieve performance, there's something wrong with the framework.

MichaelGG3• 11 years ago

Why remove it though? What harm is being caused?

pron3• 11 years ago

Chill.

There is no disaster in the making. Oracle is very much aware of the issue, and is actually already taking the very same steps suggested by the author. It has assembled a public working group that is coming up with a spec for a public "Unsafe" API[1] (that is safer than Unsafe).

In the meantime, use of Unsafe in Java 9 will be possible with a command-line flag in order to allow the working group more time to create the new API and for Unsafe consumers to migrate, and of course to allow existing libraries to run on Java 9.

[1] https://docs.google.com/document/d/1GDm_cAxYInmoHMor-AkStzWv...

jryan495• 11 years ago

The point is if you have to pass a flag to allow the use of Unsafe, you're breaking "works-out-of-the-box" backwards compatibility.

pron3• 11 years ago

"Works-out-of-the-box" backwards compatibility -- without flags -- has never been guaranteed (even the default GC is about to change), let alone for non-public classes. Whenever you compile a class that uses Unsafe you get a warning (that you can't disable!) that says "this class may go away at any time". Still the reality is recognized, and you'll still be able to use Unsafe -- despite all warnings -- and all you need to do is add a flag. I think that's very reasonable.

snuxoll5• 11 years ago

Backwards compatibility is only guaranteed for official Java SE and Java EE API's existing in the java.* and javax.* packages, anything in sun., com.sun., etc is internal to the JDK and has no such guarantees.

Every time you hear of an application that only works on a specific release of the JRE you can know for sure it's because they use undocumented internal API's — keep in mind this not only impacts compatibility with other releases of OpenJDK/Oracle JDK, but the sun.* API's are not typically part of third-party JVM's like JRockit or the IBM JVM.

Stop using internal API's and then bitching when Oracle makes changes to them, you only have yourself to blame.

ZitchDog• 11 years ago

Entire companies exist around functionality provided in Unsafe. I don't see this going well if they remove it without providing access to the functionality it gives.

Java has plenty of problems, but backward compatibility over the years has been simply amazing. This would chip away at one of the few legitimate reasons to use Java in 2015.

xxpor• 11 years ago

sun.* has ALWAYS been exempt from that guarantee though.

LoSboccacc4• 11 years ago

yeah they went the fast and cheap route instead of doing the right thing, and they are now going to be bitten by it, and they can go cry a river for all that I care.

those are the sloppy engineers that are holding back evolution of software at large, from little things like obscure functions to huge operating systems like http://blogs.msdn.com/b/oldnewthing/archive/2005/07/28/44439...

jryan495• 11 years ago

Java is very explicit that anything not part of the public API may not work in the next version. So I'd say them disabling it, they're in the right. Since so MANY projects use it though, I think it might need some special attention.

geofft8• 11 years ago

Entire companies existed around DOS extenders and TSRs. Microsoft rearchitected and everyone is better off for it.

The argument needs to be that the use of Unsafe by these companies (or the projects listed on the blog post) both is good and has no alternatives, not that the use exists.

taeric4• 11 years ago

There should be some argument made that the use by these companies is causing trouble, as well. DOS had real identifiable reasons not to use it. The rhetoric used so far just has Unsafe upsetting some engineers for existing.

razorsese• 11 years ago

Literraly meme language

geofft8• 11 years ago

The quote stops short, and for someone like me who doesn't follow Java very much, it feels like it's misquoting via omission:

"Let me be blunt -- sun.misc.Unsafe must die in a fire. It is -- wait for it -- Unsafe. It must go. Ignore any kind of theoretical rope and start the path to righteousness _/now/_. It is still years until the end of public updates to JDK 8, so we have /years /to work this out properly. But sticking our heads in the collective sands and hoping for trivial work arounds to Unsafe is not going to work. If you're using Unsafe, this is the year to explain where the API is broken and get it straight....

"Please help us kill Unsafe, kill Unsafe dead, kill Unsafe right, and do so as quickly as possible to the ultimate benefit of everyone."

The quoted engineer understands that people use it, claims that there's no immediate risk (since Java 8 will stay around), and very very strongly implies that there is a plan to move from Unsafe to something better than Unsafe providing equivalent functionality in a stable way, and asks for help with that. Is this correct?

jryan495• 11 years ago

Yes you are.

BhavdeepSethi• 11 years ago

Here is the link to the full quote.

http://mail.openjdk.java.net/pipermail/openjfx-dev/2015-Apri...

pdeva12• 11 years ago

The post directly links to the full quote. If the intention was to misquote that would not be the case.

there is a plan to move from Unsafe to something better than Unsafe

It is this plan and the implications of it that the post is discussing

gclaramunt• 11 years ago

Actually I don't see the post discussing anything, is just feels like "Unsafe is going away, the sky will fall!"

scott_s2• 11 years ago

The intention may not have been to misquote, but that is the effect that I saw. I had a similar reaction that geofft did when I read the whole email: it sounds like a reasonable effort to start the process of moving to something better than what is currently available. The submission here does not seem to take that into account.

colin_mccabe2• 11 years ago

I disagree that the article is "misquoting via omission." The article claims that "this engineer hates the Unsafe class for no real reason at all." The full quote which you reproduced doesn't offer any reasons for hating unsafe.

I also disagree that there is "no immediate risk." The end-of-life date for Java 8 is 2017, only 2 years from now. It can easily take a year or two to develop a product or internal tool. Would you sign off on a big new performance-critical project in Java, knowing that you might not be able to get the performance you want when these low-level APIs are removed?

Java has had a bunch of security problems in the last few years, but most of them have not been related to Unsafe at all. For example CVE-2013-0422 related to Java webapps, CVE-2013-4444 related to Tomcat, and so on. There were a bunch of bugs for POODLE and other SSL issues, similar to many other programming languages. Perhaps you could argue that CVE-2013-0422 was related to Unsafe, but since in this case it was an internal standard library class, presumably Oracle would continue to give itself permission to use whatever unsafe-equivalent they come up with.

If new APIs are needed are needed to replace uses of Unsafe, Oracle could simply add them gradually over time. This would be low-risk and easy for everyone. I can't see any reason why Unsafe needs to be "kill[ed]... as quickly as possible."

xg152• 11 years ago

Oracle plans to remove sun.misc.Unsafe in Java 9. This is an absolute disaster in the making and has the potential to completely destroy the entire ecosystem around Java.

You know your language has a problem if the entire ecosystem threatens to fall apart as soon as you actually try to enforce the language's conceptual model and try to make use of the guarantees it provides.

bitmapbrother2• 11 years ago

With the caveat that you need to believe in the hyperbole of a chicken running around with its head cut off.

sbov• 11 years ago

This seems a bit hyperbolic. Especially the comparison of Python 2 to Python 3. Its not like you could run Python 2 and Python 3 under the same interpreter by passing a flag.

karavelov2• 11 years ago

What about shipping sun.misc.Unsafe as 3rd party package? Is there anything in it that can't be done as through JNI?

snuxoll5• 11 years ago

No, you can do everything in Unsafe via JNI, but that comes with all the disadvantages of JNI (including .dll's and .so's is not so easy for a fat-jar deployment).

vardump3• 11 years ago

It'd be unusably slow. "sun.misc.Unsafe" compiles in the instruction stream. JNI implementation would need to go through a complicated calling mechanism. I think it'd be up to 100x slower.

karavelov2• 11 years ago

At least in OpenJDK it is in fact implemented as a JNI package, so it is already going through the JNI calling convention: http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/s...

exabrial• 11 years ago

Document it, make it public. It's very much needed!

jart• 11 years ago

I just read an article that says in order to even use Unsafe, you have to use reflection to unwind the stack and go searching for a reference to the Unsafe instance, because the Java authors worked so hard to make it private. So it seems to me that they'd be well within their rights to remove the class.

Normally when you go to that level of epic hackery, you can't expect for official support or backwards compatibility. So the author of this article using scare tactics to keep that feature seems rather disingenuous.

orf• 11 years ago

Could you provide a link to the article? I'm not saying you're wrong (I'm not a Java guy) but that sounds convoluted.

twic3• 11 years ago

This is how you get access to Unsafe:

  Field theUnsafe = Unsafe.class.getDeclaredField("theUnsafe");
  theUnsafe.setAccessible(true);
  Unsafe unsafe = (Unsafe) theUnsafe.get(null);

Reflection, but no stack unwinding.

kkmickos• 11 years ago

Many people seem to be upset with Oracle, but I think they should be upset with the vendors who ship applications/libraries that depend on the "sun.*" packages.

Oracle (then Sun) have since _1998_ strongly discouraged the use of them: https://web.archive.org/web/19980215011039/http://java.sun.c...

Someone12343• 11 years ago

Fine, but why isn't the argument: "This then needs to be standardised in a Java.* package and correctly documented."

Unsafe is useful, and is evident by how popular it is. Heck even Sun/Oracle must agree because they created it for their own internal usage. So why not just go ahead and standardise it.

vbezhenar3• 11 years ago

Why libraries need Unsafe, after all? What problems does it solve for them?

kasey_junk7• 11 years ago

It's almost entirely about performance. Unsafe allows you access to concurrent intrinsics as well as memory layout functions.

It's true that it is widely used, but the people that use it also understood the ramifications of using it. If you were trying to make jvm agnostic code you didn't use it. If you were trying to make code that could run on a variety of jvm versions, you didn't use it.

A lot of what people use it for is being rolled into documented/standard libraries and this is just the migration pains coming out.

vbezhenar3• 11 years ago

Why those concurrent intrinsics are not available with "safe" code? They are not compatible with Java Memory Model? They are not implemented at some platforms? What's so unsafe about concurrent primitives? There is API in java.util.concurrent.atomic, for example.

I understand that Unsafe is used for direct access to memory, but why this direct access is needed? May be there are other ways to solve those problems.

clamprecht• 11 years ago

At least he didn't name the article "Unsafe considered harmful."

bowyakka• 11 years ago

Ok moving past the rhetoric of don't use internal classes, there is a value to sun.misc.Unsafe. It has been well known by Oracle for some time that it has to go away right.

Realise that those of us that do use it, are fully aware that it is a hack, and that we should be doing cleaner things. However know that those cleaner things _dont exist_; and that you, Dear application developer might not quite realise some of the shadow puppetry in making your favourite library work, please lay off the bile.

As others have said, this blog post stops short, the full quote is part of a very long and ongoing process to rid ourselves of Unsafe, replacing its majors usages with new API's.

This is a process Oracle started last year after the previous sun.reflect.Reflection.getCalleeClass mess. Yes, it no one should have been using an internal class. This method did not go away; it was made only available to the JVM internals, locked out from end users. The upshot of this was that, with no sane alternative, suddenly logging became between 2x and 100x more expensive where class names become involved. The JDK team somewhat back peddled on this change, and we are stuck here in a strange limbo land.

Was it right of Oracle to do this? Absolutely, everyone who goes down the internal package route is fully aware that these are open-state secrets, sure we all know them, but we should not be talking about them. Support for these damn things requires version to version checking, hideous reflection hacks and being prepared to read a lot of JVM source code for when bugs occur.

In light of all of this, and being the general level of awesome the JDK folks are they started a very direct community outreach to remove safely Unsafe. Last year they publicly asked people on the mailing list, and privately via email to do a survey (http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-Ja...) on usages of Unsafe. I know this because I got an email from Paul Sandoz asking me to comment on Unsafe abusages. Out of previous thinking from the JDK folks, as well as from the aforementioned survey, we have a bunch of JEP's for improving things regarding Unsafe.

Where things have gone slightly sideways is that we have, currently no unified working group to deal with these changes. What this document is, is a proposal to make a working group, focused on getting the right changes into the JVM. In the same ways as Project Jigsaw is for modularising the JVM (and is indirectly responsible for removing Unsafe), or the MLVM project handled making the VM better support other languages.

Unsafe is dead, long live Unsafe! I fully expect that we will achieve the right changes in during Java 9 to allow those of us that need to break the rules, to do it in a supported fashion. I am _looking_ forward to the day when I can do setMemory or getAndSetInt via a supported API, and not one where I have to know the words Unsafe.theUnsafe. The above is a storm in a teacup, its open-source democracy at work.

sreque• 11 years ago

There's a lot of backlash here against the author with comments like:

"You used a sun.* package? You deserve to die in a fire as well!"

These comments look like they are coming from people who have never used Unsafe. First, Unsafe provides some immensely useful features to the Java ecosystem. Second, there is no alternative to Unsafe on the JVM. That includes JNI (native code integrated within the JVM).

So far, Oracle has stated their intent to remove Unsafe from Java 9. To all the users of Unsafe, Oracle is basically saying, "tell us your use cases for Unsafe, and if we decide they are valid we may try to create an alternative for you. If you're lucky, we may even ship that alternative in Java 9 or 10! Have a nice day!"

Many people, myself included, find this unacceptable. For instance, it looks like several groups are currently collaborating on a document discussing uses cases for Unsafe and whether alternative features satisfying those use cases are expected to appear in Java 9:

https://docs.google.com/document/d/1GDm_cAxYInmoHMor-AkStzWv...

As you can see, the majority of the cells in the column "Expected in Java 9" flat out say "no". There is a good reason people are up in arms about the removal of this API.

Personally, if I had to choose between Oracle keeping Unsafe and Oracle implementing Java 8 Lambdas, I would have picked Unsafe in a heartbeat. At the end of the day, Lambdas are mostly syntactic sugar, especially in their current implementation. Unsafe, on the other hand, actually empowers you to do something more.

geofft8• 11 years ago

That link is a very useful contribution to this discussion (and possibly the first really useful comment in this thread), thanks. Everyone's entitled to their opinion, informed or not, but if you want an informed opinion, the question needs to be what the current uses of Unsafe are, why they're used, and whether there are plans for a migration path.

It sort of seems like the major use case (judging from those numbers) is reading and writing raw byte buffers, possibly mapped from other objects/files, with native performance. Is there a potential sensible design for a bounds-checked raw data API ("unsafe but only within these pages") that maintains the JVM's usual safety guarantees for all other data, and can be implemented in a performant way?

pdeva12• 11 years ago

The document is also linked in the blog post, on the very first line...

bcg14• 11 years ago

If the author is correct (which I am not questioning or doubting) and very high profile projects are using this feature... and Oracle refuses solicit input from the community... those developers with clout can and should maintain a fork of OpenJDK to include this feature. If things like Netty only run on the forked version of the JDK, and the developers are committed to not porting or working around a lack of Unsafe support, Oracle will be forced to reconcile with the community or risk burning down their own house.

Do not be surprised if they remove this feature from the GPL'd OpenJDK but maintain an analogous feature in their proprietary JRockit JVM. There is a good chance that this is a "shot heard around the world", and now is the time to start fighting back.

pjmlp6• 11 years ago

Oracle did a survey last year exactly as input for what to do.

pjmlp6• 11 years ago

Except that those of us on the Java community that follow Java development are fully aware of the 2014 survey, done by Oracle about how Unsafe is being used and what is our opinion on it.

Of course, many HNers that hate Java aren't aware of those discussions and post comments without knowing the full background.

MichaelGG3• 11 years ago

What is the purpose of enforcing module boundaries? That appears to be why none of these internal classes will be accessible. Outside of a sandboxed environment, who cares what you do? Is it for some magic optimization? Or just some folks on a rampage? (The quote from the guy saying it needs to die, and I read the mailing list, well it seems to be baseless.)

jkot• 11 years ago

Compression in my library is 3x faster with unsafe. I think it is lesser evil then native code integration.

skrowl• 11 years ago

Still using Java in 2015 – A disaster in the making

bitmapbrother2• 11 years ago

What would a MS fanboy suggest as an alternative?

hyperpape• 11 years ago

Why on Earth is it a big deal that you need to pass a flag to the JVM at startup?

I understand that backwards compatibility is valued, but if I went to my CTO and said "we're gonna drop Java 9 in production, and we're not gonna test beforehand"...the reaction would not be pretty.

rebootthesystem3• 11 years ago

This is related and unrelated and maybe even meta to some degree:

There are a couple of companies that are guilty of using their free software update mechanism to try and trick people into installing crap they don't want (and might even mess-up their systems): Sun and Adobe.

I almost happened to me last night as I updated Flash on a test system and some bullshit anti-virus crap-ware was going to be installed because I was moving fast and neglected to un-check the check-box on the site. I was able to stop the install process and start over. No harm done.

These companies need public shaming to stop using this bullshit method to make a few more bucks. Are they so hard-up that they need to trick users to install crap to make money? I sure hope not.

Sun, Adobe, please STOP trying to trick users into installing crap-ware they were not looking for in the first place.

kasey_junk7• 11 years ago

You understand of course that Sun has been out of business for a very long time right?

rebootthesystem3• 11 years ago

Sorry, brain fart, I meant Oracle

throwaway2048• 11 years ago

java hasnt been bundled with crapware since oracle bought it

rebootthesystem3• 11 years ago

Some installations I've seen have crapware pre-selected. I haven't looked at it lately, maybe they changed this.

scubaguy• 11 years ago

I read this article yesterday and I really dislike how it is worded. The "this engineer" has a name, Donald Smith, this is what he said:

"If you're using Unsafe, this is the year to explain where the API is broken and get it straight.... Please help us kill Unsafe, kill Unsafe dead, kill Unsafe right, and do so as quickly as possible to the ultimate benefit of everyone."

It reads like a request to have a discussion on Unsafe, to talk about it and find solutions. Unfortunately, I don't see anyone pointing out why Unsafe should be kept around on that thread (http://mail.openjdk.java.net/pipermail/openjfx-dev/2015-Apri...).

And this quote from Oracle shows an effort to do things right:

http://mail.openjdk.java.net/pipermail/discuss/2013-October/... "This is definitely something we plan to propose for SE 9. I expect to see a JEP from the current maintainers of sun.misc.Unsafe..."

This (http://blog.codefx.org/java/dev/how-java-9-and-project-jigsa...) seems like a more balanced article on the situation.

"So far we focused on the problematic aspects of Project Jigsaw. But that should not divert from the exciting and – I think – very positive nature of the planned changes. After reading the documents, I am impressed with the scope and potential of this upcoming Java release. While it is likely not as groundbreaking for individual developers as Java 8, it is even more so for everyone involved in building and deploying – especially of large monolithic projects." near future. (This is far too small to require a whole JSR.)"

Removal of Unsafe in Java 9 – A disaster in the making

Related Stories