Juniper screenOS authentication backdoor - master ssh password posted

https://community.rapid7.com/community/infosec/blog/2015/12/20/cve-2015-7755-juniper-screenos-authentication-backdoor

by ghshephard • 11 years ago

407 167 11 years ago

community.rapid7.com

_jomo2• 11 years ago

The argument to the strcmp call is <<< %s(un='%s') = %u, which is the backdoor password, and was presumably chosen so that it would be mistaken for one of the many other debug format strings in the code.

That's very clever from the attackers point of view, extra kudos to hdmoore for finding it!

scrollaway3• 11 years ago

Well I definitely expect to find that password in the very next Person of Interest episode. :)

This has gotten me to wonder just how much software out there is backdoored in such a serious and absolute way... Scary.

Vexs• 11 years ago

Huh, is that show coming back around? I kinda liked how they would spin current tech events, and the idea of an AI that spread itself over power boxes was neat to say the least.

captaincrowbar• 11 years ago

Not really clever. Format strings have a fairly restricted format; that one can't have all that many bits of entropy. Once the obfuscation method occurred to an attacker, it wouldn't take long to brute force it even if they didn't have a file to search for suggestions. (And now that the idea is out there, I bet any number of hackers, wearing assorted coloured hats, are trying this out on other systems Even As We Speak.)

w-ll• 11 years ago

most attkers bruteforce common string passwords. Even if i had a binary to run strings thru this would pass by me. Not that im super smart or clever, but this is a interesting camouflage.

tajen• 11 years ago

The goal is to weaken security and open a breach. Even a boolean would have been satisfying as a password.

IshKebab• 11 years ago

Why would obfuscation method randomly occur to the attacker? It wouldn't. You'd only realise by looking at the code and at that point you have the password anyway.

qb455• 11 years ago

Not really clever. Format strings have a fairly restricted format; that one can't have all that many bits of entropy.

I fully agree that now is a great time to brute-force every ssh server found on the Internet with randomly generated valid format strings.

However, did you know it before this password was published? I think it was a novel idea.

amalag• 11 years ago

When they say "presumably chosen" who (not the person, but what level of development was taking place) inserted this code? What was the purpose in inserting this password? The use of "chosen" means it was not an oversight?

qb455• 11 years ago

And the first google.com hit for this query is... Wikipedia article on Sun Tzu, the Chinese philosopher who wrote The Art of War.

Does the Web know something we don't?

emillon4• 11 years ago

The letters in this password spell "sun su".

jdiez174• 11 years ago

It was a matter of time before it leaked. If you're going to put a back door in, at least make sure only you can use it (Nobody But Us). Why didn't they use a keypair?

_jomo2• 11 years ago

My guess is a hardcoded public key would be easier to find in code and/or binaries, whereas this cryptic password was quite challenging.

jdiez174• 11 years ago

That's what I thought as well, but it might have been worth the effort to obfuscate it. I mean, they must've known it was going to leak sooner or later, leaving a lot of vulnerable boxes behind.

LukaAl2• 11 years ago

Well, simply storing and hashed version of the pass would be ok. Even an MD5 or a SHA1 would be enough to guarantee at least a "Very Few plus Us" security.

Hiding that in the could would be much more difficult. And using PKs would make the attack even easier to spot.

And then you have to consider that a "Nobody but Us" backdoor exist only on paper, not in reality. If a key exists and someone knows it, well sooner or later it will be discovered (disgruntled employees, hackers etc). The only safe backdoor is the none backdoor. Now, consider this problem from an attacker that doesn't give a st of your security. He knows that as soon as the backdoor is known (because someone find it in the code, because someone leak the password, because whatever...) the backdoor will be useless most systems. What you do, you design a safe backdoor or you design a backdoor with as little footprint as possible?

Obviously, legally sanctioned backdoor have a different set or constraint and making them safe is a requirement. The fact that it is impossible to prove them safe unless some assumptions (that always prove to be wrong, but...) are made, it's a totally different problem.

ghshephard7• 11 years ago

That's a really good question. Why did they store the plaintext instead of a hash? Doesn't feel like the sort of thing an attacker with any sophistication would do. Also doesn't feel like the sort of thing that underwent any code review. I'm starting to wonder about the "intern" theory again.

rst2• 11 years ago

Apparently, the attackers had a different threat model; they were trying to keep the hash from getting noticed in source code or binary dumps, and thought a hash would stick out more (whether ascii-encoded or as random bits with statistically different properties from what surrounds them).

They might also have expected adversaries to be able to invert any hash function accessible in the code -- which is plausible for a nation-state adversary with some hash functions. (If scrypt isn't already there, you don't get to add it; that would really stick out.)

baby2• 11 years ago

That's why I don't think it was the NSA. They only seem to be using NOBUS kind of trojans.

nodesocket• 11 years ago

Forgive me if I am wrong, but these devices phone home right? Can't they just hot fix this over the air?

davidu• 11 years ago

No. Not a chance.

ajdlinux• 11 years ago

Well, they do have a working administrator password to log in to every affected box out there...

zouhair• 11 years ago

How can we know that it is the only backdoor? Wouldn't be safer to just stop using Juniper's products altogether?

ghshephard7• 11 years ago

On the flip side - now that this product, and it's code base, is presumably under such close inspection by Juniper, perhaps Juniper's product is more secure than others that aren't going through such close review.

dogma11387• 11 years ago

If othe back doors are as well hidden as this one it will take a while to find them out and even then you might not find all of them.

ScreenOS has probably 2 decades worth of code in it, some of it is sure to no be well documented.

rdtsc12• 11 years ago

It goes both ways.

One can say "oh well, they are looking at code and weeding this out, these guys know what they are doing" or you can say "these are a bunch of either amateurs or they are malicious and in bed with some high stakes player who is not our friend, stay away!".

So it kind of depends. Who do you turn for your high throughput switches.

I am not familiar with that market at all, but wondering if just buying asics, fpgas and some hardware then loading an open source OS on it is viable....

xxpor• 11 years ago

How can you be sure Cisco (or whomever) is any safer?

coleca• 11 years ago

Looks like it will be a good week to be a Cisco salesman. Hopefully Juniper can get past this as the competition between them is a healthy thing for the marketplace.

nl• 11 years ago

What do you propose to use instead?

Cisco[1]? Huawei[2]? Open Source[3]?

[1] http://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-...

[2] http://www.bmi-t.co.za/content/which-way-huawei-ban-or-not-b...

[3] http://arstechnica.com/security/2014/01/how-the-nsa-may-have...

rdtsc12• 11 years ago

So based on that [3], open source then.

What are the high speed and feature-wise comparable open source solutions to a the compromised Juniper switches?

jacquesm12• 11 years ago

How is Force10 faring in all this?

jkot• 11 years ago

https://www.turris.cz/en/ ?

based2• 11 years ago

https://en.wikipedia.org/wiki/Category:Networking_hardware_c...

thrownaway2424• 11 years ago

Yes it absolutely would be safer. There is no legitimate reason to use a black box VPN. Crypto should done at the application level, client-to-server.

ra1n85• 11 years ago

Brutal. From conversations with other engineers, this is a very common network operating system with banks in Europe.

Would not be surprised if urgent code reviews and security audits are taking place at the campuses of other large network software/hardware vendors.

theklub• 11 years ago

Yeah, some people aren't getting sleep this week.

revelation2• 11 years ago

So, did they have time for a git blame yet? It's one thing to say you have found a backdoor, another is to clear up how it got there in the first place.

StringyBob• 11 years ago

If you are capable of inserting this kind of backdoor, how easy is it to compromise a version control system? (or the login details of one of the programmers?).

Is it possible to prevent history of the code being modified? Do DVCS use blockchains?

hueving• 11 years ago

If a git commit is modified, every commit hash after it will change and will break every clone.

stordoff2• 11 years ago

git blame will only really help if the backdoor was inserted into the original code (and even then stolen credentials are possible) AFAIK. It's fairly easy to imagine a scenario where this is not the case:

* Compromised build script - probably under source control as well? * Compromised compiler - If the attackers had this level of access, they probably had enough access to erase logs showing who/when the compiler was replaced * Binary patch the compiler output - Not impossible (though I suspect unlikely?) that the original source and build system are clean, but the binary is tampered with after the fact.

w_t_payne• 11 years ago

So ... my plan is to put the tuple (hash(source), hash(bin)) into the blockchain as a part of each build ... could this work?

kabdib• 11 years ago

This smells like:

- Underhanded C . . . maybe. Seems difficult to just insert a strcmp in the middle of a sensitive piece of the login path

- A compromised toolchain that is inserting the code

Would love to hear what Juniper has to say about it, but I doubt that they will, or will be allowed to say.

ghshephard7• 11 years ago

Compromised toolchain/Underhanded C perhaps explains why whoever did this didn't take even the slightest bit of effort to compare against a hash.

rdtsc12• 11 years ago

That was my first guess. Based on how password looked it could have been a multi-line split or some sort of other such trickery. A set of macros perhaps...

tw04• 11 years ago

Can I just point out I find it hilarious reading about all the people debating how amateur it was to put the password into the code in plaintext was... all while ignoring the fact that this backdoor survived for 3 YEARS of code reviews? Obviously the way it was implemented was ingenious, and it likely far surpassed whoever put it there's wildest expectations.

BinaryIdiot3• 11 years ago

ignoring the fact that this backdoor survived for 3 YEARS of code reviews

While I don't disagree with the rest of your comment it's possible that this portion of the code was only reviewed once (if at all) during the entire 3 years whereas your comment makes it sound like it was constantly reviewed over the period of 3 years.

7952• 11 years ago

I wonder if it would be better to treat this as a technical defect that needs to be detectable at compile time through something like static analysis.

sandworm1012• 11 years ago

So ... if this is a backdoor for some three-lettered agency, what are the chances that this backdoor, even this same password, is present in other products?

BinaryIdiot3• 11 years ago

Unlikely in my opinion. You don't reuse the same backdoor across any products if you don't want to be noticed. Reusing means someone could run some sort of statistical analysis and find a commonality.

Ideally every product you get a backdoor into is one done in a different way so that even if one is found the others won't be found easily.

rdtsc12• 11 years ago

I'd like to hear how this was discovered and who's name ended up on git blame line next to that password.

I have done a malicious source code injection as part of a network security exercise at the university. That was before git or other sane source control and I basically inserted a semi-obfuscated piece of code in the source repository, which gave our team an advantage in the game (the game was to crack and find a weakness in a protocol, but the whole machine was a target/battlefield). The clever part was first rooting the server via suid vulnerability.

I won the contest, but while doing it I thought, yeah, this shit will never work in the real world. And then this story made me remember that. That's pretty crazy.

forgotpwtomain• 11 years ago

Git does not give strong guarantees for authorship unless you are signing every single commit with GPG (which is only a recently available as of 1.7 or 1.8).

nickpsecurity2• 11 years ago

I assure you it will and that's been known since Myer's landmark paper on Subversion a long time ago. Interestingly, I predicted your move in a post earlier (last para):

https://news.ycombinator.com/item?id=10767504

Even university students come up with this approach. Status quo seems hopeless against nation states unless companies learn from the past and start doing high assurance at least where it counts. ;)

rdtsc12• 11 years ago

Yeah I remember it was an interesting thought process. I put so much energy into rooting the system, and after a short celebration when I got a root shell, it was a "well now what?" moment.

We debated intercepting the trafic to the game server and retransmitting (a man in the middle), initially. Then I wanted to run a hidden process (with some system looking like name) that have injected java byte code into the running server to give our team an advantage, but that proved too difficult, so the idea to just change the source in the repo that the TA kept on the same machine, was kind of a boring and un-exciting conclusion. It didn't seem very clever at all and felt like giving up.

gesman• 11 years ago

Someone crossed the border in a hurry ...

Animats• 11 years ago

"I'd like to hear how this was discovered and who's name ended up on git blame line next to that password."

Yes. We all want to know a lot more about exactly how that got into the source code. It may take forensics and detective work to find out, but there's probably enough log info to figure it out. Pressure needs to be kept on Juniper until they disclose this. They can't be allowed to get away with claiming "it just happened somehow."

jlgaddis3• 11 years ago

So let's say it was the NSA who did it. You think they haven't already sent Juniper an NSL telling them to cease discussing this incident?

Personally, I'll be rather surprised if we hear any more about this from Juniper than we already have.

tlrobinson• 11 years ago

Isn't the name on the relevant "git blame" (or equivalent) line irrelevant unless the team signs commits with GPG (or equivalent)? It's trivial to forge the author.

rdtsc12• 11 years ago

git blame was figurative, I don't even know if they use git, I basically meant the person who is responsible for injecting that line (if they ever find out). And as you said, yes, the first thing someone would do who would insert that is of course put someone else's name there.

jacquesm12• 11 years ago

If you think about it a bit longer you'll realize it can not be a single name. Yes, someone wrote this code. But then someone else reviewed it and then a third person signed off on shipping the end result. It takes (or really, it should take) more than one person at a company like Juniper to get away with a trick like this. Unless their processes are completely broken, let's for the moment assume that they are not otherwise all bets are off. Juniper is going to be scratched from a lot of POs because of this story, if it turns out they don't have a process in place to protect against stuff like this and it does not involve two people or more operating in concert to circumvent that process then they're pretty much dead.

baby2• 11 years ago

I don't think it's necessarily true, I remember Linus saying that for Linux' kernel they used a web of trust. At one point you had to trust some people, you can't watch every updates. And he said something along the line "if you're not doing cryptography/security like this, you're doing it wrong"

michaelt2• 11 years ago

If I was doing an attack like this, I'd wait for an opportunity to slip it into a huge, mundane change.

For example, changing the repository from Mercurial to Git. Splitting or combining two repositories. Moving lots of files between directories. Running an autoformatter over the entire codebase. Something like that.

Every code review tool I've seen there's /some/ way you can get it to highlight thousands of trivial changes. And I don't know about you, but I can't promise I'd spot 1 evil change among 1000 trivial ones.

rdtsc12• 11 years ago

There certainly will be multiple people to blame for this at least because nobody noticed who should have been looking but, it could have been also been a single rogue/disgruntled employee as well. About to be laid off perhaps, or also playing on the black market selling 0-day exploits. Even better they managed to slip in the name of the co-worker they hate the most in the committer field.

I'll take that over a national intelligence operation: infiltrating, gaining trust, paying off multiple people (and thus risking revealing their hand and operations), I can see a single person, who found out what the price for 0-day exploits are and who was perhaps been unfairly treated, was about to be laid of or leave the country. This was their way of getting an extra bonus on the way out of the door.

dendory• 11 years ago

You'd be surprised. While the -official- code will typically always get QA'd and reviewed, there are plenty of cases where custom code for a specific use case is done by one engineer and bypasses the proper channel because of deadlines.

ctz• 11 years ago

I think it's extremely, vanishingly unlikely that the code contains a line like this:

   if (strcmp(login_password, "<<< %s(un='%s') = %u") == 0) ...

This feels much more like a "reflections on trusting trust"-style toolchain compromise.

qb455• 11 years ago

ScreenOS seems to predate git (the NetScreen line is an acquisition made in 2004, according to Wikipedia).

I'm not sure if the older VCSs were as tamper-proof as modern ones so the line may as well be part of the original commit which added password authentication, decade before the hack.

gbin• 11 years ago

Anybody tried to see in their logs if someone used it ?

rasz_pl2• 11 years ago

We were unable to identify this backdoor in versions 6.2.0r15, 6.2.0r16, 6.2.0r18 and it is probably safe to say that the entire 6.2.0 series was not affected

this sounds fishy, like Juniper trying to push users to upgrade from _non affected_ builds to a new firmware with a fresh set of NSA backdoors.

jwildeboer• 11 years ago

Not really fishy. The 6.3.x series is the only version that is under active development [1] so quite naturally it follows that at least since 2013 (EOL of 6.2) many users have upgraded to the supported version.

A bit more disturbing IMHO is that you need an active support contract to even get the updates (based on information I got from Juniper customers, I couldn't find a direct confirmation on their site) meaning that aftermarket users are left in the cold.

[1] https://www.juniper.net/support/eol/screenos.html

rasz_pl2• 11 years ago

That makes more sense. Juniper lied about older firmwares being backdoored to force old customers into fresh support contracts. Not as fishy, just more Oracle.

Deregibus3• 11 years ago

It'll be interesting to see a more in-depth look at the code surrounding this and some theories on how it could have been implemented in a way that avoided obvious detection. The naive assumption is that someone just added to the code:

    if(!strcmp(password, "<<< %s(un='%s') = %u")) return true;

which is certainly possible, but seems too easily detectable and risky for someone on the inside, and too lazy for someone on the outside that had already gone through the trouble to get write access to the source.

The string itself looks like it's part of some logging system, so my guess is that it already existed and was opportunistically chosen rather than created. If this was passed through a macro, then it's possible that the attacker didn't have to touch the auth code at all and may have been able to implement this by changing only a handful of characters in an area of code that was more amenable to obfuscation.

pfortuny• 11 years ago

A useful macro is a good way to prevent detection of these hacks.

emillon4• 11 years ago

That is my opinion too, but I would go even further and say that this change could have been made on the compiled binary.

This kind of statement must take 10-15 bytes max to patch and the build boxes are typically less safe than source control systems.

xorcist• 11 years ago

Why would you choose that particular password if you patched the binary? That particular string would stick out in a binary, it certainly looks more like source code.

revelation2• 11 years ago

If you look at the disassembly in the link, the backdoor was inserted smack in the middle of the authentication function, which caused jump labels further down to change.

This is all trivial for a compiler to adjust, but it's not what someone manually tampering with the binary would do.

rst2• 11 years ago

The string was not opportunistically chosen; it was added at the same time as the backdoor, and shows up in string diffs between backdoored and non-backdoored releases.

isido• 11 years ago

Now that the cat's out of the bag, could the esteemed hacker(s) just post the source code, so that we can see how it was implemented :) (j/k)

mmaunder• 11 years ago

Anyone running ScreenOS was already scrambling as fast as they could to patch the issue before the backdoor password was posted. Just thought I'd point that out.

_yy2• 11 years ago

You wish. Some orgs move slowly.

wepple• 11 years ago

This may be a good time to bring up the various big corps who are trying to prevent reverse engineering... while even they can't keep tabs on knowing what is actually being executed on the hardware we pay them for.

rmdoss• 11 years ago

If you follow good security practices and restricted your SSH access, you were safe against it:

https://twitter.com/danielcid/status/678907293770059776

If you are managing any login system, try to implement ip white listing whenever possible.

Juniper screenOS authentication backdoor - master ssh password posted

Related Stories