Bit.ly compromised – More Details

http://blog.bitly.com/post/85260908544/more-detail

by shdon • 12 years ago

60 47 12 years ago

blog.bitly.com

dewey• 12 years ago

"If you registered, logged in or changed your password after January 8th, 2014, your password was converted to be hashed with BCrypt and HMAC using a unique salt. Before that, it was salted MD5."

Are they saying the passwords were hashed with MD5 before January? Because that'd be a bit scary.

camus22• 12 years ago

You'd be surprised how much "PHP login" tutorial use md5 for password encryption. And then people are surprised these devs dont known better? it's because of all the mis-eductive content the web cant get rid of.

krapp2• 12 years ago

Also, many startups consider security to be a waste of time, since you don't need a secure site to generate revenue (arguably, appearing secure has more value in that sense than actually being secure.)

Also I still see people now and then arguing that salted md5 is perfectly secure and that using a slower hash or any hash at all either doesn't add value or actually harms value. So they may not know better, or they may know better and simply not care.

voltagex_• 12 years ago

I once registered wrongtutorial.com to call out incorrect and dangerous tutorials. I couldn't write enough content to get it off the ground. Do you think it'd still be a good idea?

codecondo• 12 years ago

It's not scary, it's the truth, and it's bad. But, what can you do?

Dylan16807• 12 years ago

Salted MD5/SHA is basically the standard even though it's awful, and the flaws in MD5's algorithm don't really apply to the password case.

tptacek• 12 years ago

The flaws in MD5 don't really apply to passwords, but the flaws in salted MD5/SHAx sure do.

KMag• 12 years ago

It's still a bit scary that they're apparently using some ad-hoc KDF (key derivation function) built from bcrypt and HMAC. bcrypt is a decent KDF, but there are several ways to make a bad KDF from a combination of bcrypt and HMAC. For instance, concatenating bcrypt(salt, pw) and HMAC-SHA512(salt, pw) makes guessing passwords way too fast.

I'm hoping that all they've done is use bcrypt(salt, HMAC-SHA256(salt, pw)) to get around the bcrypt 72 character limit. However, using scrypt would have been better.

nialo• 12 years ago

why does concatenating bcrypt(salt, pw) with HMAC(salt, pw) make guessing fast?

Intuitively it seems like you'd still need to guess the bcrypt portion, making it no worse than just bcrypt.

reedloden• 12 years ago

Hopefully, the 'salt' passed to HMAC-SHA256 is a pepper (preferably stored outside the database), not the same user salt as what bcrypt uses. As for scrypt, it definitely has great potential, but I still feel it's too new and not as well understood to be used 100% correctly in production by most folks (see also http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scryp...).

azinman2• 12 years ago

What's wrong with bcrypt?

michaelfeathers• 12 years ago

I've never been clear about what value-add registration brings to a link-shortener, so I've just used bit.ly without registration.

I'm kind of happy about that now.

edraferi• 12 years ago

it allows you to track stats more easily. for example, you can link bit.ly to Buffer and get a unique bit mark for each link/platform pair. then you can track engagement separately for each platform.

brownbat2• 12 years ago

Couldn't you just have an IFTTT pull anything you bitly into a bucket of links to watch stats on? Why does bitly need to know who you are, or what links you submitted, if you can just remember (maybe through automation) which links are yours, and bitly or buffer can monitor each link individually for stats?

My guess is that logins probably help bitly track usage patterns, improve the product, and keep an eye out for ways to increase their value by measuring engagement with their userbase. Maybe it was just "everyone's adding a social layer" peer pressure. Not sure.

There are security costs to "use passwords everywhere, for everything," but they are more diffuse. We've ended up with a system where security best practices would recommend using hundreds of distinct passwords that are each too complex to memorize (so you can dump them in a locker or support one login services, each creating a single point of failure, or you can use a similar password or mental password algorithm for each site, it's all tradeoffs).

I don't think there's an obviously better solution, I think it's just a tragedy of the commons we were probably inevitably going to trip over sooner or later.

patcheudor• 12 years ago

Hopefully this will cause companies to think twice about implementing someone else's link shortening service within their software solution. I've done several reviews of software solutions which have used bit.ly and other shortening services for things like providing a link to corporate identity portals.

It's quite common for a company to want to use their own identity portal for employee authentication with a particular piece of software and a few companies implement this via giving users shortened links which resolve to the identity portal location. They will hardcode the request to bit.ly & then give the users the token to enter when the software starts. It will then send the full bit.ly request and get a destination location back for the portal.

In every case I've seen this, I've written it as a high risk finding since neither the software vendor nor the company using the solution has ever done an audit of the security controls at bit.ly or the other link shortening services. If a link shortening company like bit.ly is compromised & a bad guy is able to modify link destinations then it's pretty much over in terms of security for a software solution implementing this model. There also tends to be some issues in terms of implementing public key or certificate pinning when using a third-party but I'll save that for another day.

alokyadav15• 12 years ago

Agree . as a individual user's point of view, me too don't understand signup thing for a url shortner .

balls1872• 12 years ago

For the individual user, there probably isn't.

For companies, marketers, etc, bitly, and other url shortners provide data and analytics ontop of the links, allowing more insight into link shares & clicks.

When you're interested in tracking the data from a link, or series of links, having a persistent account is necessary.

theboss• 12 years ago

"Security Team"?.

I'm honestly curious. I've never seen startups that aren't security startups hiring a security team. Is this a real team or just their developers doing an extra job?

bullseye• 12 years ago

I don't fault them for it if they don't. "The Bitly security team learned of the potential compromise..." sounds much better than "our programmer, Joe, got an email..."

addandsubtract• 12 years ago

Good guy Joe. Developer by day, security team by night.

nknighthb5• 12 years ago

Preferably even the smallest company would have at least a designated "security guy", even if they don't spend most of their time directly on security issues. If there are two, there's your "security team".

Reality, of course, is that most companies don't give much thought to security at all until it bites them in the ass, and security competency is rare enough that even a designated "security guy" could be worse than useless.

nknighthb5• 12 years ago

Missing step: Asymmetric encryption of off-site backups with passphrase-protected private keys kept offline.

kator• 12 years ago

Is the root issue here that they stored remote server database passwords in their source code?

I'm always hesitant to have source code with hard coded passwords, I know at some point you have to have them "somewhere" but anything you can do to reduce attack vectors for them is helpful.

What do people think should be best practices for storing sensitive configuration data? I know some people will have just a config repo with very limited access but that still seems dangerous at scale when that ends up being a lot of people etc.

nknighthb5• 12 years ago

I have yet to find any solution to this problem that I like. A separate config repo is least bad of the various bad options. The others have all been error-prone and failed to play nicely with version control.

If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.

jarrett2• 12 years ago

If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.

You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.

pavel_lishin• 12 years ago

I wonder if AWS allows security-group-controlled access to databases, similar to the way you can only allow communication between two ec2 instances based on their security groups.

kachnuv_ocasek• 12 years ago

What about storing the credentials in environment?

jarrett2• 12 years ago

Some people prefer to keep the keys on specialized servers when a) the need for security is especially acute, and b) the project has the budget.

For more day-to-day stuff, environment variables on the server are an option. If you want to be able to provision a new server 100% automatically and keep the credentials out of source control, you have a few options. You can add a quick interactive phase to the deploy script, wherein the engineer must enter the key manually. Or the engineer could enter a value which is then passed to a KDF.

andrewjshults• 12 years ago

We use git-crypt (https://github.com/AGWA/git-crypt) in our ansible repo for credentials. Credentials are stored as variable YAML files so anyone can work with them in templates, but only our deploy box and SRE team can encrypt/decrypt the credential files themselves. Generally works pretty well, but you do need to remember to run credential deploys on new boxes. Eventually we'll probably move to a HSM (hardware security module) for the actual storage, but those are pricey (especially since you need two of them).

druiid• 12 years ago

Chef supports encrypted data-buckets where you decrypt only on the server. Puppet supports hiera data with encryption as well (via plugins). There's no reason anyone should be storing password information in source, especially API keys which I'm guessing this was (S3 information more than likely?). I think about the extent of private information I'd be okay with storing in source is DB user/pass information because that will generally require compromising the machine more generally first and by that point they already had access to your DB even if the password/username wasn't stored in source.

Essentially this will serve as a wake-up call to the Bitly guys and hopefully they'll take some steps to deploy this stuff via CM or similar in the future.

thefreeman• 12 years ago

I read it a bit differently. I don't think they store their credentials directly in their source code repository. I think they are saying they have a separate repository specifically for credentials.

It seems like a bit of a strange way to manage sensitive information, but not really any worse then a lot of other things I've seen.

ForHackernews2• 12 years ago

Link-shorteners are a cancer. We have enough of a problem with historical links going dead, without interposing a third party.

When bit.ly goes down, what happens to all their links?

helium• 12 years ago

By that rationale, any website is a cancer.

barsonme• 12 years ago

True, but for example if I'm posting an article I wrote it's much more efficient to save space using my erclgrgn.us short link because it's usually ~20 chars which is less than half of what the full URL is.

Unfortunately, sometimes brevity is a necessity.

edit: original comment mentioned twitter, then I remembered twitter automatically reduces a URL's chars to 20 or so.

ForHackernews2• 12 years ago

Yeah, or websites could just allow actual links. I don't see why hackernews (and twitter, etc.) can't just render <a href="http://somereallylongdomain.com/stupidlylongpath/?what=thisi... short!</a>

tedd4u• 12 years ago

For security stuff, if you can't find a well-tested implementation to re-use, I always suggest referring to the OWASP Cheat Series [1] - there's a nice one just for password storage [2].

[1] https://www.owasp.org/index.php/OWASP_Cheat_Sheet_Series [2] https://www.owasp.org/index.php/Password_Storage_Cheat_Sheet

Bit.ly compromised – More Details

Related Stories