When a bad day gets worse – getting hacked twice in one day

http://chrishateswriting.com/post/84931829578/when-a-bad-day-gets-worse-getting-hacked-twice-in-one

by killwhitey • 12 years ago

264 81 12 years ago

danso• 12 years ago

The disclosure of these mistakes is both kind of relieving, in that the errors were obvious in nature, and terrifying, in that even though they seem "basic", they were still committed by a dev as experienced as moot. This would be a good post to create a checklist from, if nothing else.

I wonder how old the code was for "Mistake 2"? SQL injection is something most sites have patterns/frameworks to prevent, but unless the site started out with such practices...the code that was written when the site was just a fun side-project might go unchecked even as the site becomes a well-run project in its later years.

moot5• 12 years ago

they were still committed by a dev as experienced as moot

I'm not a terribly good programmer, and have been very hands-off with 4chan's code for quite some time. I still direct development and am responsible for the servers/sysadmin tasks, but there are far more talented developers out there than I. In the case of Canvas/DrawQuest, I was 100% uninvolved on the tech side.

But again, in both cases I accept full responsibility for the breaches since ultimately it's up to the project leader to ensure these things don't happen -- even if not active on the technical side.

I wonder how old the code was for "Mistake 2"?

Very new. It was in a once-off file that we used to quickly pull stats about reported posts, which a) shouldn't have been on a domain without HTTP auth, b) should have been deleted long ago, c) shouldn't have had a bugged auth check or injection vuln to begin with.

noir_lord2• 12 years ago

one-off files have a habit of hanging around and temporary often becomes permanent.

This isn't to say that you should treat all one-off's and temporary solutions as permanenent but it is a good idea to audit them periodically.

Storing that kind of metadata about code is something I've often pondered we could do better, putting it in comments is a nasty hack, storing it away from the code means it instantly gets out of date, commit messages are not a good place to put that stuff either.

I've never come up with an elegant solution even in my head but it would be something I'd love to have for my own uses.

tudborg2• 12 years ago

How do you query your database? There is still a bunch of new PHP projects that use mysqli::query, which is like playing with fire compared to using PDO::prepare If you are still using mysqli, changing to PDO for future commits might help reduce chance of creating these kind of vulnerabilities.

ufmace2• 12 years ago

This makes me wonder what kind of practice change would be best to help avoid this sort of thing. Maybe anytime you put in something that you know is hacky and insecure as a one-off, you should put a note in wherever you would look regularly, and maybe some kind of reminder too, to make sure that your one-off is either removed or properly secured reasonably quickly.

Or maybe just a rant on PHP and/or frameworks that make it easy to do the wrong thing, and hard or time-consuming to do the right thing.

jaryd• 12 years ago

Looks like a good opportunity for an open source tool that scans a git repository for interesting/private information. Obviously it could be used for nefarious purposes, but it would be helpful for anyone looking to open source an existing repo.

Another alternative is to store credentials in environment variables :)

moot5• 12 years ago

Looks like a good opportunity for an open source tool that scans a git repository for interesting/private information.

These already exist in droves, which is exactly how our Amazon credentials were found.

http://www.itnews.com.au/News/375785,aws-urges-developers-to...

ufmace2• 12 years ago

Probably a good practice. For my current project, I've been careful to never commit any API credentials or other secrets to the repo, even though I don't currently have any intention of making it public. It's harder to know what you're missing, though.

yeukhon• 12 years ago

Time for this question to come back live.

http://security.stackexchange.com/questions/56911/does-anyon...

Anyone has any idea about this website I am trying to find?

cycojesus• 12 years ago

How about a commit hook that warns when it looks like you're about to commit a secret key?

moot5• 12 years ago

Sounds like a good plan to me!

Mikushi• 12 years ago

Did something similar, just built a module for our CI tool that checks against sensitive information (secret, salts, hashs, hidden feature), pretty efficient as every dev (myself included) can get lazy and use keys directly in scripts, instead of our Config loader that safely retrieves all that.

Not bad in itself, sometimes all you need is a dirty script, but as other have said, they tend to stick around, a key part of our CI module is actually putting expire date on scripts, or check dates. I will receive an email telling me to check and a warning on the CI at build time. It's been great to keep our code clean while allowing us to still do things dirty when required.

knowtheory• 12 years ago

Having been through open sourcing an app platform repo (https://github.com/documentcloud/documentcloud ) that wasn't explicitly created w/ open sourcing in mind, moot has my deepest sympathies.

Moot left out the most sensible response to Mistake #5:

I went through a rushed 8hr rebase to remove all of our keys, certificates, cookie secrets and the like before DocumentCloud's platform was opened. Two days after, a friend asked me why I hadn't simply revoked and replaced all of our secrets to the sound of me facepalming repeatedly.

It is always much easier just to revoke and replace your secrets. Doing so should always be relatively easy for you to execute, and having a chance to practice that is also always good.

moot5• 12 years ago

Good point -- frankly I'd suggest doing both (clean repo and revoke/replace the secrets). The developer in question simply overlooked the issue, and since we were busy spinning down the company, nobody else caught it.

stouset• 12 years ago

Also, just swear of ever committing secrets to a repo to begin with. It's a bad idea, period.

johnnymonster3• 12 years ago

Its a very good point that if you are not able to easily revert and change all your secrets and key's then you are in more trouble than just removing them from the commit log.

BoppreH• 12 years ago

I would scrub the repo anyways because revoked secrets can still be used for social engineering. "Hey $HOST, I got hacked. My old password was $PASSWD and you can be sure it's me because $PUBLIC_INFORMATION. Can you send a password reset to $ATTACKER_EMAIL?"

coldcode• 12 years ago

Shows you that strict controls over your AWS credentials are worth whatever hassle they might be. 100 huge instances spun up can cost you big time.

vacri• 12 years ago

AWS also lets you put in billing alerts, letting you know when you hit the threshold.

willvarfar• 12 years ago

(Not an AWS user, so don't know how this works)

Can someone with the leaked credentials turn off the billing alerts?

matthuggins• 12 years ago

I'm not familiar with DrawQuest, but why not sell it "as is" instead of shutting it down completely? Especially when you have 100k monthly actives?

adventured• 12 years ago

If I'm not mistaken, they took on $3.6 million in venture capital, and that might be the complication in deciding exactly what to do with the product (that is, their partners in it may disagree with selling it for $500k or whatever it might fetch).

moot5• 12 years ago

Our investors have been incredibly supportive -- especially these past few [rough] months.

The biggest concern I have is not so much the price it would fetch (which is probably not much given recent events), but rather finding a suitable steward for the community.

LukeB_UK• 12 years ago

Thanks for this Moot, I love the fact that people are being more and more open with how they were hacked. These post-mortems help others identify areas where they may be vulnerable which in turn helps the community as a whole.

Sorry about you getting hacked twice in one day though. I hope things have been better since then!

loceng3• 12 years ago

So simple. Could have been much more painful.

loceng3• 12 years ago

I think people who are downvoting this comment are projecting what I meant by "so simple." There was no judgement on that statement. It was more an observation relating to that it could happen to anyone, and that they were "lucky" the person who got into their system wasn't really malicious.

pbhjpbhj• 12 years ago

IMO they more likely consider your initial comment to add nothing to the thread.

SDGT2• 12 years ago

Boneheaded cookie auth—we simply stored the bcrypted password from the database in a cookie, which was all that was required to pass PHP auth.

Wh... Why... Who...

How was that ever OK?

oldmanjay• 12 years ago

Okay, so I'm guessing this is an example of "feigned surprise" that is supposed to be considered bad?

I've been on a few sites (very recently) that just stick the password right in an auth cookie, served via HTTP no less. Luckily they're sweet enough to base64 encode it for extra security.

SDGT2• 12 years ago

Not even feigned.

I would have expected a site like 4chan, the source of all sorts of script kiddie idiocy, would know better by this time than to stick a password, even hashed, in a cookie.

But sometimes you piss off the HN bees nest, and downvotes come streaming in. Reminds me of Reddit.

rushonerok• 12 years ago

Putting the hashed password directly in the cookie? That's shameful. A hack was well deserved.

jebblue• 12 years ago

> A hack was well deserved.

That's a despicable thing to say. No one deserves to have bad things thrust upon them.

primitivesuave• 12 years ago

Some security bad practices, but it happens to the best of us. Glad you got that figured out though.

I always tell my devs on the first day, "leaving AWS keys in a public repository is as bad as showing up drunk to work". Mainly because the last guy to leave our AWS keys in the open was drunk at work.

johnnymonster3• 12 years ago

its really sad to see how many AWS Keys are public in github right now. A quick github search reveals many, many key pairs that were checking as recently as this week.

tommu• 12 years ago

Amazon are apparently scanning themselves. If they are doing this then they might figure that the quantity of exposed keys will undermine their reputation. That is quite something in itself.

dangoldin• 12 years ago

I'm guilty of this for an old project and only discovered when Amazon sent me email telling me it was public so Amazon seems to be searching Github (the web?) for mentions of the keys.

driverdan• 12 years ago

The list of vulnerabilities in 4chan read like a standard web hack CTF. I'm surprised a site as big as 4chan has multiple OWASP top 10 issues, especially after the source was leaked. Storing credentials in client side cookies? Who wrote that shit?

redthrowaway• 12 years ago

4chan has always operated on a shoestring budget. When you have a single dev, there's nobody around to catch stupid mistakes. I just spent two days trying to track down a bug only to realize I'd been adding elements to an array in a braindead manner. Stupid happens.

Kiro• 12 years ago

So Mistake #4 is something you hear over and over again is bad but I don't understand why it's so bad. If you have the password you can just login anyway and I don't see how it affected this particular case either. Please enlighten me.

mpeg• 12 years ago

Because he didn't have the password, he had a hash of the password.

So if the auth system wouldn't have let him login directly with the hash by changing his cookie, he wouldn't have been able to login as an admin.

userbinator3• 12 years ago

Am I right to say this would've been easily prevented by putting the hash(salt + password + client IP) in the cookie, instead of just hash(password)?

pearjuice• 12 years ago

moot, why do you refuse to speak to us on 4chan while go happy hand-in-hand with all these web 3.5 cloud entrepreneurs here? Want to find some investor friends? Thanks for ignoring your actual friends.

hex12648430• 12 years ago

Because of inane comments.

TreeTrunks• 12 years ago

He can't get upvotes on 4chan.

nsmartt• 12 years ago

I strongly suspect moot posts anonymously on 4chan. Posting under moot would result in his dominating the conversation.

duvet• 12 years ago

Using someone's site does not make you their friend. That remains true even when you both enjoy some aspects of the culture. That said, his threads on 4chan are probably exhausting. It isn't like he's keeping anyone in the dark, either.

zalzane• 12 years ago

im pretty sure the point of posting on an anonymous imageboard is to post anonymously

elithrar• 12 years ago

"Mistake #3: Unescaped SQL query..."

This should be "failure to use parameterised queries." The term 'unescaped' may lead some to think that simply escaping characters is sufficient.

annnnd• 12 years ago

Exactly!

The problem is not that this query was not escaped, the problem was that: 1) unparametrized queries are allowed in code, 2) user input is obviously not properly checked in all cases and 3) that WAF (Web Application Firewall) was not used / set up properly.

Other than that, thanks to OP for in-depth writeup - it helps to be reminded from time to time why security matters.

userbinator3• 12 years ago

Are you implying that there's a way to get past escaping...? That's a new one to me. I thought escaping was precisely so you can put any value in a string and have it still work.

mnw21cam3• 12 years ago

Yes, but you had better be 100% sure that your escaping function is completely reliable, and the server hasn't introduced some new syntax since you wrote it that you aren't escaping properly.

I would trust parameters much more (although I have used proper escaping in the past).

astrodust2• 12 years ago

Not using placeholders should be an automatic fail of any code review. Period.

Without placeholders the risk is way, way too high. A single mistake can be the end of your entire application if not career.

Don't think "Oh, this is just test code, I'll fix it later" and leave a vulnerability in your application. Always write using placeholders. No exceptions.

dkarapetyan• 12 years ago

Security is a usability nightmare. All the instinctive and usually lazy things people do are just bombs waiting to explode. I don't know why this continues to be the case. Most platforms for web development should have taint checking turned on to the highest setting by default. SQL access points should scream in your face every time you use a bare query. With a bit more you could even get developers to do the right thing in terms of salts, passwords, and hashes. I think it is clear that pointing people to OWASP and asking them to read their security articles before bedtime is not as useful as it could be. Library writers also need to be involved and use safe options by default instead of opt-in.

riquito• 12 years ago

Nobody mentioned the fact that _you don't put secrets in a versioning system_. .gitignore the configuration files and have example files available, or use configuration files that can import the missing parts (the secrets) from an ignored file. Another way is to put the secrets in environment variables outside the project. In any way, if they're missing provide a meaningful message to the developer. I do understand that you must keep some of these secrets available somewhere: just don't version them.

ansible• 12 years ago

I can understand the desire to version control your secrets.

The bad idea was not to maintain the source in a separate repo that could easily be released.

If you want to keep the checkout process simple for your developers, maybe use git subtree for the overall project.

patio11• 12 years ago

Of related interest to HNers: Admin consoles getting owned up is a fairly frequent way that even security-conscious shops lose their web apps / databases / networks, because one often does not code defensively against malicious actions executed by "trusted" users. You can mitigate this quite a bit by putting them on a private network interface and requiring the admins VPN in to use them.

StavrosK3• 12 years ago

As an easier alternative to set up, you can require SSL client certificates.

selmnoo• 12 years ago

Speaking as someone who got owned even when using SSL client certificates (made mistakes setting things up, long story), I think patio11's suggestion is better. Put it on a private network, seriously it isn't that hard.

nicpottier• 12 years ago

I actually think this is something GitHub should help with. When you flip a repo from private to public, it should prompt you asking if you want to wipe out all the version history first.(essentially create a brand new repo in the background)

Seems like far too common a mistake and something they could help fix once and for all.

When a bad day gets worse – getting hacked twice in one day

Related Stories