We got hacked

http://www.plumbr.eu/blog/we-got-hacked

by codegen • 13 years ago

70 21 13 years ago

plumbr.eu

VaucGiaps• 13 years ago

Clear and open about it. Good.

priitp• 13 years ago

Asked my team to review their Jenkins passwords and Jenkins user rights...

reginaldo2• 13 years ago

I don't know if you're doing this, but I think it's a bad idea to leave Jenkins publicly accessible. Indeed, IMHO, it's a bad idea to leave stuff that should not be accessible by the general public publicly accessible. Especially things that have access to your code.

Do ask your team to review passwords and user rights, but also put this service and others like it behind a VPN. Then both the VPN server and Jenkins will have to have holes simultaneously before you get hacked.

Nikem• 13 years ago

As the main hero of that story, I can assure you that we are working on VPN setup right now. I guess this is not in top-10 checklist for a startup.

skrebbel• 13 years ago

Should've coded the whole thing in Makumba and this would've never happened!!one

waitwhat7• 13 years ago

Was the original attack via jenkins? all it says some vague privilege escalation was used to upload c file. what?

reginaldo2• 13 years ago

I believe the issue was related to the fact that the user running Jenkins was a full passwordless sudo user.

Maybe the attacker used the groovy console too...

ihsw• 13 years ago

No the author seems to indicate that it was on their application code and an attacker was able to get OS access, and the attacker subsequently replaced the ssh service with one that instead ignores login attempts and harvests the username/password pairs.

It seems to have been a coincidence that their Jenkins service was not secured.

OptimusSubprime• 13 years ago

This whole post reads like "How Not to Use AWS" instruction guide.

If you're running anything important on a single EC2 instance, you're doing it wrong. If you're logging in and manually configuring an EC2 instance, you're also doing it wrong.

alan_cx• 13 years ago

That's helpful, cheers.

How about letting us lesser plebs know how to do it properly then?

zacharydanger• 13 years ago

This is probably beyond the scope of such a thread, but configuration automation via Chef or Puppet is where you'd start.

bmelton• 13 years ago

Puppet and Chef are great tools, but my first entry into AWS management was a little more attainable without having to learn anything new.

I simply set up the instance to automatically download the current codebase from our 'production' repo on every boot, automatically install requirements, run database migrations, etc. Then freeze that AMI as the production AMI.

If you migrate your databases off of the instance to RDS, don't use EBS, and manage user uploads and static assets to go straight to S3, then what you end up with a very easily load balanceable configuration.

To set up load balancing, just spin up two of those instances, attach them to an elastic load balancer, attach the load balancer to your elastic IP, then you ought to be more redundant and management free. (Note, this doesn't fix your security issues, though you could very easily bake a nightly apt-get run for security updates into that image).

After that, you want to look into Fabric (or language-specific alternatives if they exist) to allow for remote management of your machines for things like bouncing web services, pulling new code deploys, etc.

If you want to sidestep all this management altogether, I'd strongly suggest looking into something like dotcloud - http://dotcloud.com/ - which effectively does all this for you.

ajross• 13 years ago

That may be true from a scalability and maintenance perspective. I don't see how those criticisms relate to security. Automatically-maintained hosts are no less subject to exploit...

krobertson• 13 years ago

"my fellow co-founder who just happens to have a lot more experience in the field of hacking and getting hacked"

I see irony in the "getting hacked" part. But seriously, if he is more experienced, then why do they have machines unpatched like that for 6 months?

"it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks"

It is rather bold to be posting about how you just got hacked by not applying simple patches, and to then bash the "kid" who couldn't install a proper rootkit.

ivom2gi• 13 years ago

That is indeed the case - both players in this match were not careful. We are open about our shortcomings and also think that the other player was not too careful to say the least ...

OGC• 13 years ago

So... if the attacker would not have had deactivated SSH by accident they would not have noticed?

scoot• 13 years ago

no customer data was exposed during the attack

How did you confirm that?

jlgaddis• 13 years ago

That's a good question. Given some of the "fundamentals" of security that were obviously ignored, I'm reluctant to believe that they can make that assertion with any confidence.

ladzoppelin• 13 years ago

"C source named backdoor.h" Why would an attacker leave a file with that name? Seems like misdirection or a mistake. How are you going to verify they did nothing else?

drivebyacct2• 13 years ago

First of all, your stupid copy/paste script has completely broken copy-paste. When I copy, I get nothing except the spam you're trying to inject, so congrats on managing to piss me off right off the bat.

Second, why even list A, B, or C? Whoever thought running Jenkins as a passwordless sudo user shouldn't be doing sysops. Why was Jenkins even public facing? At worst, put it behind a VPN.

We got hacked

Related Stories