Former Hostgator employee arrested, charged with rooting 2,700 servers

http://arstechnica.com/security/2013/04/former-employee-arrested-charged-with-rooting-2700-hostgator-servers/

by benhomie • 13 years ago

113 59 13 years ago

ghshephard3• 13 years ago

This all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse."

Not something I'd want on my personal system, but it's exactly the sort of thing that I think every NOC/Secure environment should have for post-mortem assessments.

freehunter2• 13 years ago

It's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.

jonknee• 13 years ago

Expensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double or more for your cash.

As for processing, you need to find a new computer if you can notice a screenshot being taken.

gav• 13 years ago

A previous employeer used Spector 360[1] on the majority of workstations. It would monitor everything including taking a screengrab every 5 seconds that you could then watch later.

They'd sit down employees and playback fast-forwarded video showing how much time was wasted on Facebook, personal email, shopping, etc. It's horribly invasive but it meant everyone was too scared to use work computers for personal things.

[1] http://www.spector360.com/

nephesh• 13 years ago

I'm sure productivity skyrocketed /s

venomsnake• 13 years ago

It would be fun to see the people's productivity on that system. And who of the tech people will try to break it as a sport.

hfsktr• 13 years ago

My old job used spector. I'm pretty sure everyone knew it but people would still be on FB playing games when you walked past. I believe IT was the last to get it and for about a week after they installed it on our machines it didn't work because Microsoft Essentials disabled it as malware.

I don't think they ever looked at it unless they wanted to fire someone and didn't want to pay unemployment and needed proof that they weren't doing their job.

tekacs• 13 years ago

Given the way that hovers over employees, I think there's a fair case in there to unofficially rename the product 'Spectre'. -.-'

shudder

davidandgoliath2• 13 years ago

Not nearly as exciting as the microphones they had installed all around the building to monitor discussion amongst teammates. :)

homosaur2• 13 years ago

I don't think this is appropriate for general office use simply because you might leach private data from employees (bank accounts, retirement accounts, medical records, whatever), but in secure, restricted environments I think it's totally appropriate and probably needed.

pc862• 13 years ago

There is an argument to made that you shouldn't be looking at your personal bank accounts or medical records on a work computer, particularly if you work at a company that cares so little about their employees that they'd implement a system like this.

lawnchair_larry3• 13 years ago

I'll never work for any employer who does that, so I'd have trouble calling it a reasonable thing to do.

davorak• 13 years ago

What alternative do you think is reasonable to help track down problems like this after the fact?

andrewljohnson• 13 years ago

Any powerful IT guy should be monitored and have his power checked. IT personnel have almost unmatched power in an organization to cause damage without detection.

Anything with lots of confidential information, or anything financial, and you are going to want to monitor all the people with access constantly. You may not want to snoop real-time, but you are going to want to be able to find and fix breaches after the fact, and do root-cause analysis.

It's not a matter of trust in the IT people, it's a matter of people go crazy sometimes, and people make bad hiring decisions sometimes.

ghshephard3• 13 years ago

The areas where it would be important are on security/NOC systems, where there is almost unlimited power given the proper credentials.

The Runbooks that NOC teams have, quite often have them connecting to a lot of systems with greatly heightened privileges - It's not unusual for a NOC employee to have expansive sudo privileges on many of the unix hosts they manage. They are also often on privileged VLANs, with direct IP routing to a lot of hosts that normally wouldn't be reachable.

Most of our NOC guys have their own personal laptops, and they can hop onto the (unprivileged) wireless system and do their own thing when they aren't working an incident.

I'd have no problem having my screen captured once a minute when I was working in that type of environment.

incision• 13 years ago

>Not something I'd want on my personal system, but it's exactly the sort of thing that I think every NOC/Secure environment should have for post-mortem assessments.

Tools of that sort [1][2] are pretty standard in call center environments.

I'm not a fan.

First, the software tends to be incredibly expensive. Second, in my experience, it's primarily used by managers looking for reasons to bludgeon their $30K, entry-level call takers over trivial infractions.

1: http://www.nice.com/contact-center-interaction-recording

2: http://www.callcopy.com/products/screen-capture

jamescun• 13 years ago

> While his root access gave Gisse access to private data stored on a large number of customer websites, there's no evidence he used it, the Hostgator executive said.

I think the article is quick to jump to the conclusion that he was attempting to be malicious with his actions however this could be a case of Hanlon's razor.

His actions could easily be attributed to a less-than-aware sysadmin developing his own solution to get around often arduous security restrictions. Stupid, yes. Malicious, no.

gvb• 13 years ago

He attempted to access the HostGator from outside computers: Hetzner Data Center in Nuremberg, Germany, and efnet.pe (Peru) are mentioned in the article. The Hetzner access was the day after he was dismissed.

Malicious, yes. Stupid, absolutely.

tracker1• 13 years ago

If he hasn't accessed any of these systems since he was terminated, he could state that it was for "emergency" access to remote systems upon other compromises. Since most of these systems are likely headless, then remote access is the only way to get in. A lot of remote exploits will nuke SSH, and other access tools, so having a "backdoor" is often a good idea.

That said, it's still likely that this guy is just a douche with a bad attitude, and deserves everything he has coming. Big difference between this, and "stealing" a bunch of reports that were government funded, and open to any and all users on the school network they were accessed from.

homosaur2• 13 years ago

I don't think this would work. Installing your own solution on customer facing hardware? Might get you out of prosecution, but I think you'd have a nearly impossible time explaining that.

roel_v• 13 years ago

%| You serious? A dude backdooring several system utilities, giving him access to 2700 customers' data and applications, and you're defending him with 'it's just all a mistake' nonsense?

dotmanish• 13 years ago

Did these 2700 servers play a role in any DDoS attacks as well?

It would be quite a lucrative stance for the employee to sell access to these servers to one or more groups who could potentially make more use of them.

davidandgoliath2• 13 years ago

Doubt it, based on the timeline. Doesn't sound like he accessed the systems at all beyond implementing the rootkit and/or patching things up.

cperciva2• 13 years ago

2700 servers all on the same network makes for far less of a DDoS attack then 2700 similar servers on different networks - and it's far easier to detect and block too.

They would be more valuable for bitcoin mining most likely.

nwh2• 13 years ago

2700 servers probably wouldn't be worth much for that. It would be noticed fairly quickly, given that only CPU mining would be available, and how monitored servers usually are.

ultimoo2• 13 years ago

Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.

w1ntermute• 13 years ago

Reminds me of Ken Thompson's Reflections on Trusting Trust: http://cm.bell-labs.com/who/ken/trust.html

saurik3• 13 years ago

While this isn't guaranteed (all tools, including the compiler, may be patched), you can use checks and balances: verify /proc doesn't contain phantom processes, compile your own copy of ps, try more-obscure tools like top. If by "understand" you just mean "notice"... well, you don't, until one day you accidentally stumble across one of the above and start digging. (Maybe, for example, you install some kind of server monitoring tool, and when you log in to the web portal it provides you see a process that you find very suspicious; when you use ps, it doesn't show.) In my case, I've noticed this kind of thing twice: once, when the tool was binary pacthed to death (and just crashed), and once when the "patch" was "replace binary entirely", and the replacement was older and did not support a command line argument I knew that it should.

ultimoo2• 13 years ago

Nice. So both times that you noticed this, was it malice on someone's part due to which the tools were patched? I have just never heard or encountered such a situation and am frankly paranoid about something like this happening to one of the tools I use.

saurik3• 13 years ago

(When I said "more obscure tools like top" I meant to say "pstree" but edited the statement one too many times before posting.)

badgar• 13 years ago

He had root, so he could have instead installed a rootkit, which can hide the existence of processes from all of userland. In a graduate OS class I took, we had an assignment to do hide a process live on OpenSolaris 8 using the kernel debugger (kdb). I wrote some assembly and overwrote some bytes in the syscall functions for process listing. We were on developer builds so you could just use the function symbols by name in kdb. I forgot to cover /proc/ though.

__david__• 13 years ago

> I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'.

So this happened to me once, but it wasn't ps or netstat but sshd that had been replaced. We only noticed it because the replacement sshd was bad and didn't set up the PATH correctly so "darcs push" stopped working (it was presumably set up for a different linux distro than we were running).

drzaiusapelord• 13 years ago

Ideally, you should be running some kind of tripwire scan. If the hash for common utilities changes and you didn't update those binaries, then something bad has happened.

Hacked versions of common utilities is a common payload for rootkits.

shuzchen• 13 years ago

That would be helpful for rootkits coming from outsiders, but would only serve to slow down (not stop) an insider. An insider knows what protections are in place (probably implemented them too) and can defeat the hash check if she knew how the hash was calculated, or can ship the binary alongside a regular update.

tptacek• 13 years ago

If you lose superuser, you've almost certainly lost your kernel, at which point your only hope for Tripwire-type scans actually working is an attacker that doesn't know where to download a good rootkit from.

You should still run the scans, just be aware of the limitations.

ttrreeww2• 13 years ago

Amateur hour, quite a few rootkits out there are almost completely undetectable...

lawnchair_larry3• 13 years ago

Really? Can you name one?

MertsA• 13 years ago

If it's a Redhat based server just run rpm -Va and that will verify the integrity of every rpm package on the system. If you're in a situation where you can't trust the current computer you can run it in a chroot from a known good copy.

bigiain2• 13 years ago

It's been some time now, but I've been there...

Once you've lost root to a sufficiently competent attacker you can't trust _anything_ on that box any more.

One thing that'll help against (some) script kiddies with rootkits is to have available statically linked copies of every tool you might want to use to see what's happening on your box - for a long while, every colo-ed box I managed had a cdrom drive with read-only versions of /bin, /sbin, and useful bits of /usr - all with the binaries statically linked. They can in handy a few times (mostly to confirm that "yep, we're screwed. Get this box off the network and powered down immediately and implement the bring-up-a-new-server-from-scratch plan right now".

At some stage though, you can't trust the kernel or the hardware - an attacker who's got into your booted kernel or your bios or your network card firmware, if they're good enough, they probably can't be detected by examining anything you could see logged into the box itself. The only way to identify that level of attack is by monitoring the traffic from the box from some trusted piece of network gear upstream of your rooted server (and against a sufficiently talented attacker, even identifying unexpected outbound traffic might be impossible. If your list of likely attackers includes three letter agencies or nation states, I hope your getting your secutiry advice from somethere other than HN comments…)

trotsky2• 13 years ago

Read only media is good advice and low effort compared to most viable defense. Certainly valuable at times worth doing. Starting from the assumption you'll get beaten typically pays off over perfecting the hack-proof nirvana.

Nobody should lose any sleep about BIOS embedding and similar - that level of attacker and sponsor imply a level of threat that no typical organization has a chance against.

In my opinion, after years of pondering dozens of intrusions with many types of ways in and regular failure of all kinds of defenses I don't think there is much advice to give aside from the flaw is in your custom software, stupid.

I have become a really big advocate of CM and push button provisioning with identical replacement hosts that build from scratch and commonly get refreshed - relying on code and configurations managed centrally.

The best way to remove an attacker is a complete rebuild. If you're already using Chef etc why not just dump them proactively? Some roles don't lend themselves to this, but I assure you it is like a massive weight has been lifted.

efnx• 13 years ago

Would be way cooler if he rooted 2600 servers ;)

Shadow_Death2• 13 years ago

I'm not surprised with that company actually, hostgator is a joke and the employees in Austin, Texas are too. I think the one thing that bothers me how did he get the SSH key? The fact that he had it tells me that someone higher up dropped the ball somewhere.

aptwebapps• 13 years ago

"Gisse didn't return a voicemail and e-mail seeking comment for this report. A Court docket shows he is scheduled to be arraigned next month and gives no indication he has entered a plea in the case. He's being held at the Harris County Jail on $20,000 bond, a spokeswoman at the district attorney's office said."

Wait, wait, just because the guy's in jail is no reason not to return voicemail and emails!

nnq• 13 years ago

This smells really funny. They could have buried this instead of going to court (with a 1+ year delay!) and committing PR seppuku by making this public and giving their clients a reason of distrust. Now, this is indeed the right thing to do, the guy shouldn't go unpunished and they should disclose their security breach, but if they are doing it for the "right" reasons, why is it 1 year later?

mikeurbanski• 13 years ago

Sounds like he knows Linux...

Shadow_Death2• 13 years ago

It's funny you say that because I remember their add. I made a jab at the company and - 2 rep.. If they knew the company they would have +2 rep me haha

d23• 13 years ago

> "He did not access customer content," Pelanne told Ars. "We caught it well before he had any chance to do any of that."

> Given the rapid discovery, the malware was on Hostgator systems for less than a month.

Then yes, he did. If the malware was on there for more than a few days, I find it extremely unlikely that at least some data wasn't compromised.

Former Hostgator employee arrested, charged with rooting 2,700 servers

Related Stories