Verelox Wiped by Ex-Admin

http://www.verelox.com/

by jonmarkgo • 9 years ago

241 380 9 years ago

verelox.com

ceejayoz• 9 years ago

Yooooowch. They appear to be VPS and dedicated host.

rexicus• 9 years ago

There goes the 99.95% uptime guarantee.

oneplane2• 9 years ago

In theory that still gives you a lot of downtime in a 365-day span :p Then again, once you are down for more than 4 hours (as they are), that uptime starts ticking away rather fast.

preinheimer• 9 years ago

Wayback Machine link if you want to know who they were:

https://web.archive.org/web/20170603212121/https://verelox.c...

michaelbuckbee• 9 years ago

Wow, they were a VPS hosting provider. Lots of unhappy customers (and this likely qualifies as a data breach for all of them as well).

__jal3• 9 years ago

Yeah, that could become legally awful. Which is appropriate; it appears $admin ruined a lot of people's days.

I suspect that person will soon have a fair amount of time to decide on what new career to pursue to pay down the fines when they get out...

3dm0nd• 9 years ago

They should be able to use https://github.com/hartator/wayback-machine-downloader and get at least a static version of the website back online.

DKnoll• 9 years ago

If a VPS provider didn't manage to have even an offline backup of their own website they should dissolve the company.

tanilama• 9 years ago

Damn, they are fucked. How could people trust them any more???

Sir_Substance• 9 years ago

Dick move from the ex-admin, but I'm curious to know what would compel an ex-employee to take such a brazenly criminal and traceable yet damaging action.

I'd like to know more, I think...

ktta• 9 years ago

I think this one would qualify as a criminal move, not just a dick move

mikestew• 9 years ago

Well, I’m guessing it starts with rationalization of why it is not criminal, and how you’re so smart that it won’t be traceable. Wrong on both counts, usually, but that’s my guess.

jacquesm4• 9 years ago

There is no excuse for it, no matter what the company did this affects the customers of the company just as hard as it affects the company. Likely some of those will go out of business, people will lose their jobs, go bankrupt.

treyfitty7• 9 years ago

So do whistle blowers. That line of reasoning doesn't set the bar for what's right and wrong.

Are you sure no matter what the company did? What if the CEO threatened the ex-admins family? Or if the ex-admin found child porn on the CEOs computer?

There's a fine line between right and wrong in most situations. The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.

idealogue4• 9 years ago

That's the smallest thing I've ever seen "too big to fail" applied to.

zokier2• 9 years ago

Wiped by ex-admin, or by the ineptitude of current admins that can't maintain proper exit procedure?

Kenji• 9 years ago

Who says he used his credentials and didn't leave and use a backdoor?

tqkxzugoaupvwqr• 9 years ago

The current admins ineptitude did not wipe anything. Yes, with better procedures this could possibly have been prevented, but the current admins are not the malicious party.

soared2• 9 years ago

My most favorite of all razors is Hanlon's: "do not attribute to malice what can be explained by stupidity"

__jal3• 9 years ago

Well, for one thing, if you can't ensure that a given privileged person can be locked out of your systems quickly, you have a problem to fix. This should be a 1-minute operation.

Note that I don't know the details and am making assumptions that may be wrong about the case in question, but in general, if you can't deny access quickly to any given account, you really want to fix that. Not just because of rogue ex-employees - what happens when $important_person's account is compromised?

ascendantlogic• 9 years ago

Ok, I'll bite. If you are an admin generally you have God powers over everything within your umbrella. If things are going south with you and your employer and you're a giant asshole you can most certainly use your God powers to quietly open firewall ports, install software in dusty "corners" of networks, etc. We can all armchair quarterback this but in the end humans make mistakes and beyond that the presence of actual malicious intent by someone with keys to the whole kingdom makes the detection and/or mitigation difficulty factor go up significantly. It's real easy to make righteous proclamations about other people's misfortune when it's not you.

zokier2• 9 years ago

I'll admit, my tone might have been tad overly poisonous (too much internet for me..).

I'd agree that defending against malicious admins is really difficult. We have really little context to go by here, but I think there is important distinction to be made if the malicious actions (planting backdoors or whatnot) were done while the malicious actor was still employed or after their employment was terminated. Proper exit procedures protect against the latter, but generally are not that effective against the former.

but in the end humans make mistakes

And it is useful for us outsiders to highlight the real mistakes so that we can learn from them, because that is really the biggest value of stories like this for the majority of people who are not directly impacted.

canada_dry• 9 years ago

Exactly. Happened to my shop on my watch. Fired a contract admin... about a week later the network went to hell. Unfortunately I couldn't nail the bastard, but at least we found and closed the back door.

hamstercat• 9 years ago

Not following/having a proper exit procedure for employees with a lot of access is being negligent. Wiping servers of your ex-employer out of spite is criminal (and being pretty bad at that whole being a human thing).

cwyers• 9 years ago

Was he an ex-admin at the time he did this? It's not made clear here.

wilhil• 9 years ago

Other than treating staff well, how would you go about stopping something like this?

As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.

dmuth• 9 years ago

Since the title says "Ex-Admin", revoking credentials when someone leaves the company is a must. If there's centralized auth (AD), this is pretty straightforward. If not, then at the very least it makes sense to use Puppet, Ansible, etc. to lock that person's account on all production servers.

Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)

tooltalk• 9 years ago

I bet the ex-admin was also responsible for keeping backups and auditing/logging all access, in addition to managing pretty much everything else.

swozey• 9 years ago

There really should be more systems that require 2 keys to do things like delete servers, load balancers, etc. I'm not aware of any. It's crazy to think that if an admins AWS/GCP/Azure account is compromised (or the cloud provider is compromised) a few commands/clicks to an API server could delete everything.

benjohnson2• 9 years ago

We swiped an idea from launching Nuclear Missiles: Typically you need two keys from two people to launch.

For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage.

For example if the root password was CatDogFish then

Person1: CatDog_

Person2: _DogFish

Person3: Cat_Fish

Two people can then log in and watch what the other person is doing.

jasonjayr• 9 years ago

This is like an informal version of Shamir's Secret Sharing

https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

rcthompson2• 9 years ago

Maybe someone should implement a PAM module or something that requires authenticating as two different users with sudo privileges, so you can implement this for as many users as you like and let them each have a single password, instead of half of a password for each possible pair of users.

ars4• 9 years ago

That only sort of helps. Do both people also watch each other while they run commands?

Because if not, once you are admin, you can install programs that let you become admin again at will.

shif2• 9 years ago

What if one of the people that has part of the key gets mad and leaves without sharing it?

MichaelGG2• 9 years ago

So one person runs a keylogger and captures the full credentials?

But it's not a bad system.

ams61102• 9 years ago

Well whoever set that up knows the whole password right?

ugh1232• 9 years ago

If this was truly a former employee (wasn't made 'former' after the incident occurred) then all of their accounts should have been blocked from having access. Unless the guy built a back-door or something. IT and Network team may have dropped the ball on that one.

JustSomeNobody• 9 years ago

As crappy a job as it is to think about, start planning now. Have a checklist of everything everyone has access to. Have a procedure in place for what happens upon termination.

I worked retail to pay for college. Could always tell when a manager was getting the boot; they'd order new cylinders for all the doors. You kind of have to have that plan in place in IT too.

superflyguy2• 9 years ago

It has parallels everywhere. How can you stop people poisoning food in a shop; walking into a school and stabbing a bunch of young kids; irradiating coins; deliberately driving cars into people; spreading sexual diseases; closing organisations with bomb threats etc. There's plenty of things you could do with practically zero chance of getting caught (and not everyone is worried about that) and for a low cost. You can't run a society that has a 100% success rate of preventing these sorts of things; you have to trust that they won't do it.

seanp2k25• 9 years ago

to your point: chasing terrorism is like trying to solve for all these (and many more) nefarious ways in which society isn't looking for wrongdoing. It's basically impossible to safeguard against every type of threat. Manage to lock down one sufficiently so that it's no longer attractive to accomplish ${evil_stuff}? Move on to the next one. We now see people getting run down with large trucks, since air travel is much more restricted. Everyone still suffers from restricted air travel, but will the TSA start inspecting every car driving over a bridge or an on-ramp now to prevent these new attack vectors? I sure hope not.

nthcolumn• 9 years ago

Usual DR stuff covers this and everything else, no one person should have the keys to the kingdom or in this case access to the offsite backups. Offsite because fire, earthquake, zombie apocalypse, covfefe,... google 'disaster recovery'. If you are a small company take the backups home with you and lock them in the safe in the den. Trust nobody including yourself.

pavement6• 9 years ago

  covfefe

So THAT'S what it means!

thaumaturgy• 9 years ago

There should be a backup device or system offsite somewhere that automatically grabs incrementals of everything (and yells at you if it can't for some reason), essentially nobody should have any kind of remote access to it, and the fewest possible people should have physical access to it and gaining physical access should be such a hassle that you don't do it unless it's critical.

That's how I do it, anyway.

We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.

bobbob19213• 9 years ago

What you're describing here would require the company to be able to access their clients servers and data.

I don't know of any server provider (bare metal or cloud) that forces users to allow the company full access to their data (outside of managed providers were you voluntarily give this up , as you're paying them to fully manage your server)

stevefeinstein• 9 years ago

You prevent this by really trusting people (Trust, but verify). Don't screw them over. Treat them better than fairly. Give your employees raises before you give yourself anything.

And do backups. And then backups of those backups.

JBReefer• 9 years ago

Yeah, the key is inspiring loyalty and creating a nurturing environment. I loved my first two employers and would go in to either, today, if something I worked on was broken and they couldn't fix it.

That's probably because when my grandmother died, my boss at QueBIT said "Ok, go home, call me when you can work again - however long that takes." There was never a discussion of PTO/HR policy, just human treatment.

Also remember to test restoring your backups or they don't count.

mullen• 9 years ago

If you are in AWS, you can set the employee's account permissions to not be able to delete servers or drives. In AWS you can also us CodeCommit and set permissions so that the Repos can't be deleted either.

If the CI/CD is done right, then no DevOps staff has any access to any servers and no one can delete anything except a scripts and AWS configurations.

The whole problem with limiting permissions is that you have to do all the work of deleting files, servers and drives.

jldugger2• 9 years ago

Been waiting for a company to announce shutdown after this was posted: https://news.ycombinator.com/item?id=14476421

Possibly related?

Dreami• 9 years ago

Well it's kind of the opposite, isn't it? ;)

fps• 9 years ago

In both cases, critical data with no backups was deleted by an ex-employee. In the "first day" story, it just happened to be both his first day and last day. I guess the difference is that one was a developer and the other an admin. No telling from this story if the person who deleted the data was an employee when they deleted the data.

jldugger2• 9 years ago

If you take Verelox's word for it. Or if you want to be more conspiratorial, you could see your firing on the wall, tweak the new hire docs to lead to trashing prod, and just wait for the inevitable.

Or these two events are unrelated. Or the whole deleted prod on day 1 story is made up.

emeraldd• 9 years ago

Anecdotally, the timing doesn't match. Based on this https://www.lowendtalk.com/discussion/116329/what-s-up-with-... it looks like they went down on early on the 8th give or take a bit and the other was posted around then 2nd ...

Edit: Make that the 7th based on: https://www.facebook.com/Verelox/posts/1886196381643427?comm...

pavement6• 9 years ago

I would expect to see some kind of police report, and prosecution of an individual charged with a crime, no?

ntaylor• 9 years ago

Seems like nothing short of a felony. I can't conceive why someone with half a brain would do this.

Cshelton• 9 years ago

Yes, it is criminal and the ex-admin will go to jail. The cases for this are pretty straightforward... not sure why someone would throw their life away by doing it...

yaur• 9 years ago

Well, whats actually not clear from their message is if the person who is currently an "ex administrator" was an ex-admin at the time that they deleted everything or if they were fired because of a (bad) mistake and are now an ex-admin.

slantyyz• 9 years ago

> Yes, it is criminal and the ex-admin will go to jail.

What if the admin was a remote worker in a country that doesn't have an extradition treaty with the Netherlands (Verelox hq country)?

Paul-ish• 9 years ago

Hrm. I wonder how this would work if the ex-admin is working remotely from a country that wouldn't extradite for this.

ryanlol4• 9 years ago

Yes, it is criminal and the ex-admin will go to jail.

Maybe in North Korea, or the US. Relatively unlikely in civilized countries.

walshemj8• 9 years ago

Assuming that the police care about cyber crime and can find enough evidence.

yangtheman• 9 years ago

Proper exit procedure should have disabled all access from this ex-admin..., unless s/he had some sort of cron job or launched some process that would execute commands at certain time? I am very curious to know how it was done.

redm3• 9 years ago

That's all true, but in reality, we often count on the better nature of people, goodwill, for a proper break on both sides.

If someone is planning a malicious exit, it can be very hard to stop them depending on how "integrated" they are.

weaksauce2• 9 years ago

IT admin people have the keys to the building and pretty much all data at the end of the day. Trust is everything and reputation is extremely important. This will not go well for the ex-admin one way or another either by lawsuit or blacklisting.

gumby• 9 years ago

So true. A former employer continues to share google docs with me -- not just updates to docs I had been using but new docs as well. I don't read them because they're no longer any of my business, but I haven't been able to stop it from happening.

Gibbon13• 9 years ago

Often people about to get fired knew long before that axe was coming. Making sure everything is properly backed up and secure is a better option and what you should be doing anyway.

jlebrech• 9 years ago

timebomb?

satsuma• 9 years ago

Dead man's switch? Embed a process somewhere to trip when an account is removed from AD/some other trigger.

ars4• 9 years ago

should have disabled all access from this ex-admin

You can't. Not from an admin.

Same as how if you are rooted the only advice is to reinstall. It's simply impossible to reliably undo everything from inside the machine.

If you are a company, reimage the machine, then reinstall everything, and copy the code fresh from known good source control (and hope someone was watching source control that the admin did not check something in).

_ph_3• 9 years ago

Of course you can. The moment his account is deactivated, he should not be able to access any machine in the system. Unless of course, he installed proactively backdoors, which is a criminal offense, at least here in Germany. And with a proper setup, he should not get random remote access.

ams61102• 9 years ago

Unless you have exceptionally good controls it's very hard to be sure there is not an SSH key sitting on some machine that would allow access and possible nefarious activity by a dishonest ex-administrator

seanp2k25• 9 years ago

This is one of many great reasons to rotate them regularly in an automated way. e.g. https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansibl... or update it in your master image / wherever it comes from if doing immutable system images for deployments.

edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.

throwaway20483• 9 years ago

A "proper exit procedure" designed and implemented by who exactly? And what if they go rouge?

This problem is not as simple as you are pretending it is

funnyfacts365• 9 years ago

Once you turn rouge you'll never be rogue again.

valuearb15• 9 years ago

Bad employees know they are bad employees. Sometimes they plan ahead for the dismissal.

kashif• 9 years ago

Use Vault from Hashicorp where possible.

oneplane2• 9 years ago

That would most likely have made 0 difference.

thrillgore• 9 years ago

It's a process problem, not a product problem.

yardie4• 9 years ago

So in addition to the criminal side of things I guess the ex-admin wants to work in manual labor or fast food. There is no way in hell he'd have the references or pass the background.

BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.

whalesalad• 9 years ago

We hired a guy once who, a few weeks after starting, had to take three weeks off to serve jail time for his 3rd DUI. Good times.

vkou• 9 years ago

A little bit off-topic, but stories like this lead me to wonder why mandated ignition locks are not more prevalent.

Spooky23• 9 years ago

I had a retail customer once (I sold pc stuff to SMBs in college) who made a big purchase via credit card. The cashier got a weird code, and got transferred to a person who asked for a supervisor (me), and then asked yes/no questions about the physical description of the guy. Then they gave me some code to clear the transaction.

I was checking out the Sunday paper a few days later (this was the 90s), and it turned out that the guy was a fugitive who basically killed his wife and fed her to the fishes.

Pretty freaky stuff.

valuearb15• 9 years ago

"We hired a guy once who, a few weeks after starting, had to take three weeks off to serve jail time for his 3rd DUI'

You flunked your own background check.

ugh1232• 9 years ago

They didn't share what was in his check but it wasn't good.

Always ask to see the background check details if you're the hiring manager (you should have the rights to see that despite what HR might tell you). Could be just a personal issue an HR employee might have with a former colleague. Or discrimination-based (happens).

yardie4• 9 years ago

Didn't know I could do that.

I've had my own background checks in the past and no one has shared them with me. Speaking to HR, unless there is a negative item on your record they aren't required to.

tptacek2• 9 years ago

I'm not sure what you're referring to. If you mean a formal background check, which summarizes your criminal history or your credit history, then candidates are entitled to copies of the specific report the company claims to rely on in taking an adverse action, but nothing else.

If you're referring to private reference checks, of the type that would surface "personal issues people might have with formal colleagues", you're not entitled to anything whatsoever.

If you're a hiring manager in an organization where HR handles background checks, you personally as the hiring manager are entitled to nothing. I would venture further that it's inappropriate for HR to provide criminal/credit background check information to hiring managers.

Gibbon13• 9 years ago

Reminds me one of my old bosses hired someone. And then HR fired them because they had a pot conviction. So then my boss hired them as an outside contractor. Every six months HR would discover he was on the 'no hire list' and terminate his contract and my boss would cut him a new contract.

cperciva• 9 years ago

Discrimination is exactly why you want to leave that decision in the hands of HR. In the event of a lawsuit they'll have evidence to show that they followed the process your lawyers signed off upon and the decision to say "no hire" was based solely on permissible factors. At large companies, this will go as far as "here's a database dump of all the applicants with X on their criminal record... as you can see, we didn't hire any of them".

As soon as individual managers want to make those decisions for themselves, it becomes much harder to refute claims of discrimination.

damm4• 9 years ago

Background checks don't stop things like this. The problem is you have to hire trustworthy people in these positions.

Regardless of what crime they may (or may not) have made; you are looking for the person and what they do. Not what they did.

This is a common problem; we look at the past a bias of the future. Life only works out that way if the person is too unwilling to change; and that again is something you should look for in the hiring process.

Lastly; hiring ex-hackers isn't a bad thing. Caring about a background check when hiring an ethical hacker or someone who turned their life around; only shames them and pushes them back where they came.

So be careful or you only end up criminalizing being a criminal.

yardie4• 9 years ago

Sorry. If someone intentionally destroys company property I would want nothing to do with them, ever. This isn't a simple b&e where the perp rarely knows the victim. He was given the tools to do his job and intentionally used them to undermine his company and colleagues.

Also, he is not a hacker, just an asshole. Having the admin login and password doesn't make you elite. The only weakness he exploited was himself.

mdekkers• 9 years ago

The problem is you have to hire trustworthy people in these positions

Yeah, that. Also, secure backups and compartmentalized systems and data access.

redm3• 9 years ago

I've encountered many companies that are scared of lawsuits from bad references so responses, mandated by HR, are often limited to employment dates, and other very generic information. The mentality is, that person isn't our problem anymore, let someone else deal with them.

damm4• 9 years ago

On this case the person's problem is law enforcement. Basically they just need to pass off details to the FBI / local police and have the ex-admin arrested and put on trial. What he did was a crime

If a company walks away from this and doesn't take legal action; they should themselves get sued by their customers.

Destroying company property is a crime; wether it's defacing a website or ... it's not your property; you are just hired to maintain it (in one shape or another)

MichaelGG2• 9 years ago

He had the references to get hired at this job, right? So just leave the job you screwed off the resume.

A similar thing happened to a client. Sysadmin logged into GCP and Azure immediately after termination and just deleted everything. He was in the UK, we were in the US. Wasn't worth it to try to get someone to prosecute, and I'm sure we're not listed as references.

Did get people motivated for a multifactor delete bucket for extra backups.

alt2319• 9 years ago

I guess the ex-admin wants to work in manual labor or fast food

You'd be surprised. I did something somewhat similar and was convicted of a federal felony. No fast food place or retailer would touch me with that record. But ironically I've found plenty of IT work with smaller companies.

douche4• 9 years ago

Ironically, we are much laxer on those with more responsibility. When was the last time a middle manager or senior executive had to pass a piss test to be employed? It's routine in blue collar work.

cbsmith6• 9 years ago

HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.

I always have a hard time with those contexts. HR sometimes has the wrong idea of what is unacceptable and what is appropriate. There really isn't a good reason for them not to tell you. The only thing that would make sense is if they shouldn't know it.

dsacco• 9 years ago

Why didn't you ask to see details from HR? There's no transparency in their decision making - how do you know he was turned away for a good reason?

Ensorceled• 9 years ago

HR's view point is that the candidate has a right to privacy and their personal problems are none of the hiring manager's business.

1024core2• 9 years ago

I had a manager who was consistently underpaying me, significantly. When I quit in disgust, he threatened me that he'd make sure I never got another job. He had been milking my success to get lots of bonuses and raises for himself, even though I did absolutely all of the work and he just "managed" (which meant, ignored me completely). So he wasn't happy that I was quitting, and though he could bully me. Luckily for me, I didn't have to use him as a reference, as 2 other PMs agreed to vouch for me.

So, just because a guy has a bad reference it does not mean that he's necessarily bad.

mb_72• 9 years ago

"So, just because a guy has a bad reference it does not mean that he's necessarily bad."

This is so true. Years ago I was brought into a company specifically to improve their software quality process, but without being aware one of the company owners (of which there was two) was against be being employed for that purpose. I uncovered a lot of incompetence and outright corruption with some employees. The 'good' owner went on stress leave, and then pressure was brought to bear on me, resulting in me quitting on the spot one Monday morning; not my proudest moment, but I couldn't take the pressure any more, and my ally was no-where to be seen. Fast forward a decade, and I interviewed with someone who used to work at the same company; it turns out that my leaving had been framed as 'fired for incompetence', and word had been put around part of the local industry that I was hard to work with, unreliable, bad at my job etc. I laugh about it now, but at the time it really bothered me about the possible damage done to my career and reputation.

morganvachon• 9 years ago

First, I'm not a lawyer and this is not legal advice, just my own limited knowledge and experience.

I don't know what country you're in so I won't speculate on what's legal or not in your area, but here in the U.S. it's illegal for a prior employer to provide false information to a prospective new employer during a background/reference check. If your former boss tried to blacklist you like that, he'd put his company at risk for a civil suit. And, while it's not illegal to truthfully say that a current or former employee is a bad employee, doing so rides the thin line of opening the company up to a libel lawsuit.

Generally speaking, a company might "dish the dirt" on a former employee if there are criminal charges to back up the claims. Even then, legal and HR will likely frown upon it. Usually, when the new company calls the old company for a reference, the old company will say something like "Yes, $employee worked here from $startDate to $endDate" and refuse to divulge any other information in order to avoid any semblance of libel.

Once again, this is my limited experience in SMB and government settings, I have no experience in mid-to-large businesses, Fortune 500 companies, and Silicon Valley startups. We've all heard stories about managers at such companies going around HR and discussing potential hires at the bar or on the golf course.

therealmarv2• 9 years ago

Is this deep background check again a US thing? In Germany it's even unlawful to ask potential employees if they are pregnant. It's essentially discrimination. Maybe some HR will check your social media accounts and search on Google for you but generally police records (at least my experience in IT) or even health records are not requested (or against law!) unless it's a high responsive job like e.g. becoming a police officer or maybe military.

Zak2• 9 years ago

Asking potential employees if they are pregnant, married, plan to become pregnant, have children already, etc... is illegal under US law as well. Common German practices like asking for a photograph on an application or CV or asking about a prospective employee's religion are also usually illegal in the US, as it's pretty hard to think of non-discriminatory reasons an employer would need that information.

Asking previous employers about their experience with an employee, however is not illegal. It's usually not illegal for them to say something negative if it's true, though some businesses are conservative about what they will say out of fear of being sued for slander. Accessing public court records or news stories about criminal cases and using that information for employment purposes is usually not illegal. Asking prospective employees if they've been convicted of serious crimes is usually not illegal.

stormking• 9 years ago

What are you talking about? At all (german) companies I ever worked at is was common to request a "Polizeiliches Führungszeugnis" (essentially police records or the lack of such) first thing during the trial period. This is absolutely common, maybe you just forgot about that?

cannonpr• 9 years ago

A lot of 'managed' hosting providers are pretty bad with security, there still is a major provider that just gives root credentials to all servers to all techs not just admins, doesn't audit who accesses which credentials, and doesn't rotate credentials, doesn't rate limit dumping credentials... That's before we go into more interesting issues with their security. Frankly I am surprised this sort of thing doesn't happen more often ? In some ways it both restores some of my faith in people while reducing some of it at the same time in a different vector.

gaius• 9 years ago

Who even knows it was an ex-admin? Could be the current one fat-fingered it and is trying to shift the blame! We just don't know.

philfrasty• 9 years ago

Exactly. Mind-boggling how easy it is to present people a logical reason.

pmarreck2• 9 years ago

This is so stupid. If you have a problem with your employer, either you quit or they fire, you move on, full stop. If you're in a relationship and someone isn't happy enough with you and breaks up with you, the dignified response is NOT to key their car. I see employment relationships mostly the same way. Either it works (for both) or doesn't (for either or both ends).

And having switched jobs quite a few times, the next one is always better for you, regardless.

HipHopHacker• 9 years ago

I see employment relationships mostly the same way

Apparently they did not.

At the end of the day, the people working at the company are the ones who are doing the work, and who have control of the means of production. The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.

I worked at a Fortune 100 investment bank where this happened. Everyone knew layoffs were coming. One week after layoffs came, a digital "bomb" went off wrecking many servers. So security went through, trying to find evidence (nothing incriminating from what I heard, although they had a strong suspect) and also looking for more bombs. They missed out on finding and defusing one, because another one went off a month later.

The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.

jjnoakes• 9 years ago

A person who can destroy servers is no more indispensable than an employee who can destroy a physical office.

hoodoof4• 9 years ago

Integrity is choosing to behave in the right way, even though you have the option of behaving otherwise.

Therefore, to behave with integrity, you must have formulated your own set of values about what is "the right way" to behave.

Every minute of every day, we all have the option to behave with or without integrity in a whole range of ways.

You earn respect by demonstrating behaviours over time where you have taken the interests of others into consideration, generally people consider someone who behaves like this to have "integrity", especially when they continue to behave that way when no-one is looking.

Saying things like "The ex-admin's bosses probably thought they were the important ones" indicates a childish set of values where there is a power struggle between employers and employees ........ of course the "bosses" are the important ones, they act for the business which is an independent legal entity, upon which many people depend for their lives to work effectively. If, as an employee, you feel poorly treated or otherwise dissatisfied, then the right thing to do is leave in a polite and respectful manner, even if you feel you were not treated in that way. Depending on the circumstances, if you were actually treated really badly, then the right thing to do is pursue your complaint through the appropriate legal channels.

Someone important in my life once said to me "the only thing you have is your reputation". Take that reputation, defend it, enhance it, nurture it and earn the respect to grow it. Don't throw it in the garbage by smashing other people (or their business) in a childish tantrum. I admit this is hard to do - I regret many things I have done in my life, but I try to lead a life consistent with my own sets of values that I think are meaningful and I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.

paulsutter• 9 years ago

Yes and I know a guy who went to prison for a revenge-deletion after leaving the company. He's a really good guy and he made a mistake in anger. While I disagree with the exact outcome in that case, I also think it's wrong to regard it as heroic (as implied by your post).

samlevine• 9 years ago

The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.

The solution to this, should it become a regular occurrence, is to make the folks with the keys to the kingdom replaceable.

It's doable, companies just don't do it because most people don't want to destroy their high paying and relatively comfortable careers committing felonies and getting sent to prison because they had to spend a few weeks or months looking for a new job.

jdoliner• 9 years ago

The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.

I don't think this really indicates that at all. Maliciously inflicting damage on the company when you're fired is very different from being irreplaceable. It makes it risky to replace you, but that's not the same thing.

swampthing• 9 years ago

I don't think it's realistic to expect this company to now treat their employees less like replaceable cogs than they were previously (to the extent they were at all). What is far more likely is that this company will now just have more red tape in place to ensure that this doesn't happen again.

colanderman• 9 years ago

The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.

Sabotaging servers doesn't mean you're unreplacable any more than a terrorist attack means Western culture is depraved.

I'd believe you if the servers simply started falling apart without this person around, but that wasn't the case.

valuearb15• 9 years ago

So if the company doesn't have enough revenues to afford all the employees, it's a bad guy for laying some off so it can stay in business and have jobs for the rest of the employees?

crdoconnor3• 9 years ago

"I see employment relationships mostly the same way."

It's not an equal relationship. One side usually has significantly more power than the other.

valuearb15• 9 years ago

LOL. You've obviously never had to try to hire good employees.

pmarreck2• 9 years ago

What is "relative power" in the context of, say, an employer with excellent business acumen, a manager with excellent interpersonal acumen and an employee with excellent technical acumen?

jacquesm4• 9 years ago

This is why you have the backups stored under a different account than the primaries and you make sure that nobody has access to both accounts.

treyfitty7• 9 years ago

This may be an unpopular opinion, but I want to preface this by saying: "before passing judgement, context is always necessary."

Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important.

But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet.

My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.

nebabyte• 9 years ago

speech

Reply

treyfitty7• 9 years ago

Wow. That just shows just how reliant I've become of autocorrect. It didn't autocorrect, nor did it present a red underline, and I had a feeling "Speach" looked wrong. But I doubted myself and figured autocorrect would have my back.

Reply

xupybd2• 9 years ago

I don't know man. I can't see any situation where these actions are justifiable. Revenge generally helps no one.

Reply

treyfitty7• 9 years ago

For what it's worth, revenge doesn't require a net benefit to the situation. The word "revenge" is kind of similar to "retaliation" where an actor is purely seeking retribution for a past event without regard.

To us, it's not justifiable. Hell, the ex-admin may think so too. But that doesn't mean we should automatically side with the employer and subsequently crucify the ex-admin. We don't have any information yet.

Reply

moron4hire6• 9 years ago

How do we even know the story is at all truthful? Maybe someone who still works there did something stupid they couldn't recover from and is now blaming it on "an ex-admin" to make it sound nice.

Reply

JungleGymSam• 9 years ago

Why are we so quick to side with the employer in this matter when we know nothing of the situation at all?

It's called collateral damage. There is no context, outside of fantasy, where the admin could be in the right to do this.

Reply

treyfitty7• 9 years ago

Eh, I'm inclined to agree with you that the ex admin isn't "right" but the essence of my statement is that he may have been carrying out what was his optimal strategy.

If you treat people bad, they'll treat you bad. If you nuke Russia, Russia will nuke you. It's the Nash Equillibrium where each party is faced with a game and certain situations call for your best move. But your best move should account for what I'll do, and that should be factored into your initial move.

Mutually assured destruction is actually a powerful deterrent. Do we know what the ex-admin's situation was prior to all this? Only then should we pass judgement.

Reply

nickpsecurity10• 9 years ago

What if the customers' data was preserved, archived, and sent to them as links? And just the VPS was trashed? Now, we're talking a brief disruption instead of total loss which the customers indirectly signed up for by going with a cheap, small player instead of one qualified to provide higher availability and security.

The collateral damage would low to non-existent for the users depending on how critical the service was. The direct damage would be to the employer. If the employer was the bad guy (hypothetically), would you still state there's no context or any situation whatsoever where the employee should cause them damage?

Reply

cnnsucks• 9 years ago

Mental gymnastics.

Reply

idealogue4• 9 years ago

I agree with you. The majority of people in this thread have completely internalized that property rights are more valuable than human rights. Only one comment I've seen has acknowledged that all we have is a single webpage with a single accusation, and yet many people leap straight to the conclusion not only that the individual in question exists and is guilty, but that there can be no justification for a person destroying a business. Businesses destroy people all the time. I for one would like to see the tables turned more often.

Reply

SomeStupidPoint4• 9 years ago

HN is full of baronets arguing in favor of the modern social order -- because while they're not real aristocrats, they're not peasants.

Corporations function as de facto titles of nobility in the US, so crossing one is crossing social rank.

Reply

xupybd2• 9 years ago

We have responded to the article based on the information presented.

There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.

Reply

pythia__• 9 years ago

I for one would like to see the tables turned more often.

The most prominent example of this in recent memory is Peter Thiel sponsoring Hulk Hogan's lawsuit against Gawker. Do you approve of what he did?

Reply

_Codemonkeyism2• 9 years ago

" I for one would like to see the tables turned more often."

I don't know you.

But what I have learned that people who do not care about other peoples property, care very much about their own. Destroy other people business? Go for it! Destroy my car which I could kill someone with? No way, it's mine, I've worked hard for it. Vandalize houses of rich people? Go for it! Steal my iPhone? Hey, where is the police when you need them?

In Berlin people cheer the burning of other peoples BMWs - yes this is a thing. The same people go to court when the police scratches their table tennis table during a raid.

Reply

valuearb15• 9 years ago

"Businesses destroy people all the time".

Lol. Citation please.

I'd like to see the justification for forcing thousands of customers into expensive disaster recovery because the company fired a sysadmin, likely for good reason.

Reply

cbsmith6• 9 years ago

without context to the situation, who are we to decide who's right or wrong?

You can comfortably make a determination about what is right and wrong. We don't know the facts, but if the claims are true, wiping out not just that company's property, but that of their customers, is a crime.

Now, sometimes a crime is justified, but I don't think it is a rush to judgement to work from a starting point that criminal behaviour is bad until proven otherwise.

Reply

thanksgiving4• 9 years ago

I disagree. Once again, we can find a scapegoat at the lowest possible level and "stone" them or we can do a proper post mortem and try to learn from it.

Why did this happen? How can we prevent it from happening in the future? These are the questions we need to stress.

In particular, why does an ex-employee still have access to production? I say when something like this happens and heads must roll, they must roll at the top. Fire the CEO. Fire the board. Leave the sysadmin alone.

This is a civil matter. My tax dollars should not pay for a criminal lawsuit. Screw that.

Oh and by the way if you're reading this: please help repeal cfaa.

Reply

moron4hire6• 9 years ago

I take it you're one of those folks who doesn't care for the whole "innocent until proven guilty" concept.

Reply

Viker• 9 years ago

Who defines criminal behaviour? And why is criminal behaviour always placed on the little guy when companies can always get the leeway of claiming ignorance?

Reply

bobbob19213• 9 years ago

Employment is a voluntary transaction, entered into by both parties. One should always focus upon increasing their own skill set such that an employer may be able to make your life temporarily difficult, but your skills and value to an employer (or future employer) are such that they can't "ruin your life".

Reply

SomeStupidPoint4• 9 years ago

Employment isn't voluntartly: there is strong economic coercion to be employed and high degrees of friction are introduced to make that coercion more effective.

Urbanization fundamentally has been engineered to create "wage slaves", who are basically modern sharecroppers (though the system has some extra steps).

Reply

pikzen• 9 years ago

Employment is a relation of subordination, inherently unbalanced. People also happen to need money to live, making work not a voluntary transaction, unless you consider death a viable option. That is not considering other factors, such as having to feed other members of your family. It is made even more unbalanced by the amount of influence your employer can have in your life by firing you, as opposed to the influence you can have on him by quitting. Employment is coercion.

I wish HNers would get their heads out of their tech arses and face the reality. Most people do _not_ have a choice where they work. Most people cannot afford to quit. Most people don't have jobs where they can increase their skillsets. Most people do not have time to read HN while they enjoy their 10 AM pause in their comfy sofa while working from home. You are, for the most part, amongst the most privileged people in the world. Yet you continue to spew the "employment is voluntary" propanganda because you never had to face actual hardships. Worse, you push people of lower classes even further in the ground, when what you should be doing is elevating them to make a better and fairer society.

Reply

microwavecamera• 9 years ago

Free speech: telling your boss off, quitting, organizing a walk out, refusal to work, picketing, posting your story online to warn other potential employees and customers just who their doing business with

Not free speech: punching your boss in the face, burning the building down, vandalizing the office, deleting all the data on the servers and hurting a bunch of people that had nothing to do with your conflict with management

I'm all about workers right and have walked out on many a job because management were raging assholes. I have convictions and I stand by them but conviction and doing the right thing in life often come with sacrifices. I'm currently underemployed because of my convictions but I can sleep at night knowing I'm doing the right thing. The way for us to take back power in the tech industry is to organize, unionize and refuse to work for abusive and exploitative employers. We're the ones with the skills. They need us not vise versa.

Reply

antfarm• 9 years ago

I wonder whether the ex-admin was already an ex-admin at the time he wiped the servers.

Reply

bobbob19213• 9 years ago

Lots of comments are interpreting "ex-admin" as someone who was fired and then after went and did this. Just want to float the possibility that "ex-admin" could also mean someone was employed there, then did this and is now no longer employed as a result of doing this.

(Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )

Reply

sqldba3• 9 years ago

What if their information was all secretly being leaked to intelligence agencies and there was no way to ever let them know except to burn everything?

There's probably excuses and justifications. I personally wouldn't do it and they're probably wrong for doing it but I don't want to jump to conclusions and moral absolutes so easily.

Reply

stevenh• 9 years ago

If I ran a hosting company and all of my servers were compromised by ring -3 malware exploiting the Intel AMT vulnerability, the first thing I'd do is privately inform Intel that I intend to go public with the story and sue for damages, after which Intel would perhaps offer a very generous bribe for my silence and a week-long window to replace all of the server processors for free, on the one condition that I bury the truth by fabricating a story about an imaginary ex-employee who improbably was both smart enough to gain an administrative position in a large company while also being stupid enough to risk decades in prison for petty revenge over workplace drama.

Reply

forgottenacc57• 9 years ago

Weird. That's exactly what happened to the data centers at Area 51 too.

Reply

ronilan• 9 years ago

Also similar to when Gillette paid a very generous bribe to bury Occam's and Hanlon's razors. /s

Reply

qb45• 9 years ago

No, these were stolen by aliens but CIA covered it up by pretending to be covering up AMT malware ;)

Seriously though, would Verelox still be running unpatched AMT many weeks after the disclosure of this authentication bug? Or does GP think there are more bugs which Intel hopes to sweep under the rug forever by individually covering each incident? They would spend quite a money on these bribes while AMT bugs can simply be fixed with BIOS updates.

This Intel conspiracy doesn't make sense. It's aliens, folks, I know it.

Reply

midnitewarrior• 9 years ago

What kind of hosting company is going to have any kind of reputation after admitting their security allowed an ex-admin to ransack everything?

Rephrase the question -- what idiot customer is going to do business with such a place that allows such a lapse in security to happen?

Intel would basically have to buy the company.

Reply

atmosx• 9 years ago

What kind of hosting company is going to have any kind of reputation after admitting their security allowed an ex-admin to ransack everything?

The kind of people that:

  - Use Gmail, iCloud, etc. post Snowden
  - Buys SSL certificates from Comodo, etc.
  - [put other companies here]

So, pretty much everybody, me included. Your idea that mis-management can damage a company's reputation permanently, has been proved wrong. The complexity of moving an infrastructure in and out of a service provider is apparently bigger, os as you say, we're all stupid - we manage to survive somehow though.

Reply

chris_wot• 9 years ago

Either way, their reputation would be trashed. Intel knew about potential issues in their products for many, many years. I'd shed no tears if lots of people do this to them now.

Reply

hoodoof4• 9 years ago

>admitting their security allowed an ex-admin to ransack everything

What, exactly can be done to secure a company against a malicious systems admin? These are the guys typically with not only the keys to everything but also the knowledge of how it all works.

You say that the company cannot be trusted for "allowing" this to happen.

I know quite alot about this stuff, and for MOST companies, they simply have to trust that the people with the keys to the castle with behave responsibly.

There are ways to design infrastructure such that it is protected from its builders and keepers, but this is very very hard and complex and expensive.

Presumably you work for a company that has taken steps to ensure this will never happen, what are they?

Reply

_Codemonkeyism2• 9 years ago

VW has record sales.

Reply

ryanlol4• 9 years ago

How Intel would most likely respond to you: "HahaHAHAHahAHAHAHahahAHAHahAHA, fuck off"

Reply

draw_down• 9 years ago

What's the opposite of Occam's razor?

Reply

quicksilver03• 9 years ago

The thread title should be changed to "Verelox allegedly wiped by ex-admin": we only know one side of the story.

Reply

svakacast• 9 years ago

Anyone of us know what is the best way to get refunded? My Company lost 20.000€ for this joke.

Reply

bm13622• 9 years ago

Lawsuit, most likely.

Reply

tgtweak2• 9 years ago

This exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean).

There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.

"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.

They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.

They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.

Good cautionary tale for segregation of credentials and proper user key management.

Reply

mintplant• 9 years ago

Looks like DO has managed to keep this detail out of the record for the most part, save some forum posts that pop up if you search "realitychecknetwork digitalocean". It would explain the odd gap in time in their Wikipedia page history:

In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.

Reply

tgtweak2• 9 years ago

Yeah Ben and Moisey are both incredibly competent sysadmins and shrewd businessmen. They deserve the success they've had but they were very close to desperation after this incident. I can't blame them for omitting it from the records, the reality is likely that digital ocean and serverstack are both much more secure as a result.

Reply

FussyZeus2• 9 years ago

Problem is at some point some person or more likely group of people has to be entrusted with the Master Keys. Using different keys for production machines and that sort of thing is all well and good but they're kept somewhere, meaning if an admin decides to screw you, there isn't a lot you can do.

Seems like the better option is keep your admins happy as much as possible.

Reply

pvg3• 9 years ago

By that logic, all employees should be kept as happy as possible because they might commit arson.

Reply

tw042• 9 years ago

It's always interesting watching startups learn the lessons that thousands of enterprise learned along the way. "Why would you ever want offline tapes sitting in iron mountain, how inefficient".

Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.

Reply

nulagrithom• 9 years ago

If we go to war with China I don't think you or anyone else will care much about your company.

I know you meant it as an example, but this sort of extreme attitude towards security is just another footgun.

Reply

kstrauser• 9 years ago

If we go to war with China I don't think you or anyone else will care much about your company.

Why? We went to war with Europe and Asia a few times and businesses kept chugging along here in the states.

Reply

tw042• 9 years ago

You're assuming war will involve armed conflict from the start. If we go to war with China that will be the end of the war.

Reply

DashRattlesnake7• 9 years ago

If we go to war with China I don't think you or anyone else will care much about your company.

Probably not in the immediate aftermath, but someone might decades later, if the company actually does something valuable.

Reply

scott_karana• 9 years ago

Heck, it doesn't even have to be Iron Mountain if your data size isn't too big! Just rotating tapes to a safety deposit box at your local bank every week will do a damned good job.

Reply

keithpeter3• 9 years ago

https://www.lowendtalk.com/discussion/116329/what-s-up-with-...

Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox

Reply

CM30• 9 years ago

This is why a hosting company needs to both segregate credentials to only what an employee needs for their job, as well as to revoke them the minute they leave the company.

Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems.

They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back.

As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole.

So yeah, what a disaster all round.

Reply

Verelox Wiped by Ex-Admin

Related Stories