The Looting of ShapeShift

https://news.bitcoin.com/looting-fox-sabotage-shapeshift/

by danielvf • 10 years ago

198 90 10 years ago

news.bitcoin.com

danielvf3• 10 years ago

This is certainly the worst case scenario - your security officer installing remote access software on developers machines, stealing bitcoins from production, then selling the company source code, access credentials and access to the internal network to a Russian hacker.

Building a security system to handle this level of attack is a whole level beyond stopping even determined external attackers. Are there any best practices guides on this?

One thing that the article showed is the importance of external security review to deal with the threat of internal incompetence or evil.

tlb• 10 years ago

Ultimately, the founders of bitcoin companies need to understand security themselves or they won't be able to hire and manage a competent team.

Despite all the theoretical advantages of blockchain over conventional banking protocols, they are not doing categorically better than banks at avoiding major losses. Although, the numbers at http://www.risk.net/operational-risk-and-regulation/opinion/... are shocking.

danielvf3• 10 years ago

To partially answer my own question, there's a Crypto Currency Security Standard, with Level 2 claiming on preventing solo internal attacks.

https://cryptoconsortium.github.io/CCSS/

srdev• 10 years ago

Are there any best practices guides on this?

I'm not aware of any, but we did have quite a lot of procedures meant to mitigate this sort of things when I worked in banking. For one, none of the developers had any sort of access to production machines.

joe_the_user• 10 years ago

I am not sure why those who essentially believe in utopia through an organized "war of all against all" wouldn't expect that anyone in position of trust wouldn't try to rip them off as much as possible.

"Damn you, you've betrayed our trust and prevented us from building a world where we didn't have to trust people..."

raesene9• 10 years ago

Well the traditional banks have been dealing with exactly this kind of threat for a long time.

There's a load of practices that are designed to stop this kind of problem in that world.

Things like :-

- Enforced holidays. Frauds are hard to maintain when you're not there to keep cooking the books. - Audit reviews. an independant function with the ability and authority to review key processes - split authority. Key actions need multiple people to complete (statistically fraud incidence drops a lot when you need multiple people involved) - Strong background checking. When I worked at a bank they went back 10 years of employment history and required accounting for any gaps. + credit checking + criminal record checking etc.

It's all possible, just quite expensive...

illumin83• 10 years ago

It's extremely expensive. Many banks and companies in the finance industry (hedge funds) do this:

Hire at least 2 or 3 people for every job. Have them watch each other whenever touching systems that connect to production or deploying code to production. Never trust any one of them with the private keys or passwords to anything - they can only get half of a secret and their co-worker gets the other half.

To do this effectively, you have to build a zero trust environment, and rely on surveillance to ensure that nobody is a bad actor. It really makes CIA/NSA level security look somewhat weak.

It's also very expensive, as you can imagine.

rodgerd2• 10 years ago

Hire at least 2 or 3 people for every job.

This is actually the key one - the more people who need to be corrupt, the harder it is to get away with being a crook. A surprising amount of internal fraud can be completed simply by requiring people to take solid blocks (2 weeks plus) of leave every year.

That "inefficiency" the "lean 10x disruptors" congratulate themselves over does not always end well.

cortesoft3• 10 years ago

Step one is to properly vetting the person in charge of all your security.

oh_sigh• 10 years ago

What's the status of civil/criminal charges against Bob?

quanticle2• 10 years ago

    We learn some more things. Bob has prior police records in Florida, where he’s from.

So they didn't even do a background check before hiring "Bob"? For a position where he would have access to systems that handled financial data? That's just grossly incompetent, in my book. I've worked for 5-man startups and Fortune 500 companies. In every case, the offer letter has stated that the offer is conditional on the successful completion of a background check, and none of the positions I've held have been remotely as sensitive as the position that Bob was hired for.

cloudjacker4• 10 years ago

They don't do the background check.

quanticle2• 10 years ago

They do. At least, my latest position involved submitting my employment data for the past seven years to a firm that conducted background checks. Again, this was for a position that was several layers removed from any kind of direct access to customer data or financial data.

randomfool• 10 years ago

They do- one company found a warrant on me that I didn't know I had for an unpaid traffic ticket.

jessaustin3• 10 years ago

That's sort of boilerplate though, isn't it? They reserve the right to do all sorts of terrible shit, but because most of that shit costs money they don't actually do it.

TAForObvReasons• 10 years ago

There are few positions that merit a hiring background check more than ones directly involving the financial transactions of a company. Even if it costs a lot of money, it is absolutely money well spent.

tekromancr3• 10 years ago

Facinating. More interesting reading seems to be available here: https://www.patrolx.com/wp-content/uploads/2016/04/309591980...

jessaustin3• 10 years ago

It seems odd to me that "Bob" hasn't been outed. It almost makes me suspect that someone isn't sure how much of "Bob's" role as portrayed in TFA is true and how much is a frame-job by an untouchable hacker [EDIT:] or wishful thinking by a frustrated executive.

adekok2• 10 years ago

It seems odd to me that "Bob" hasn't been outed.

Since Bob hasn't been criminally charged with anything, "outing" him is legally "libel". He could sue, and win.

jessaustin3• 10 years ago

Why didn't you quote the next sentence, in which I directly addressed the possibility that TFA is wrong about "Bob"?

Let me guess, YANAL? In USA, it ain't libel unless it's provably false. If "Bob" could prove that, why did he run?

ascorbic3• 10 years ago

It's not libel if it's true, and they'd only need to prove that to civil standard.

fasteo2• 10 years ago

Am I the only one to think that all this narrative to blame Bob is pathetic ?

This is pure and simple Mr. Voorhees (CEO) incompetency. After all, Bob is a criminal and he was just doing his "job".

pavel_lishin• 10 years ago

Are you suggesting that the criminal should not be blamed for his criminal actions or the losses that he knowingly and willingly incurred?

fasteo2• 10 years ago

Not quite. I suggest that a CEO should assume his responsibilities. He is quick to trash Bob[1] several times, but I see assumption of any responsibility at all.

[1]

Despite our note to all employees to come into the office urgently, Bob, our head IT guy, the one responsible for security and infrastructure, arrives at 11:30am.

Soon after, Bob decides it’s time for his lunch break, and we don’t see him for an hour, during the worst incident in ShapeShift’s history

danielvf3• 10 years ago

The article seemed pretty open about major mistakes that ShapeShift made and lessons learned. It's a good postmortem to learn from, and far more open than most would have posted.

tekklloneer• 10 years ago

Their fundamental business model didn't account for and compensate for the incredible amount of risk involved in handling large amounts of money.

illumin83• 10 years ago

One of the striking things in this article was when he said they might have been compromised by their "CloudCo" (Cloud Provider). If I'm going to build any systems that handle money or bitcoin in a cloud provider, I will make damn sure I don't trust the cloud provider with anything.

Everything should be fully encrypted such that even a breach of trust from the hosting provider would not compromise your data/funds. I know this is hard to do, but it's mandatory when you're handling digital currency.

pessimizer4• 10 years ago

No. There are Bobs everywhere. Do your job, and Bob doesn't rip you off. Leave Bob to the courts - blaming Bob is like blaming your dog for stealing your lunch, or a wave for soaking you on the shore. You will not hurt Bob's feelings.

edit: Bob didn't betray you. Your friends and family betray you. Bob stole from you.

vonklaus• 10 years ago

Semantics aside, if this story is to be believed the unknown hacker theif was even a bit put out by Bob. Especially if they guy robbed his employer after compromising & sabotaging key infrastructure and tech, then selling that to other people to exploit.

I mean, that's baseline treachery right there.

I like shapeshift and have used it a bit. If you(generic) hate them, alt currency, or the CEO, I think it is fair to say that Bob is pretty shitty. Yes there are sociopaths and criminals everywhere. If you have met a sociopath in real life, it isn't simply a myth that they are charming and appear normal. So I agree with the sentiment that you should expect bad things at your company, but you seem to imply:

1. it should have been obvious Bob was a criminal

2. Bob is blameless?

ascorbic3• 10 years ago

The Russian hacker didn't betray them, but Bob did. They trusted him enough to hire him. He betrayed their trust.

ascorbic3• 10 years ago

Are you really saying that criminals can't be blamed because "thieves gonna thieve"? Sure ShapeShift can take some blame for bad security, which he readily admits in the post, but the bulk of the blame is owed to the person who did the crime.

rodgerd2• 10 years ago

Eh, when CEOs lay off tens of thousands of people to pocket a sweet bonus, they're "just doing their job", regardless of whether it ruins their ex-employees lives. I think it's a bit of a reach, but "just doing my job" seems to be a modern get-out-of-jail card for pretty much any action that harms others.

cortesoft3• 10 years ago

Man, calling a social security number a "social serfdom number" is really dumb and off putting. So is the continual reference to 'fiat money' constantly.

I always love the irony of people so against the basic social contract are always so quick to turn to authorities when things predictably go wrong.

mfringel• 10 years ago

Indeed. Using tribal signaling when trying to reach a wider audience requires at least a small amount of skill.

bionsuba• 10 years ago

Would you call a communist a hypocrite for buying food at a grocery store?

tootitandbootit• 10 years ago

Only if they were forced to buy food at a grocery store because their farm communes ran out of food because no one wanted to work on it.

fleitz• 10 years ago

If they led / lived in a communist area, yes.

eli• 10 years ago

Nope, but some people would!

https://twitter.com/edroso/status/721760775530876928

cloudjacker4• 10 years ago

You might appreciate

http://www.newyorker.com/humor/daily-shouts/l-p-d-libertaria...

ikeboy• 10 years ago

http://www.theatlantic.com/politics/archive/2014/04/nlpd-non...

neurobuddha• 10 years ago

I was thinking more along these lines: http://www.theatlantic.com/politics/archive/2014/04/dont-swe...

civilian• 10 years ago

Idk, I'm so tired of NewYorker-style liberal's attempt at humor about other political groups. It's disrespectful because they don't show that they've attempted to actually think about the issues, they're just knee-jerking with their emotions. Their treatment of Trump is similarly ridiculous. They're just waving their blueTribe flag as hard as they can.

And when liberals talk about Ayn Rand? It's like she's the worst depraved demon ever. But if you read some of her quotes, they're kind of motivational. Sure it can be a little too selfish, but I know a lot of people who would benefit if they internalized that THEY are the main driver in their own lives, rather than playing victim their whole life. http://www.goodreads.com/author/quotes/432.Ayn_Rand “If you don't know, the thing to do is not to get scared, but to learn.”

I've been watching the Rubin Report b/c Dave Rubin is a liberal that will actually fucking sit down and talk it out with someone on the otherside. Great show. https://www.youtube.com/user/RubinReport

medic• 10 years ago

That's only funny until you realize that's exactly how health care in the US works.

walrus013• 10 years ago

Whole article tl;dr: randian libertarian idealist buttcoin people are extremely surprised to discover that a black market economy turns out to actually contain real blackhats.

pessimizer4• 10 years ago

And that the banks aren't so dumb after all. Through centuries of experience, they have learned the secret that you need more than one employee.

BTW, currently looking for an Senior Infrastructure and Security Engineer/Manager position at a company that handles large amounts of money.

floordaemon2• 10 years ago

At least he can laugh during calamities. Takes courage to do that.

koolba• 10 years ago

Not the best writing but it reads like a modern hardboiled[1]. Fun piece.

[1] https://en.wikipedia.org/wiki/Hardboiled

keithpeter• 10 years ago

"We had changed almost everything, but hadn’t scrapped our personal computers used while Bob had been part of the team. Would that have been the paranoid thing to do? Yes."

At my humble and refreshingly drama-free place of work we have standard client images. Anything weird and the techies re-image the client. Assuming 'Bob' wasn't in charge of the images, would such a procedure have sorted the rdp?

adekok2• 10 years ago

Maybe.

The larger question was why did Bob have root access to people's individual laptops? He could have done a "snowden", grabbed their SSH keys including passphrases. That would have been much harder to detect.

azernik• 10 years ago

That attack would have been prevented by their immediate SSH key rotation after the initial attack.

felixgallo• 10 years ago

I'm surprised that anyone would take this at face value. The cryptic black hat responding to e-mail with "one word: bob" is straight out of the most teenage of hacker fanfics.

a_small_island• 10 years ago

I thought this was a great story to read. Explained in laymans terms when needed and kept my attention throughout.

Bob sucks.

floordaemon2• 10 years ago

When things are handled professionally, you'll have a lot less drama.

Why is the author being so nice to the theif over and over again? Even the last sentence!

Eric brought this all on himself. Obviously 'scared' and intimidated to discuss important things WHEN they need to be discussed.

Doing everything out of order -- like the background checks -- is a classic rookie move.

You are basically running an Online Banking Website. You need to be aware of immense risks...

...it just goes on and on. But its clear why there is so much drama, crime and corruption and this company.

Sad really. Who is to blame?

marco_salvatori• 10 years ago

For those interested in security engineering in the financial industry a good reading source is the banking section of: http://www.cl.cam.ac.uk/~rja14/book.html One interesting and relevant take away (of many) -- The greatest fraud threat to a financial firm are insiders. About 1% of staff across the industry is fired every year due to fraud. Within the fraud incidents, the most damaging fraud is perpetrated by senior and trusted individuals.

Nutmog2• 10 years ago

Let's not lose sight of the fact that their cold wallet was untouched and all they lost was on the order of a hour's worth of turnover. That's more than can be said for a lot of the other bitcoin hacks.

Ontheflyflyfly3• 10 years ago

"Very quickly, we realize he is pretty much useless. "

When you hire IT people, have no clue how distinguish between a good one and a fake one, in other words have no clue, this happens.

Also not enough oversight and auditing admins when money is involved is a bad sign.

Ontheflyflyfly3• 10 years ago

When being accused of bigotry, it's never a good sign to use "social serfdom number" in your post mortem.

tekromancr3• 10 years ago

That's just some shit that ancaps like to say. Also note the constant references to fiat currency. I don't think he was actually being racist with that.

Ontheflyflyfly3• 10 years ago

I agree, I don't think "serfdom" is racist, but it doesn't shine a good light on you to use, and erodes the trust people put in you by default.

buttershakes• 10 years ago

This reads like a case study in pure incompetance at every possible level. Lack of vetting, no third party auditing, poor segregation of customer funds. It's a total shit show. This should permanently damage their business and reputation, but the Bitcoin community has always been forgiving of people who lose their money. Fool me once...

The Looting of ShapeShift

Related Stories