The Critical Security Flaws that Resulted in Last Friday's Hack

http://blog.cloudflare.com/the-four-critical-security-flaws-that-resulte

189 48 14 years ago

blog.cloudflare.com

Wow - that's some epic transparency. Kudos to Matt and the Cloudflare team for that.

ejfox• 14 years ago

I like the infographic showing the series of events, definitely goes a long way in terms of aiding transparency. I wonder if it could be a new trend?

dfc2• 14 years ago

I would love to know if the graphic was made by hand or in an automated fashion.

eastdakota4• 14 years ago

Made by hand by Kevin our great graphics designer. Was up late very last night putting it together. Filled in the last few details this AM.

citricsquid• 14 years ago

I too like it but it feels... I'm not sure how to describe it, but I felt like I was having to concentrate to read it properly and keep track of what point I was at. I'm not sure if that's a problem with me or the format, but trying to read it properly was less enjoyable than just scanning it.

aaronbrethorst• 14 years ago

Me too, especially in light of the posting on here the other day criticizing wannabe infographics. Now this is an infographic worth the name.

j_s• 14 years ago

Rule #1 when publicizing security incidents: always publish something else to the blog within 1.5 hours so that the security incident isn't the top post.

Edit: semi-tongue-in-cheek per comments below; as a CloudFlare customer I went to the blog when this first came up expecting to see something but bounced when the first post was a discussion of SSL BEAST since that was the hotness back in the fall of 2011.

I do believe it was not planned, but I also feel that vulnerability disclosures should be pinned for a while somehow if possible. I think one way this is done is having a separation between 'new feature' blog and 'ops' blog.

ohgodthecat• 14 years ago

That seems kind of shady, I mean sure it sucks that it happens and it is kind of bad to have it as your top post but publishing just because you don't want it to be the latest post kinda downplays the publicity factor of being honest about what happened.

eridius• 14 years ago

I'm sure j_s's comment was meant as tongue-in-cheek and not as an actual criticism. I have every confidence that eastdakota's response (http://news.ycombinator.com/item?id=4066982) indicating that it was just a coincidence is true.

eastdakota4• 14 years ago

Haha. Wish we were that organized. We have a big announcement on Wednesday and need the announcement about Polish (http://blog.cloudflare.com/introducing-polish-automatic-imag...) and the feature we're announcing tomorrow (Mirage) to come out before then.

I think being among the top stories on Hacker News will take care of people seeing it. And, for the record, I voted the breach story up.

dguido• 14 years ago

Hey, just wondering, did someone make that infographic by hand or is there some software that does it for you? EDIT: Made by a graphic designer, answered below.

And now I will shamelessly take this moment to request a few features related to account security :-).

* Alerting: SMS or e-mail notification when an unrecognized device logs into my account or when records in my domains change.

* 2-factor Authentication: Prompt for a code delivered via SMS, e-mail, Google Authenticator, or DUO Security to login from an unrecognized device.

* Login Accounting: Let me see what IPs logged into my account, when, geoip info for each, and preferably what actions they took while logged in. Provide an API for this info so I can write an automated script to analyze it for suspicious events.

If you end up making any of these features, it would be cool to open-source a library you used to do it. There are a bunch of large SaaS providers out there that use features like these but they're all homegrown implementations afaik.

Btw, the Google Apps Admin Audit API exists but I have never seen anyone do anything with it and it makes me sad. A few hours with [name a scripting language] and you could probably have a pretty robust Google Apps monitoring system, but no one seems to care: https://developers.google.com/google-apps/admin-audit/get_st...

rdl• 14 years ago

Rule #1b: always get hacked the week of WWDC so nothing stays in the news :)

Bob_W2• 14 years ago

The hack seems very well planned - I wonder just how many smaller sites have been hacked the exact same way as practice, and not picked up since they didn't have direct access to Google's security people?

ceejayoz2• 14 years ago

I'd imagine you wouldn't blow a discovered flaw in Google's two-factor authentication setup on a small site. You'd sit on it until a big enough target came along, which CloudFlare certainly has become.

Bob_W2• 14 years ago

I don't think you'd be blowing it if you have a reasonable idea that the site owner just isn't big enough to get Google to return their calls.

Mizza• 14 years ago

Incredible hack! Hats off to the hacker and to Cloudflare for the transparency of the response.

What customer was the target?

neilwillgettoit• 14 years ago

Another poster on r/netsec (https://pay.reddit.com/r/netsec/comments/ui0k4/cloudflare_wa...) mentioned that it was 4chan.org.

startupfounder• 14 years ago

"AT&T was tricked into redirecting my voicemail to a fraudulent voicemail box"

Capitan Crunch called...

aSig• 14 years ago

Does anyone else get the feeling that the attacker is going to be someone the Cloudflare team knows? Firstly they would have had to have known Matthew's phone number. Then, assuming the attacker always had the plan of disrupting the target site, they would have had to have known that the password reset mails were BCC'd to admins.

eli• 14 years ago

Getting someones phone number seems pretty insignificant compared to using a previously undisclosed google security flaw.

And it's probably safe to assume that once you control the admin email account for a site, it's game over. You could request resets from other providers

jakejake• 14 years ago

That's what I thought too. With only 2 hours in the system they seemed to know a lot of details about internal stuff.

madao• 14 years ago

The fact that they are relying on an external mail vendor, and had passwords in their emails is a very sad practice.

ams6110• 14 years ago

I'm guessing that they didn't have passwords, but were BCC-ing the password reset emails/links to the gmail account.

But yeah... I don't like the idea of using Gmail for this either.

sp332• 14 years ago

O.o This reminds me so much of the hack sequence from the game Uplink. The game was based on "hacking" but intentionally used hacking techniques from Hollywood :) It was pretty fun. Anyway for the highest-level targets, you had to get a voiceprint from the phone of an admin, crack the password on the box, and break the encryption while bypassing monitors.

Just found out: Uplink is on Steam, and in the Ubuntu Software Center now.

techinsidr• 14 years ago

http://www.securityweek.com/exclusive-google-two-factor-auth...

metafour• 14 years ago

Would most of the attack been rendered impossible if Matthew answered his phone at 11:39 instead of letting it go to voicemail?

cobbal• 14 years ago

I'm not sure of the details here, but it wouldn't be too hard to make sure the reset call arrived in the middle of another call.

eastdakota4• 14 years ago

That makes a lot of sense and could have been what happened. It would also make it more difficult for Google to do something like ignore responses that come after 4+ rings.

sc00ter2• 14 years ago

Is flaw # 5 (or #1 depending on how you look at it) not having two-factor auth on the personal account? Or does account recovery by-pass two-factor auth by design?

sc00ter2• 14 years ago

Part of this attack bears a certain resemblance to the recent Bitcoinica compromise, in that, from what I understand, they were also forwarding admin emails to personal accounts, one of which was compromised leading to the attacker gaining control of the bitcoinica virtual servers. Cloudflare were fortunate - at least the changes made were reversible, whereas the bitconica compromise resulted in the virtual servers being unrecoverably deleted after breach and theft.

There are lessons to be learned from both incidents.

saturn• 14 years ago

Sounds like if the hacker had just done it out of hours, perhaps when the person in question was asleep, they would have had uncontested access to the accounts and the hack might have been far more damaging.

BryanB552• 14 years ago

Pretty intense hack... I think Google gives you the option of setting up a recovery phone to receive voice or SMS messages. It looks like SMS may be more secure.

It also sounds like he didn't have 2 factor setup on his personal gmail account. I wonder if that would of helped.

gwillen• 14 years ago

I believe that once you have it set to use a phone number, it's always possible for the attacker to choose to have it call instead of sending a text. (I seem to recall it provides that option on the 2-factor login screen.)

jsprinkles• 14 years ago

I'm amazed Cloudflare got a response from Google that quickly. I'm a paying Apps customer and I don't see responses for 24 to 48 hours on security incidents, not to mention that Google doesn't have a stellar reputation when it comes to things like "support".

Goes to show, it's always who you know (or it's bullshit, which is less likely). Or I don't have enough users.

jrockway• 14 years ago

Paying Apps customers can call Google and talk to someone immediately.

kijin• 14 years ago

Why does Google allow the hacker and the account owner to keep resetting passwords in rapid succession? The timeline indicates that two lengthy ping-pong sessions took place during the incident. That kind of behavior should immediately raise a red flag. How often do legitimate users reset passwords alternately from two different locations 10 times in 15 minutes?

I'm surprised that Google doesn't detect two people fighting for control of one account. They could have easily detected ping-pong sessions and and locked both parties out of their accounts for a couple of hours. Or they could have penalized the newly added recovery address by forcing an exponential delay between resets using that address. This is not the first time I've heard of somebody breaking into a Gmail account while the account owner is using that very same account.

manmal• 14 years ago

I was in the same situation about 1,5 years ago, when some security hole in Gmail allowed my account to be hijacked. The account was suspended automatically after the attacker had sent out 7 spam mails, and I could reset the account again without the attacker returning ever again. But, the ping ponging did go on for a while and Google did not bother.

lucb1e• 14 years ago

Good idea I think! Lock it after 2 account changes for a few hours at the very least, then the original owner will have it back, who is most likely the legitimate owner.

BryanB552• 14 years ago

I would just be worried about how they lock us out, what if the block both of our IP's and then the hacker manages to change his IP and can now get in and has an advantage over the legitimate account holder.

gojomo• 14 years ago

So, the "Five Whys" analysis came up one short, eh?

Just kidding, this is a great level of detail and much appreciated, to understand CloudFlare's process and how to protect against or recognize these tactics elsewhere.

slavak• 14 years ago

Great job on the transparency.

And now I know about the Google Authenticator app. Fancy little thing, that; glad to find out about it.

radicaldreamer• 14 years ago

Hrm http://exiledonline.com/isucker-big-brother-internet-culture...

lsh123• 14 years ago

The funny part - cloudflare still hosts the DNS for the guys who claim responsibility for this attack:

   Domain Name: UGNAZI.COM
   Registrar: ENOM, INC.
   Whois Server: whois.enom.com
   Referral URL: http://www.enom.com
   Name Server: LEE.NS.CLOUDFLARE.COM
   Name Server: RUTH.NS.CLOUDFLARE.COM
   Status: clientTransferProhibited
   Updated Date: 29-may-2012
   Creation Date: 22-jan-2012
   Expiration Date: 22-jan-2013

The Critical Security Flaws that Resulted in Last Friday's Hack

Related Stories