CloudFlare was down

https://www.cloudflare.com

by zhoutong • 13 years ago

129 143 13 years ago

cloudflare.com

xPaw• 13 years ago

Yep, any website using CloudFlare's DNS is not resolving.

Ateoto• 13 years ago

For those of us using cdnjs.com, hopefully you have local fallbacks.

zhoutong5• 13 years ago

It seems that CloudFlare's DNS is down, and affecting NameTerrific as we have a CNAME record pointing to them. I had to change the CNAME record to get our site working again.

EDIT: Based on Twitter search, all CloudFlare sites seem to be down.

tonyjin• 13 years ago

Confirmed, several of my sites are down.

EdisonW• 13 years ago

Was just about to post this. All of my sites are down. :(

Edit: 5:01 EST ..seems to be up and down according to mass pingdom messages.

jgrahamc• 13 years ago

https://twitter.com/CloudFlareSys/status/308154786316963841

- There is a global problem that affects the CloudFlare proxy and DNS services.

- The problem appears to be due to bad routing.

- We are working to restore correct routes in order to bring both DNS and proxy services back online.

- The operations and networking team are all online and treating this as an emergency.

- We do not have an ETA on the response time but will continue to post updates via Twitter as we learn more.

UPDATE. Sites are being restored now. DNS is operating.

I don't know all the details as bugging the network team while they were fixing wasn't going to help. We'll get a postmortem blog post up.

shanelja4• 13 years ago

Just discovered this problem on my company website, thanks for the update.

Seems to be getting better now - intermittent 502's and 504's, the occasional load (speed is choppy though)

Back to near-instant load times, great job guys! I look forward to the blog entry on this (P.S. It might be worth hosting your status page elsewhere in the future, whilst this might not happen very often, it's when it does that your site needs to be working, the lack of redundancy here is startling.)

endijs3• 13 years ago

Even their status page is down. And sure - all my sites too. And funny part is that for most of my sites i have stopped Clouflare features and use just their DNS. Never thought that I will fail because of DNS not being available.

mmahemoff4• 13 years ago

Cloudflare user here. Can I ask why you turned CloudFlare services off?

endijs3• 13 years ago

btw - sites are back. I hope they will not go down.

Why I have switched off services? Because once i enabled them sites got slower, not faster. Sure - I doubt that most users noticed that, but for example, if i checked Pingdom or Google Crawl Stats - 'Time spent downloading a page' situation was very clear. With Cloudflare it took Google 2x more time to download page than without. I had no time to investigate why thats so, but after switching Clouflare off i was again back to 500ms.

Edit: Sites are not back. I guess that work the ones for which i have DNS cached. Lets wait...

aytekin• 13 years ago

This is down as well: http://www.cloudflare.com/system-status

They should host this page on a third party provider.

MichaelApproved• 13 years ago

In that case, it should be http://status.cloudflare.com

tuananh5• 13 years ago

down as well

dbuxton• 13 years ago

Except that I think they are hosting their own DNS... I'm not able to retrieve their SOA information even at the moment.

bromley• 13 years ago

Better still cloudflare-status.com or similar: separate domain, separate registrar, separate DNS, separate hosting.

jelled• 13 years ago

imgur seems to be down for me as well.

mappu• 13 years ago

A workaround: try replacing imgur.com in the URL with filmot.com.

DigitalSea• 13 years ago

Cloudfare may be there when your sites go down, but who is there for Cloudfare when they go down? Nobody it would seem, ha.

mh-3• 13 years ago

including their status page

mmahemoff4• 13 years ago

That part surprised me. I thought it was common for ISPs to not use their own services for critical web presence, i.e. places users might visit when it's down.

Hengjie• 13 years ago

Yeah mine is also down.

vini• 13 years ago

down here too and I can't change the dns of my site cause my registrar is down too, oh boy.

tuananh5• 13 years ago

Olps, major fuck up

Neso3• 13 years ago

You are really lucky :p

tuananh5• 13 years ago

all of my sites are down as well. Should I change name server or this issue should be resolved quick (enough)?

jtchang• 13 years ago

Whatever happened to primary and backup DNS servers?

ryancl2• 13 years ago

And who exactly didn't expect this?

ddaeo5• 13 years ago

Still down. When they get back up, I will cancel my subscription and gladly DDOS them. :D

Neso3• 13 years ago

Our fault, we should have back NS ...

ensky2• 13 years ago

we all expected cloudflare to be more stable than our server ...

overshard3• 13 years ago

As someone who hosts hundreds of PAID sites with CloudFlare this is pretty unacceptable. I'm giving them thousands of dollars so that this doesn't happen. Will probably be moving off unless they have some very good reasoning behind a world-wide shutdown of a geo-redundant service...

robotkad3• 13 years ago

Have you called the enterprise support number? What was their response?

We are in the same boat, but guess where I have the phone number for enterprise support? Webmail on our domain. Lesson learned.

overshard3• 13 years ago

The response is "sorry we are looking into it, we'll call you back when we know what's wrong"... pretty generic.

zhoutong5• 13 years ago

CloudFlare offers CNAME option for paid customers. So you can use an enterprise DNS service and only point to CloudFlare via a CNAME record.

When disasters like this happen, a quick DNS change can be a life-saver.

foobar2k• 13 years ago

There's no such thing as a quick DNS change.

overshard3• 13 years ago

Yep, this is definitely something I'll do in the future and I guess it was my fault for trusting a single service... even one that is designed around keep your site up/redundant.

earless12• 13 years ago

Yup, this CNAME service just allowed me to make a quick update to DNS and get us back up. We will probably need to automate this in the future.

thomseddon• 13 years ago

Would love some details on how to do this, I've been a (business) CF customer for a while now and have never seen this service and I haven't turned up anything googling/scouring their website for it just now.

Details much appreciated!

EDIT: Spoke too soon: https://support.cloudflare.com/entries/22054357-how-do-i-do-... shame it's so convoluted

grose• 13 years ago

Seems like half of the internet is broken just because of this...

oellegaard• 13 years ago

I'm so happy we didn't go on that wave. Redirecting your DNS to someone else seems like a bad idea in any case. In any case, what do they do, that I could not have done with Varnish?

bvdbijl• 13 years ago

A content delivery network?

saurik9• 13 years ago

They are only sort of a CDN; in addition to DNS, they specialize in site optimization via content transformation (a la mod_pagespeed in the cloud) and "DDoS protection" (which is pretty much them replacing your website for new users with some JavaScript that tries to determine if you are a legitimate client).

They don't promise to cache much, and they in fact don't: even on very simple single-page static information websites, such as evasion.com, they have an abysmally poor 81% cache hit ratio. They don't help at all with dynamic content due to having poorly located nodes and lots of heavyweight code running in their proxy. Their lack of many nodes in good positions (compared to something like CDNetworks or Akamai, they are one or two orders of magnitude smaller) also means they can't provide very good latency even for the times when they actually happen to have something in cache.

(Note: if someone is now going to say CDNs don't generally do well with dynamic content, they are wrong: normal CDNs actually improve the performance of dynamic content incredibly by maintaining large-window pre-connected HTTP sessions to customer origin servers, often over private networks that already provide better bandwidth: you can easily see 2x latency improvements with a normal CDN even for fully dynamic content).

So, they really shouldn't be compared with a "CDN": they have an interesting service that actually provides something valuable for many key use cases (4chan comes to my mind: in essence, something that is actually likely to experience a true DDoS attack sufficiently often and with sufficiently little provocation that it makes sense to add an external system to your infrastructure), but if you need a "CDN" there are many more reasonable alternatives that don't have as many moving parts and are thereby going to break much less often (and, if they actually do, should break only in localized regions).

richo• 13 years ago

We're in the process of implementing just that. I'll blog about as much as I can and hopefully publish most of not all of the code.

mmahemoff4• 13 years ago

Bringing content closer to the user; throttling and protecting against certain attacks; providing a response when your site is down (yes, ironic).

Also, randomly useful filters like adding the user's country to the request header, tweaking outbound images, and auto-injecting google Analytics.

ryancl2• 13 years ago

Yes this and increasing load times tenfold.

robotkad3• 13 years ago

Their DDOS mitigation is what sold us.

ck2• 13 years ago

https://twitter.com/cloudflaresys

hahainternet• 13 years ago

It's not just a case of DNS being down, I can't see any BGP either. Pretty major failure.

noveltysystems• 13 years ago

What a let down... Very disappoint.

andyhmltn• 13 years ago

I use a pretty major forum that has a huge amount of traffic. The owner migrated it to CloudFlare. For the past 5-6 weeks the site has 50% of its request go to a 'Sorry xyz is not available right now'

earless12• 13 years ago

You should make sure that the site is not actually returning 500's for those request. We had some similar issues when we first started using the service

fiendsan• 13 years ago

its sunday! seems like they pushed another faulty update (like last time)! yep confirmed, all is down including their own site, thats pretty fucked up, when they dont even have offsite status! good thing i dont use cloudflare in all my sites...

tuananh5• 13 years ago

Seems like the main website is back up

joe5150• 13 years ago

I'm getting alternating 504 and 502 errors for the main website. The server signature is "cloudflare-nginx", so it's definitely getting through, though.

rorrr• 13 years ago

504 Gateway Time-out for me.

tuananh5• 13 years ago

it's back up for me http://monosnap.com/image/pYrdbObsYUhKwAWla5gDzzsg0

ultimoo• 13 years ago

My feels go out to the ops folk at Cloudflare. Mistakes happen no matter how many years of experience people bring in, or how much they're paid. We're all humans after all. It must be a pressurizing task to be responsible for potentially millions of dollars of losses during this downtime.

I hope the issue is resolved soon and if a person caused it, they're not in too much trouble.

moot3• 13 years ago

Amen.

shanelja4• 13 years ago

I guess you're here because 4chan is down?

Something I always wondered - do you use Cloudflare simply for the CDN in respects to photos, or does 4chan also frequently become the target of DDoS and other external attacks?

laumars3• 13 years ago

Human error is unacceptable. Online services should never have a single point of failure which can take their entire global presence offline. Not only because it minimizes the risk of catastrophic accidents, but also because hardware does occasionally break.

This is why every piece of kid should be mirrored with a redundant backup and why many businesses even have entire duplicated standby systems for such disasters. Even if that gear cannot support the entire infrastructure, it's usually enough to at least publish an official status page. Having to use Twitter to update users is just amateurish in my opinion.

vidarh3• 13 years ago

This response indicates lack of experience to me.

No systems are immune to failure. No matter how much redundancy you have, chances are you have interdependencies you did not anticipate, and sooner or later run into failure scenarios that violate your expectations.

It's very well possible that Cloudflare messed up here, but to claim so categorically that "human error is unacceptable" is a bit of a joke. We build systems to withstand the risks we know about, and guess at some we don't.

But the number of possible failure scenarios we don't understand properly is pretty much infinite.

Corrado• 13 years ago

I would hope that they would treat this problem as an opportunity to improve their systems instead of haranguing an employee. All systems fail, the key to success is to turn that failure around and make sure it doesn't happen (the same way) again.

If this was a human error, then change the process to make it difficult to do. If it was a software problem, then they have just found a new set of automated tests to write. If it was a vendor problem, contact the vendor to see how they plan to prevent this problem in the future. If it was a design problem, change the structure of the system to make this type of problem less likely.

Quite a few big names have had similar outages in the past but the ones that I forgive (Google, Amazon) are the ones that talk openly about the issue and outline the changes that they are making to fix them. I'm looking forward to CloudFlare's explanation.

BTW: My boss asked me this week for more information on Railgun and if we should change CDN providers. Railgun sounds incredible and I think it could really help our platform. How CloudFlare responds to this incident is critical to my decision to move forward with Railgun testing or to just forget it completely.

endijs3• 13 years ago

All my sites are now back online! ~ 40min downtime.

Edit: Looks like DNS is back. However if you use CloudFlare services then you might still have problems. Like:

504 Gateway Time-out cloudflare-nginx

ippa• 13 years ago

dns back, now I get bad gateway instead, progress :P

nodesocket• 13 years ago

Seems to be coming back, though intermittent http 502 bad gateways errors.

SchizoDuckie2• 13 years ago

I'm going to say this on the posibility of this being seen as a flamebait... But You should have chosen Akamai over Cloudflare.

It's so funny how everybody jumps on top of new companies that say they can proxy all of the interwebz for a low price. (Cloudflare, Blackberry)

And then they fail...

rdl13• 13 years ago

Akamai won't even talk to people for $0, $25, or $200/mo. I doubt they'd do so for $3000/mo, directly, unless they thought you'd grow.

SchizoDuckie2• 13 years ago

So if you spent just $200/mo on a CDN, Did you really need that CDN at all?

saurik9• 13 years ago

At that $3k/mo level, you can definitely talk to CDNetworks (the CDN I currently use), which is sandwiched between two orders of magnitude of scale, CloudFlare on one side and Akamai on the other. (That said, CDNetworks seems to be much better positioned with regards to China than Akamai.) (That said, I'm actually pretty certain that Akamai would talk to you at the $3k/mo level: have you even tried calling them?)

moe2• 13 years ago

You should avoid Akamai at all costs. They treat you like dirt under their fingernails until you have a 6-figure contract. And even then the experience remains nothing short of surreal...

Report a service problem? Welcome to the indian support-center, where you will be slowly spoon-fed the next best canned response that may or may not match your problem. Escalating to anyone with half a clue is near impossible.

Need a quote? Wait weeks at a minimum for someone to get back to you.

Oh, did we break your reporting? Of course this is your fault, Akamai doesn't make mistakes. Now talk to the hand.

If you're in the market for a CDN then there's plenty candidates who will sell you a better experience for less money (Level3, EdgeCast, BitGravity, etc.).

ksec4• 13 years ago

Um, You could have go with a Akamai Reseller instead? Like Rackspace Cloud?

The same goes for EdgeCast, Mediatemple, GoGrid, etc are resellers.

UnoriginalGuy2• 13 years ago

Akamai isn't as accessible as Cloudflare. For one example I cannot even see how much they charge for similar services, I'd likely have to get a "quote" which means that the whole thing is out of budget for most small-medium businesses.

Akamai's whole web-presents is clearly aimed at enterprise class customers. I mean there are whitepapers everywhere, the site is filled with business nonsense and I cannot even get started without at least a phone call or five.

Maybe people "jump on" to new companies because those companies actually offer a product old companies are not? I can go to Cloudflare right now and sign up, I can give them my CC info and pay, and everything is up-front and easy. I can be online with Cloudflare within the hour (or whenever DNS moves).

saurik9• 13 years ago

I have been using CDNs since I was a very small business (CacheFly, EdgeCast, and now CDNetworks), and if you are seriously using CloudFlare because you haven't even tried setting up a phone call with Akamai due to some kind of mental aversion, you are doing your business a disservice. (I mean, seriously: is the 20 minutes you might waste really that valuable? What's the harm... it might even go well!)

oron• 13 years ago

Going back to go daddy :-(

sairamkunala• 13 years ago

Its back as of 10:51 GMT . CloudFlare and my sites on CloudFlare seem to work again.

Neso3• 13 years ago

They are back online, all my sites active

ensky2• 13 years ago

UP now!!! https://twitter.com/CloudFlare

zobzu• 13 years ago

In my experience, cloudflare has been little more than a scam for anyone with half decent traffic. Not really surprised. Funny how the status page shows all green (are those just static button?) while they acknowledge there is an issue and that they don't know what's going on.

http://www.cloudflare.com/system-status

stenehall• 13 years ago

And it's up again! https://twitter.com/CloudFlareSys/status/308170566760792064

rdl13• 13 years ago

I'm starting to think it's questionable to depend on CloudFlare more than necessary, but they're still the best option for some things. (I'm a customer, but probably going to stop being a customer this week; I was mostly curious to test it out. Not really decided, though.)

1) The CloudFlare security model for SSL basically lets them MITM all your traffic. Probably not a big deal for SSLizing a normal website, or even for accepting credit cards), since they're a decent-sized US company with legal liability, although I'd be concerned about their internal security vs. your own internal security (since you're still fully exposed on your side, too -- it doesn't improve security, and can at best not be a source of new vulnerability).

2) Their DNS doesn't appear particularly redundant; it's just anycast in one big block. Using CloudFlare for DNS seems to be bad practice; you should use something else and cname to CF. Ideally something with multiple DNS servers either individually anycast or in at least two independent (probably anycast) netblocks.

3) Performance of the proxy service seems adequate in my experience but for sites with large amounts of overseas-source traffic, I've heard of people getting lots of suspected-bad-guy path. For a free forum like 4chan that's probably fine; for an e-commerce site, probably not.

saurik9• 13 years ago

Can you say more about "lots of suspected-bad-guy path"? (I do not use CloudFlare currently and am not intending to do so, but I do run an e-commerce site, and have not heard this term used before: the idea of an entire axis along which I have not been evaluating my infrastructure intrigues me.)

rdl13• 13 years ago

If they think you (as an end user accessing a CF site) are a source of DDoS or other abusive traffic, they push your traffic into a sandbox. You then have to answer a captcha before moving on; I assume the bounce rate is incredibly high among most users at that point.

benologist• 13 years ago

I saw their 'bad guy' page many times living in Costa Rica then Nicaragua. I didn't realize what it was until a friend started testing out CF and I got it on his site and then we realized what was going on.

That was over a year ago though, I assume things have improved as they got more and better data (and resources). I was in Costa Rica in January/February then again in August/September last year and didn't come across it I don't think.

I'm in Turkey at the moment and for the last couple weeks and haven't seen one at all, and that's including using an EC2 proxy because of the censorship here.

mooism2• 13 years ago

As an ordinary web user, if CloudFlare suspect you may be malicious (your IP address being on a spam blacklist seems to be enough) they will show you a warning that you should scan your PC for viruses and ask you to complete a captcha.

(They linked to a report on a spam blacklist claiming my IP address had been used to send spam during the past week. I'd had my IP address for about twelve hours. Thanks, TalkTalk!)

As a web site operator, you have the option of switching this behaviour off.

UnoriginalGuy2• 13 years ago

This took out Imgur.com (Reddit's favourite image hosting site) for several hours. So it was definitely felt.

wyuenho• 13 years ago

It took a CloudFlare total wipeout to discover how useless our browsers are against domain name lookups that take a ridiculously long time to timeout.

News flash: CDN fallback like the one below is next to useless unless the first request times out reasonably quickly.

http://css-tricks.com/snippets/jquery/fallback-for-cdn-hoste...

brador• 13 years ago

From the cloudfare business page:

"2500% guarantee This extended Service Level Agreement guarantees 100% uptime, and adds a multiplier to owed service credits resulting from any lapse: 5 times any downtime minutes and 5 times customers affected = 2500% guarantee."

TazeTSchnitzel2• 13 years ago

Ouch.

hu_me• 13 years ago

well they have 785000 sites were down for 60mins so that adds up to... a lot of service credits

robotkad3• 13 years ago

Most of them would be on the free tier

ksec4• 13 years ago

So anyone know what is the services credit for 1 Min of downtime?

packetlss• 13 years ago

Looks like they dropped off the internet:

http://www.youtube.com/watch?v=wMRaKtydILI

AS13335 = Cloudflare

ksec4• 13 years ago

Somebody pointed out about the CNAME available on Cloudflare. I never knew that and i checked out the article. The First paragraph:

"CNAME setup is a manual process generally available to paid CloudFlare plans only. If you are interested in testing CNAME setup, please contact CloudFlare first with the domain you would like to test CNAME with. Please specifically mention CNAME Setup in the subject field for faster review. Allowing for CNAME setup is entirely at the discretion of CloudFlare."

So NO: This isn't even a features at all. They made it as hard as possible to set this up and will grant you the use of it as they like.

CloudFlare was down

Related Stories