How Much Traffic is Too Much Traffic For CloudFlare?

http://phoboslab.org/log/2013/02/how-much-traffic-is-too-much-traffic-for-cloudflare

by phoboslab • 13 years ago

186 79 13 years ago

kevingadd• 13 years ago

On the one hand, I think it's insane that you hit 100TB/mo of usage on a "free" service and expected it to keep going. Maybe you figured you wouldn't rock the boat as long as it was free - I guess I can relate. But still, really???

On the other hand: CloudFlare comes off as terrifyingly incompetent here - which more or less matches with my experiences trying them out on a site serving maybe 100GB worth of traffic at most in a month. They seem to have missed dozens of great opportunities to upsell you on their paid service, and when someone finally noticed how much bandwidth you were using, they completely lost the plot.

What should have happened, IMO, is something like this: A panicked CloudFlare admin realizes your site is using 100TB/mo. Their first step is to send you a sternly worded email, explaining that for this usage level you need plan X, and if you don't upgrade within... let's say 5, maybe 7 days? They'll be throttling or limiting your service. Then you don't feel pressured to solve the problem right away, and they are trying to retain a customer that (presumably) they value.

Instead, they haphazardly change your settings behind your back (???) and then later take various steps to reduce your bandwidth usage before finally deciding that you need to pay them, without even figuring out how much money they want. Ridiculous.

To me this just says that CloudFlare is running on what may be a fundamentally unsound business model, and that by claiming their free tier doesn't charge by bandwidth (and not listing any limits) they're dramatically increasing the odds that customers will suddenly discover there are limits after all, and leave. If they were more up front about what the actual pricing structure is, it'd probably be more likely that people would start paying for the value that CloudFlare gives them.

Cymen• 13 years ago

The author stated a couple of times that they would happily pay for the service. The problem is it was free until suddenly it was $3000/month and it took two weeks with an outage in service for that duration to learn that fact.

aaronblohowiak• 13 years ago

Most reasonable people can be made ok with most things given adequate communication. Expectation management is one of the most important aspects to human relationships. As a business, what was a sales opportunity became a PR issue.

revelation• 13 years ago

The only reason a business like CloudFlare offers a "Free plan" is to attract people to their service and then try to upgrade as many of them to paid plans.

He was happy to pay, they didn't capitalize on that.

dsl2• 13 years ago

As it turns out, a free CDN and free DDoS mitigation is not a sustainable business model.

orangethirty• 13 years ago

They do need to break down their pricing structure in a way that people can easily buy into their service. The big issue here is the sheer amount of difference between the free plan and the lowest priced plan. In order to get people into your monthly billing cycle you must focus on getting them to take little steps. First get them to sign up. Then move them to a free plan. It doesn't matter if its profitable, just get them to overcome the barrier of taking out their credit card. Second step is to get them to upgrade their plan to a profitable one. This is much more easy because they are your customers. No need to pitch them the idea anymore. All you need to do is show them that the other plan meets their needs. You can also upgrade them for free for a limited time (say 3 months) to overcome the barrier of upgrading, and then start charging them at the upgraded pricing. This way they get to integrate into the new plan. They will feel the pain of downgrading, and will mostly decide to not do so. Third, you upsell additional services not included in plans. Use the same technique in the second step. Give them free trials, and start billing automatically once the trial is over. They can always cancel, but most won't bother or want because they have already been using the product for so long.

Realize that people will pay money in order to not lose something. This is a very strong mechanism to use to increase the profitability of your products.

bad_user• 13 years ago

I upgraded to Pro because the Free plan doesn't have the SSL option. The Pro account also gives you more page rules and the caching and response times are visibly better.

iSloth• 13 years ago

Interesting, I have been using their services for a while now but most of my traffic (probably 99%) actually goes straight to my web servers as i'm only really interested in the free DNS hosting service, however looks like i'm breaking the ToS as well, may be time to rethink :S

unkoman• 13 years ago

Where are their sellers? This is just a bad example of how they do not recognize their customers.

danso• 13 years ago

It's not a directly comparable service, but serving 100TB/month over S3 would cost you more than $9,000 a month. On CloudFront, depending on object size and distribution area, it would cost more than $8,000

http://calculator.s3.amazonaws.com/calc5.html

sauteedbiscuits2• 13 years ago

https://google.com/search?q=100tb

There are many cheaper options. The ones you mention seem to be the most expensive.

Uchikoma4• 13 years ago

Hetzner is around $200 for 100TB with 1gbit.

byoung22• 13 years ago

S3 and Cloudfront are targeted at the dabbler, who will likely only be serving a GB or two per month, in which case S3 and Cloudfront would be pennies with no long term commitment. Once you move beyond that and are storing and serving more than 1 TB, you'd be better off with a monthly commitment.

RKearney• 13 years ago

You have to read the fine print though. Amazon CloudFront, while expensive, is designed for delivering content globally.

Take 100TB for example (the first result in that Google Search)

    d. We strive to maintain a high level of service, and a lot of customers
       depend on our high standards of quality. As such, we will not provide
       Services to those that are using our Services for:
    
    vii. Using the Services for a content delivery network or content distribution
         network (CDN). An authorized CDN network offered through 100TB is
         accepted. Special requests to use the Services to run an unauthorized
         CDN network may be approved on a case-by-case basis. Failure to comply
         with this policy will result in termination of this TOS, and you will
         not receive a refund of the Fees.

If you look into the TOS even more, they basically disallow anything that's bandwidth intensive.

dalore• 13 years ago

He mentioned only one which was Amazon. It is expensive but it's doing more than serving 100tb. The S3 provides redundancy and cloud front provides edge delivery.

Getting a server somewhere that has 100tb of outgoing is something different.

papsosouid2• 13 years ago

None of the cheaper options I see there are comparable services. What danso was pointing out was that the service cloudfront was wanting $3000 costs $8000 elsewhere, it is not an unreasonable price. Getting maybe bandwidth from a sketchy dedicated server provider is not the same service as a content delivery network.

sauteedbiscuits2• 13 years ago

Based on what I have read on many HN articles, what Cloudflare offers and what you receive are two very different products.

From being shut off when you have a incoming DDoS of some arbitrary size to actually loading webpages slower than your server does vanilla, it seems the benefits of Cloudflare are mostly hype.

eastdakota• 13 years ago

This was flagged to my attention and I've reviewed all the interactions between the author and our team. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection.

The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network. While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation.

To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.

Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)

ineedtosleep• 13 years ago

I think if the $200/mo tier was officially suggested to them, they would've jumped on that (depending on how miffed they were about the outage). However, suggesting a $200/mo plan which then gets switched to a $3000/mo plan is enough justification to just leave and be done with it.

autotravis• 13 years ago

You keep calling it an attack. This was normal (growing) traffic for the site.

From the comments on the site: "I didn't notice any "attack" when CloudFlare began to route all traffic directly to us. It looked like normal web traffic - much of it, but no more than usual."

kbar13• 13 years ago

keep in mind that the author's words are... his words. Perhaps he's not putting out the full story? How can you know which side is "right" on an issue such as this?

All we know is that the author ran on the free plan, and probably should have upgraded from the free plan when he started seeing his site getting large amounts of traffic.

In the end all is well, he got another service that served his purposes.

thezilch5• 13 years ago

He said; she said. A Layer 7 attack is not necessarily something one might "notice." The very nature of such an attack is "normal" looking. I think it's impossible for us to say who is right -- OP or CloudFlare -- without substantial hard-data. Ultimately, it's not the basis of the article, and the OP is looking for a different service -- unmanaged, limited-downlink bandwidth -- than what CloudFlare is looking to provide -- managed, edge network.

timdorr• 13 years ago

Was the customer contacted when these actions were taken? That seems to be the biggest issue I see. A simple email would have rectified a lot of the confusion here.

UnoriginalGuy• 13 years ago

That doesn't really address any of the articles points. You just re-worded the article.

The article was mostly moaning about communication (or lack of it). So can you talk about how these actions were communicated to the site/client?

moe2• 13 years ago

So they moved to two measly leaseweb servers and the "attack" magically stopped...

http://i160.photobucket.com/albums/t167/vengethis/bullshit_d...

tlrobinson2• 13 years ago

"While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation."

This reads like "We encouraged the site owner to pay 15 times more than they needed to."

Which is it, do they need the business tier, or the enterprise tier?

I've been a happy CloudFlare customer so far, but the lack of transparency in rules and pricing is concerning. You don't charge for bandwidth, but can disable sites at your discretion if it causes problem in your infrastructure? This sounds a lot like ISPs that offer "unlimited" bandwidth but start throttling you at some predetermined but unknown cap.

dlss2• 13 years ago

I don't get this. While it's cool that you don't charge based on bandwidth or TCP connections, it sounds like you basically do when that bandwidth is associated with an "attack".

Doesn't this make CloudFlare customers more vulnerable to attacks, since an attack will result in a monthly fine for the rest of the site's lifetime? (whereas a normal site just pays a one time cost)

cbs2• 13 years ago

To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite

So... what you're saying is that you bill based on traffic?

24kpwn• 13 years ago

And why is 4chan still hosted at CloudFlare?

ceejayoz• 13 years ago

I'd bet they pay, and it makes for great PR too. "We can handle 4chan, we can handle your site!"

phoboslab2• 13 years ago

4chan is an enterprise customer. They pay the $3000/mo.

thezilch5• 13 years ago

4chan (and its users) probably want a fast downlink. The author touts a cheaper provider, but the service they are receiving is an order or more of magnitude slower service from my anecdotal test through wget(s) of images on both OP's and 4chan's sites. That is, a pair of dedicated and unmanaged servers is not, AT ALL, comparable to a CDN's offerings.

4chan probably also expects to have more than an email (and timely) endpoint with which to correspond with the provider of their most core site-service -- image serving.

sp332• 13 years ago

They're not hosted at CloudFlare. CF is basically a caching reverse proxy.

papsosouid2• 13 years ago

Because moot wants 4chan to actually be online occasionally, and to load in a reasonable amount of time. Notice how terrible pr0gramm.com performs, with a tiny fraction of the traffic 4chan gets. That's why real sites can't just get phoney "unlimited bandwidth for $20/month".

Uchikoma4• 13 years ago

Could someone explain the reply from "Matthew Prince" to me?

They turned it off due to an attack and would turn it on again if upgraded to business ($200/mo).

Does this mean turn it on while under attack would cost, and after the attack would be free again? Or it would cost the upgrade to turn it on either way?

niggler• 13 years ago

They claim that other customers were affected, so they stopped handling requests.

The process would have been automatic if he was a paying customer. Because he wasn't, the company felt justified in not trying harder to bring their resources back up.

goldfeld• 13 years ago

One thing I'm yet to puzzle out is whether CloudFlare handles apex domains. They say they do but DNSimple says in reality they just route it straight to your servers so that it's pretty useless.

alax• 13 years ago

CloudFlare does handle apex domains; when you turn the service on, it changes the A records to point at CloudFlare. I don't know where DNSimple is getting their information, you can see CloudFlare working on a naked domain just by trying to resolve the hostname. It'll return two (or more) CloudFlare IP addresses, not the address of your server.

jstalin• 13 years ago

The TOS seems a bit too illusory:

"SECTION 10: LIMITATION ON NON-HTML CACHING

You acknowledge that CloudFlare's Service is offered as a platform to cache and serve web pages and websites and is not offered for other purposes, such as remote storage. Accordingly, you understand and agree to use the Service solely for the purpose of hosting and serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. CloudFlare's Service is also a shared web caching service, which means a number of customers' websites are cached from the same server. To ensure that CloudFlare's Service is reliable and available for the greatest number of users, a customer's usage cannot adversely affect the performance of other customers' sites. Additionally, the purpose of CloudFlare's Service is to proxy web content, not store data. Using an account _primarily_ as an online storage space, including the storage or caching of a _disproportionate_ percentage of pictures, movies, audio files, or other non-HTML content, is prohibited. You further agree that if, _at CloudFlare's sole discretion_, you are _deemed_ to have violated this section, or if CloudFlare, _in its sole discretion_, deems it necessary due to excessive burden or potential adverse impact on CloudFlare's systems, potential adverse impact on other users, server processing power, server memory, abuse controls, _or other reasons_, CloudFlare may suspend or terminate your account without notice to or liability to you."

In other words, you can't host non-HTML, but you can if it isn't disproportionate, but if it is disproportionate, they can deem you to be a problem and cut off your service, without notice. That's not a contract at all. In legal parlance, that's an illusory contract -- when one side can modify their performance in any way at any time.

I use Cloudflare's $20 a month option and it worries me now that I might be deemed to be using a disproportionate about of space or bandwidth caching images, and then be cut off without notice.

EDIT: I love the cloudflare service and I'm not complaining. I just think their legal department needs to clarify this and the tech side of the house needs to be able to warn users when they are exceeding the bounds of what is acceptable.

tlrobinson2• 13 years ago

I wonder what would happen if you embedded images directly in the HTML files (using data URIs or whatever).

derefr• 13 years ago

My interpretation of this paragraph goes something like:

> If and when your bandwidth usage gets high enough that one of our customer service people gets pinged about upgrading you, they'll also have a look on the dashboard to see what mixture of filetypes you're serving. If it's all static content, you're in trouble.

I'm not quite sure how else that could be phrased into legalese, than what they already have there.

trotsky2• 13 years ago

Very interesting article, thanks!

At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget.

An interesting proposition - If we take it at face value, $3000 for 100TB works out to be $0.03 per GB. That's pretty high these days. If you are buying downmarket (which is cloudfare like traffic quality) you can get a cdn deal for maybe $0.01 on a 6 month term with these kind of levels, and somewhere around $0.005 for an xc in the us or eu no commitment. Cloudfare should be buying at substantially better rates than these (or at least, they seem to imply it - calling bandwidth free) so it's it seems they have a similar problem as many freemium models - when most of your customers aren't paying you have to really hit the ones that do.

Terretta2• 13 years ago

Even at $0.01/GBT, you're still looking at $1,000 a month. His conclusion of "two servers and 100 TBT for $200" seems unlikely to last long either.

> somewhere around $0.005 for an xc in the us or eu no commitment

From whom? Or is this reverse engineered from a fully saturated link?

trotsky2• 13 years ago

Not extrapolated from a fully saturated link, no. But I didn't mean to imply that was usage billing.

I was thinking of fdc in amsterdam in the eu on a 10gb unmetered no bw sla, but I also think they offer a similar (slightly higher) deal in denver. I was thinking of somebody else for the US - in SLC - but the name escapes me. If you actually could use the name I'll find it.

I shouldn't have said xc as really they expect you to buy power and space from them. Clearly not what I'd try to run an upmarket video cdn off of, but i would be surprised if people like fdc aren't who CF buys from.

Shish2k• 13 years ago

> His conclusion of "two servers and 100 TBT for $200" seems unlikely to last long either.

A site I'm involved with ended up on Leaseweb rather than a big-name CDN for much the same reason - we have 3 x 100mbit unlimited traffic servers, and all 3 have been pushing an average of 90mbps 24/7 for the past year or so.

It's not quite as smooth and reliable as a proper CDN, but it's an order of magnitude cheaper, and they've never given us any trouble about using too much of their "unlimited" bandwidth.

moe2• 13 years ago

Wow.

Imagine the origin server would have been Amazon S3. The webmaster would have incurred an Amazon bill of $400 USD per day after the switch.

anonymouz• 13 years ago

Slightly OT but related to the Cloudflare business/enterprise offering: They "guarantee" 100% uptime, does anyone know details of this? Obviously there is no way to actually guarantee such a thing, but what kind of compensation do I get when they do have downtime? I can't seem to find it mentioned anywhere.

"Service Level Agreement (SLA) - 100% uptime

Industry standard SLAs often feature 99.999% uptime, also known as the five 9s. At five 9’s your website could be offline for as long as 5 minutes and 26 seconds each year. All CloudFlare Business and Enterprise plans offer guaranteed 100% uptime because we know that anything less than 100% is an impediment to your organization’s success."

moonboots• 13 years ago

Usually these guarantees mean credits toward future billing. Cloudflare's enterprise plan offers 2500% uptime guarantee, which means if they are down 10% of the time, customers are credited 2.5 months.

It sounds a lot nicer than it is, much like "we never charge for bandwidth".

codexon3• 13 years ago

You couldn't monetize it yet you can spare $200/month?

dangrossman• 13 years ago

I've rented servers for high-traffic websites with no income for years. If it's something you love to run, and you can afford it, then it's just the cost of your hobby. Not everything is a business.

codexon3• 13 years ago

If he is as really rich as you suggest it seems odd to me that he would complain about Cloudflare and not have rented a 100 TB server sooner. They are not exactly difficult to find.

Whether he can tell us if he is rich or found another way to monetize adult or potentially offensive content, it is up to him to share.

raylu• 13 years ago

While I agree CloudFlare handled this poorly, there is this reply on the blog from Matthew Prince (CEO of CloudFlare):

Both enabling "I'm Under Attack Mode" or routing the traffic direct are both supposed to generate an automated message to the customer letting them know what happened. We've reviewed the logs and don't see a message having been sent. I'm investigating why that didn't happen since I agree it is not acceptable.

thomseddon2• 13 years ago

I had quite a similar experience where our site (approx 10TB / month) was taken "off cloudflare" and all traffic routed direct to our servers at peak load (we were on the Pro plan @ $10/month after many months of free). They cited network issues in the control panel and a "Layer 7 attack" in a support ticket. We quickly upgraded to a Business plan ($200 / month) and traffic was back through cloudflare within 10 minutes.

Also, similarly to OP, we are regularly (fortnightly) automatically put in "I'm Under Attack" mode without any prior warning or consent which is quite annoying as it tends to happen overnight so I am not alerted until someone checks the live site in the morning (it still returns a 200 so current checks don't pick it up)

dlss2• 13 years ago

Sounds like the $200/mo is really worth it...?

thomseddon2• 13 years ago

We use it mainly for the CDN, we have estimated a similar service using CloudFront would cost over $1500 / month (750% more - I acknowledge AWS is are not the cheapest)

vilda• 13 years ago

Why is it that you'll always get the worst posible review by a non-paying customer?

This is a common observation - less the user pay the worse and less objective the feedback is.

cbs2• 13 years ago

There are a handful of contributing causes, just off the top of my head:

Once you get past the first few tiers of paid service, providers tend to reign in the bullshiting "technically true by widely misleading" descriptions and are upfront about what they provide. Properly managed expectations don't lead to as strong of emotions when a customer finds out what they're getting doesn't work for them.

Commercial users will save face by A) avoiding discussing anything that could make them look bad, ditching a vendor is a tacit admission that you went wrong when you chose their service. Additionally, if their service is built on top of the ditched service, it casts quality concerns on their product too. B) Avoiding disparaging anyone, ever, is pretty common play-nice-save-face.

Why should I give my competitor helpful advice? I got caught up in a web, I don't need to help the competition avoid my mistakes.

In a multiple person organization, the party angry about the service and the party who decided to buy/use it are different individuals and airing grievances/venting is an internal process that they don't reiterated in public.

In a free service, a failure to deliver is the only downside. If I paid for a service, I'm going to be too preoccupied beating myself up for making a stupid purchase decision and looking for an better alternative than bothering to share my findings.

These causes and more are all often balled up and interpreted as free users acting "entitled". Sure that exists, but its a classic case of reading too much into imprecise metrics when someone assumes the cause.

Furthermore, the feedback from free users isn't inherently worse, and is just as subjective as that from paying users, it just tends to be more negative. For all we know, any given product could objectively be shit and because only rubes will buy it, the free users are the only ones that can let you know.

throwawayG95• 13 years ago

Thanks a lot for this article.

Were you already using http cache of 1 year for all the images when this happened? Do you think it could have been avoided by setting it to 1 week, 1 day, or even 1 hour?

It's funny that I have to ask this to you instead of asking Cloudflare. They have really messed up on this one.

notlisted3• 13 years ago

This kind of whining and the comments on the page piss me off greatly.

A customer is a person who pays for a service. Someone who doesn't pay for a service (yet) is a lead. Not all leads are good business. 100TB of traffic does not sound like a good lead to me, not even at the $200 level.

Looking at the site, I see an IMGUR clone which was running for free off of CloudFlare's cache servers. I really don't understand the nonsensical comments on the article. WTF is wrong with people these days thinking that everything is supposed to be free? Are you all 16 and on an weekly allowance?

Commenter Matt had a very valid point that some sort of optimization of the stored (cached) files would have been a smart option for yourselves (less local storage) as well as CF (less to cache, less bandwidth). I'd recommend http://www.jpegmini.com/server (Oh wait, it's not free, now what... cry me a river)

I have a question for the Phobos peeps. Were you making money? Seems to me like you were... since you can afford the LeaseWeb servers. Instead of bitching publicly, perhaps you should have reached out to the company when you noticed your traffic levels were reaching antisocial magnitudes.

Grumpy Gramps (who used to pay UUNET $6k+ a month in 1997 dollars for the privilege of hosting a basic database-driven e-commerce site for a luxury watch brand on a guaranteed T1 connection)

throwawayG95• 13 years ago

They weren't making money, and they did contact the company. Comments like yours piss me off greatly.

notlisted3• 13 years ago

They contacted the company after being shut down. Not before, when it should have been pretty clear to them how many resources they were using. There's a simple indicator in the dashboard. "BANDWIDTH SAVED".

This line in the article "I tried to monetize it through ads some time ago, but failed. Advertising-Networks don't want us as a customer, but I'm fine with that." was not conclusive that monetization failed via other means. Not a user of the site, but at first glance some choice links to pr0n affiliate sites would do the trick.

Throwaway ey... hmmm.

ghshephard• 13 years ago

You get what you pay for. When I'm paying for a $16/megabit @95th on a 10 Gig port, and I get hit with a DDOS, it's my providers problem, because I'm not paying $16/megabit for DDOS traffic, so they need to stop that stuff before it gets to me (ideally without impacting my customers)

Contrawise ,if I'm paying $2-$3/megabit @95th on a 1 Gig Port, the amount of support I can expect during a DDOS is pretty minimal, so I end up having to take the hit - but my damage is limited to $3000/month so I don't really care.

Any time I see a "We don't charge for Bandwidth" service, I interpret it to mean one of (A) We'll throttle you once you exceed our unspoken limit, or (B) We'll discontinue your service. (Drop your port from 1 Gig down to 100 Megabits, or slower, traffic shape you, etc...) once you breach that limit.

There is no sustainable third option for those who provision reasonably high quality transit, and those who believe there is will one day wake up with their internet property offline, or seriously degraded.

How Much Traffic is Too Much Traffic For CloudFlare?

Related Stories