Debunking Cloudflare’s recent performance tests

https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests

by mcone • 5 years ago

556 275 5 years ago

gopalv• 5 years ago

   > _We used a Wasm binary compiled from Rust rather than JavaScript.
   > We know support for JavaScript is important to many customers, but we're not yet satisfied with the performance of Compute@Edge packages compiled from JavaScript. That's why it's in beta. When a product is ready for production, we remove the beta designation.

I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration.

Having said that, I would notice anything over 150ms in my clicks, but wouldn't care whether something took under 100 or under 50 - except that the lower the latency the less the scale needed to serve the same number of active users (it becomes a question of cost rather than response time).

kailanb4• 5 years ago

It definitely comes down to the language — JavaScript is an interpreted language, which requires an engine. Rust is statically compiled and runs by itself.

azakai3• 5 years ago

In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language.

Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).

vlovich1233• 5 years ago

On Cloudflare, JavaScript is a JITted language, not strictly interpreted (it uses V8). Code that's seeing any amount of traffic gets lowered into native code pretty darn quick & V8's profile-guided optimizer is pretty impressive at removing JS overheads.

Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure.

The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute.

Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).

acdha3• 5 years ago

I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration.

It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible:

https://bytecodealliance.org/articles/making-javascript-run-...

The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform.

I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps.

1. https://github.com/bytecodealliance/wasmtime

redm• 5 years ago

I think statistics like these tests are easy to sway in your favor, hence why Fastly is responding. Its not clear that Fastly didn't do the same thing, but a good ole' CDN rivalry does make for a good read. Someone get more mud to sling.

deft• 5 years ago

point #2 is silly. So basically your product is slower but its ok cuz its a beta only... Fine. But then cf did not mislead anyone. Your beta is slow.

invisible4• 5 years ago

Not really. Cloudflare could have used their rust implementation for doing a fair comparison, instead they compared a beta product to a stable product. It's not like CF didn't have a choice to make a fair comparison.

etchalon• 5 years ago

Did CF pretend they were benchmarking something other than the product they said they were benchmarking?

hokkos• 5 years ago

how comparing rust to javascript is even remotely fair ?

jjeaff• 5 years ago

As mentioned in the article, Cloudflare expressly prohibits benchmarking their services in the tos.

I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.

charcircuit• 5 years ago

They don't prohibit it outright. They just require you to ask first. The article made no mention of if they asked or not.

deadmutex2• 5 years ago

Why tho?

girvo• 5 years ago

They don't prohibit it outright. They just require you to ask first

Frankly, that's a distinction without a difference in this context, at least in my opinion.

jchw• 5 years ago

I feel like that's splitting hairs. if you say something is "prohibited without express written permission," to me that reads as "prohibited." Especially when it comes to benchmarking, because it effectively means that you can never do an impartial benchmark of the service.

InGoodFaith2• 5 years ago

Asking before benchmarking seems to be a sure way to poison your results.

Who's to say that the performance of the services won't coincidentally perform better after you've given your request to test?

I am sure they might mention some legalese of "avoiding putting arduous strain on the services which may cause disruption to our other users" as a justification, but that PR speak won't fly with me since there is a separate section that already references that.

I say this as a big fan of Cloudflare and I am sure they don't need to use such underhanded tactics to make their services look good.

judge20202• 5 years ago

In contrast to the other comments, this is a valid concern regarding whether or not they asked first. I assume legal cleared this blog post, which would likely infer that legal either (A) got permission to publish benchmarks, or (B) doesn't think that part of the TOS is enforceable/doesn't expect CF to actually do anything about it.

The TOS clause, for reference: https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or... (ctrl-f benchmark )

barkingcat• 5 years ago

In a totally different scenario - if someone requires you to ask before undergoing a drug test, for example, if an Olympic athlete made drug testing conditional upon having notice, it wouldn't be a real test.

Dylan16807• 5 years ago

"If you get permission to bypass this, you can bypass it" is implicitly part of all agreements. So explicitly including that doesn't change it. It's the same as an outright prohibition.

austhrow743• 5 years ago

Every prohibited thing is not prohibited if you get an exception. Nothing is prohibited huzzah!

wraptile• 5 years ago

I'm not a lawyer but could someone explain to me could you be legally persecuted for this? It's "Terms of Service" - I broke the terms by performing the benchmarks and Cloudflare no longer provides service to me. Lets say I'm a scientist and I'm fine with never using cloudflare services but this benchmark is important for my field - would cloudflare have a case here? I'd imagine that would be quite absurd but corporate law very much already is.

kingcharles4• 5 years ago

Yes. Here in Illinois violating terms of service carries up to 5 years in prison.

https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...

gary_03• 5 years ago

The feds ruined Aaron Swartz's life for violating "Terms of Service", so people are right to be fearful.

https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#A...

dathinab• 5 years ago

Depends on the country you are in, in the US, thinks seem bad. See the other posts.

In the EU there are various ways of ToS are limited in their power, if that applies here I'm not sure.

But they are basically forbidding you to evaluate how well a product you (might be) paying for works, depending on what they advertised they might even prevent you from evaluation if the advertised product characteristics uphold. Especially in the later case I guess that the ToS clause would be void.

lelandfe• 5 years ago

Anti-benchmarking clauses are sometimes called DeWitt clauses after Oracle famously tried to get a professor fired for testing their offerings. https://danluu.com/anon-benchmark/

I never knew how true that anecdote was but that’s typically the boogeyman story that gets passed around for things like this

hn_throwaway_993• 5 years ago

Note the CloudFlare CEO subsequently commented elsewhere on this post that they would remove the problematic language from their ToS: https://news.ycombinator.com/item?id=29468771

jbergstroem2• 5 years ago

In the context of "should I use Cloudflare or Fastly as my edge", I lean Cloudflare. That said, I enjoyed listening to the GraphCDN crew choosing Fastly[1] – one crucial feather in the hat being cache invalidation[2] (miss ya, Phil Karlton) – and it sounds like a solid choice.

[1] https://www.youtube.com/watch?v=lpmpTJc_SP0 [2]: https://graphcdn.io/docs/how-to/purge-the-cache

tylermenezes2• 5 years ago

I've used both Cloudflare and Fastly for many years, and the cache invalidation at Fastly is a way bigger deal than people think. They're not competing for a 5% improvement in performance, they're competing with a totally different set of features which mean you can cache 80% more things, as long as you design for it. (Which I'm sure is why they don't aggressively compete on free plans -- if you try to just throw it in front like Cloudflare you won't see much of a difference from Cloudflare.)

With some careful design to take advantage of it, I've gotten 5-10x faster page loads using Fastly. (The last time I pointed this out, people accused me of being a sockpuppet account. Feel free to Google me!)

sadsoul212• 5 years ago

I'm curious, what "totally different set of features" are you referring to?

jbergstroem2• 5 years ago

I can definitely see how new use cases opens up based on a 150ms (global) invalidation. Most of my use cases relies more on local invalidation and I haven't done enough homework about on what latency I really need (read: wrong line of business). That said; in the world of Cloudflare I think durable objects/kv store probably would come to serve as the next architecture for cache itself and lead to reductions in latency. Don't quote me on that though.

snowwrestler• 5 years ago

Yes. Caching is a different animal when you can count on fast, reliable, targeted invalidation. You can cache basically everything with super long TTLs, and just sniper-shot individual cache objects as soon as the backend updates.

lambdadmitry2• 5 years ago

Keep in mind that Fastly's C@E might not even be needed, as you get VCL as standard. You don't check pre-made boxes, you write a non-Turing complete but quite powerful language that gets atomically deployed to their whole CDN in a few seconds. I'm a big fan.

breakingcups• 5 years ago

Cloudflare not allowing benchmarks in their TOS is very sketchy, that puts them in the same tier as Oracle.

Cloudflare have pulled enough shady stuff now that they've fallen out of my favor. Their generous free product bought them a lot of community goodwill but their real face has been showing the past few years.

pm902• 5 years ago

I would give them a bit more wiggle room before giving up. We’re currently living under a monopoly of AWS (GCP is the most serious “competitor”, still far far behind in market share) and I would love to see more competition in this space.

That said: this is definitely not the kind of behavior I expect from them and I’m hoping there’s a better reason than them not willing to submit their products to an impartial evaluation.

varsketiz• 5 years ago

You are forgetting Azure, which is much bigger than GCP.

latch• 5 years ago

AWS is a monopoly if you're living in a HN bubble. The vast majority of "hosting" (internet or otherwise) doesn't use AWS. Large telco's (Telefonica, AT&T, China Unicom, ...) have numerous data centers. Conglomerates (Tata, ...) have numerous data centers. You have wholesalers (Equinix, Digital Realty, ...). You have dedicated providers (OVH, Hetzner, PhoenixNap, ...). VPS' (DO, Linode, Vultr,...) and shared hosting (GoDaddy, ...)

Then you have a lot of private data centers, e.g. I worked at a major bank that had its own datacenters (multiple around the world). Admittedly, these are super small compared to an AWS DC, but there's literally thousands of these around the world (I know for banking and government, but I assume other industries do this too).

goodpoint3• 5 years ago

Wrong. Cloudflare introduce another type of monopoly on top of the existing cloud services.

A very harmful one - they are trying to centralize the web.

And kill Tor, while at it.

tzs• 5 years ago

Cloudflare not allowing benchmarks in their TOS is very sketchy, that puts them in the same tier as Oracle.

There's a non-sketchy way for a company to do that kind of thing which avoids putting them in Oracle territory. Simply require that anyone publishing benchmarks (1) publish complete configuration information sufficient to allow the company and other interested third parties to reproduce the benchmarks, and (2) allows benchmarking of their products that they are benchmarking against yours under similar terms.

It is nicer of course to not have any benchmarking restrictions, but if you do that you are putting yourself at a disadvantage to companies like Oracle. They can benchmark against you put you can't do the same against them.

Once one important player in a market goes down the Oracle route others tend to follow. But if they aren't asses they will follow with the kind of restriction with reciprocity I outlined above instead of an Oracle-like restriction.

throwaway2037• 5 years ago

I like your ideas, but they seem difficult to enforce. It assumes good faith on all sides. One of the biggest complaints about AI/ML research results: It is frequently hard/impossible to replicate the results.

One idea: The edge competitors can create a public (SourceHut?) project that runs various daily tests against themselves. This would similar to JSON library benchmarks. [1] Then allow each competitors to continuously tweak there settings to accomplish the task in the shortest amount of time.

Also: It would be nice to see a cost analysis. For years, IBM's DB2 was insanely fast if you could afford to pay outrageous hardware, software license, and consulting costs. I'm not in the edge business, but I guess there are some operators where you can just pay a lot more and get better performance -- if you really need it.

[1] https://github.com/miloyip/nativejson-benchmark

merb• 5 years ago

btw. for what it's worth their javascript to wasm is opensource:

- https://github.com/fastly/js-compute-runtime

- https://github.com/tschneidereit/spidermonkey-wasi-embedding

and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help)

its insane what they built, to do it and it will bring the whole wasm community forward if they succeed (and I hope they do)

NicoJuicy6• 5 years ago

Gentle reminder: That's mostly because they acquired the team behind wasm. Which was already an opensource project at Mozilla.

Post on HN: https://news.ycombinator.com/item?id=24897641

tschneidereit2• 5 years ago

It's really the other way around: we joined Fastly because we knew it's a place where we could do this kind of work in the open.

None of the code involved here existed a year ago, and none of it was somehow forced to be open source. (Also, the code in these two repositories is in many ways the most boring part of this solution. See this blog post for an excellent overview of the more interesting parts: https://bytecodealliance.org/articles/making-javascript-run-...)

kentonv12• 5 years ago

OK...

So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.

But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number?

We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons...

On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.

refulgentis• 5 years ago

I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all.

This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.*

A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened.

* intentionally vague here, in order to give you space to delete and leave no record

ethbr03• 5 years ago

I found parent interesting and relevant.

But mostly importantly, kentonv's last paragraph.

At the end of the day, benchmarking for your use case is the only true north. Everything else, on all sides, is marketing fluff.

brightball• 5 years ago

Fwiw, I don’t have an issue with it. It’s good information.

NicoJuicy6• 5 years ago

You're implying that he did something wrong.

But the title of the post validates his opinion and his claim about fastly and cold starts can literally be found anywhere in their blog and the compute@edge landing page.

I think it's a valid and valuable comment.

argc• 5 years ago

I don't understand why you're arguing for the parent comment to be deleted, it's useful to hear from someone, (especially an engineer and the tech lead) who is working on the product. I hope that the comment is not deleted.

unityByFreedom16• 5 years ago

their post doesn't come off like mud-slinging, at all.

The post's current title is Lies, damned lies... and the article's synopsis is,

This nonsensical conclusion provides a great example of how statistics can be used to mislead. Read on for an analysis of Cloudflare’s testing methodology, and the results of a more scientific and useful comparison.

That is definitely mud-slinging.

duxup• 5 years ago

I think the comment you were replying to was just fine. They have opinions about another service, I think that’s ok.

Aeolun• 5 years ago

their post doesn't come off like mud-slinging, at all.

Sure felt like mud slinging to me. Just the whole list of complaints they had at the start of the post was hard to get through.

They could just tell us what they did differently and show the result.

hereforphone2• 5 years ago

Agree with the other comment. Unless you have explicit permission I'd be careful what you post here about your employer

NavinF• 5 years ago

Did you notice his username? I’m pretty sure he has permission from himself lol.

yellow_lead• 5 years ago

So, I would encourage everyone to run your own test and don't believe what either side posts.

I don't have the full TOS in front of me but wouldn't this violate it?

howdydoo• 5 years ago

I was about to disagree, but I just re-read Cloudflare's language, and you're right. Cloudflare forbids performing any tests or analyses, whether or not you publish the results. Even running something like `curl http://my-workers-site | time`, for your own personal curiosity, is in violation of their ToS. That's even more egregious than Oracle DB's infamous DeWitt clause.

Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)

flumpcakes2• 5 years ago

I've been saying for decades that benchmarks are almost always meaningless

I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred.

I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.

kentonv12• 5 years ago

Oh absolutely, benchmarks are great for measuring your own performance and improving it.

I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.

1cvmask• 5 years ago

Unfortunately not all companies follow your great obvious recommendation and listen to snake oil salesman dressed up as consultants and industry analysts:

So, I would encourage everyone to run your own test and don't believe what either side posts.

---

Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.

nicoburns• 5 years ago

Sounds like you might be in a position to get some of these things changed and pave the way for fair benchmarks to take place.

ksec4• 5 years ago

Personal Opinion only.

I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS.

Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different.

Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare.

But I agree everyone should run their own benchmarks.

dcow3• 5 years ago

> So, I would encourage everyone to run your own test and don't just go on other people's numbers.
But I agree everyone should run their own benchmarks.

But they literally can't because the ToS forbids it.

kentonv12• 5 years ago

The material commonly mentions "isolates" or V8 taking 5ms to cold start, which seems clearly aimed at Workers.

But the same point applies when comparing against Lambda. It's not a cold start if the app code is already resident in memory in advance. Workers and Lambda also proactively load some apps in advance, and we don't call it a "cold start" in those cases.

dwwoelfel5• 5 years ago

It has been two weeks since Fastly's VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly's claims.

Are you going to remove the ToS clause or issue a correction on the blog post?

[1] https://twitter.com/lxt/status/1462896850055352320

kentonv12• 5 years ago

To be clear my comment here is not in any way, shape, or form, Cloudflare's response. I am here representing only myself.

staticassertion2• 5 years ago

I wouldn't expect everyone at Cloudflare to be intimately following every competitor's blog posts tbh

NicoJuicy6• 5 years ago

If you know a little about how corporations work, than a tech lead is not responsible for a ToS, probably doesn't know anything about it and that's fine. Since it's not expected either.

It's the legal department... And the CEO already mentioned that they are removing it and he gave a valid response/reason.

It seems that they will actually benchmark Fastly in detail now ( could be after another improvement week), which probably isn't what Fastly wanted.

Something definitely seems to be happening if you read their response and i'm awaiting it with actual stats!

https://news.ycombinator.com/item?id=29468771

@dwwoelfel that's what you wanted? :)

tiew9Vii2• 5 years ago

Fastly has been trumpeting for years that Compute@Edge

This is the main problem with Compute@Edge, "for years".

They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.

After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime.

Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed.

Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.

numair• 5 years ago

They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.

This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on.

(I'll note that this is separate from the problem of AI-based fraud detection run amok.)

kennethh• 5 years ago

I totally agree with this, Fastly makes it pretty obvious from their website they do not want small customers. There is no price list and the prices on domains and such is 10x Cloudfare. It is also difficult to se what type of services they provide, everything ends up in a contact us. They seem to have great product with Compute Edge but no free thier and fuzzy pricing.

If you read the discussions in the financial forums Fastly might be in a tricky place since their biggest customer is assumed to be Amazon and if one goes to their website https://www.fastly.com/partners/featured/ they show off all the biggest cloud providers and customers. Seems like Fastly are afraid to steal customers from their partners.

pbowyer4• 5 years ago

Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.

I greatly respect Fastly and their technology, but they (and Varnish) are biased against small companies and make it impossible for me to sign up customers who would benefit from their services and one day might grow into big spending customers.

AWS has shown that the days of "min $5000/year" (as both these services have quoted me for individual products) can be over and you can be profitable. Companies who don't catch up with that are cutting off their future.

dgroshev• 5 years ago

As someone who chose Fastly over Cloudflare for a small-medium business after a long, long deliberation I disagree.

Cloudflare's pricing is sneaky. They are good for small and personal projects with their free tier, but as soon as you need anything non trivial you get hit with their $xxxx/mo minimum commitment, "contact us for pricing" Enterprise plan. On Fastly you pay $50/mo (peanuts for a business) and that's it for basically anything you might want from a CDN, VCL is endlessly flexible and free. That $50 buys you access to a featureset that is arguably beyond Cloudflare's Enterprise plan. You will need quite a lot of traffic to go over $50, so overall it works out significantly cheaper for an SME that wants to use anything but a very basic CDN.

It's also important to remember that a lot of cases are perfectly served by VCL, I believe most businesses won't even need C@E. Sure, most people would need to learn a bit about it, but it's amazing to have it on your disposal for free compared to a usual rigid configuration.

Besides, something they don't advertise but is now the main selling point for me is support. Fastly was the only company I had pleasure dealing with where after shooting a technical question to their free tier support I got not just a technical answer, but a piece of code that solved the problem I had. In an hour. With a link to their Fastly Fiddle demonstrating how it works. When I encountered what looked like a bug, a few hours later their support person filed a PR to their TF provider fixing the docs. It was and is a breath of fresh air compared to my experience with any other SaaS, Cloudflare included.

I think the only thing that could've swayed me is Cloudflare's Chinese network. It's gated behind their Enterprise plan and they won't hold your hands much, but if you need it it's there.

unityByFreedom16• 5 years ago

Thank you for your hard work on workers and for your openness in these forums.

As a dev, it sounds to me like Fastly's whole purpose is wrapped up in attacking Cloudflare whereas Cloudflare is focused on innovating.

resonious• 5 years ago

Sounds like there's a lack of communication if Cloudflare published a benchmark blog post without the tech lead even knowing (better yet, finding out about the competitor's post first!).

I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post.

Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not.

Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.

jiggawatts• 5 years ago

I suspect you're mixing up microseconds with milliseconds.

5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...

kentonv12• 5 years ago

No, I didn't mix them up.

staticassertion2• 5 years ago

Thanks for posting this. I think a lot of people are hyperfocusing on some of your wording, but this post was very informative.

co_bee31• 5 years ago

> But as long as we're already mud slinging

Strange statement. Didn't Cloudflare start the mud slinging?

I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.

deanCommie• 5 years ago

In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.

Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor.

You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product.

Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.

robertlagrant7• 5 years ago

If you bother to read the thread the CEO has already commented to say they're changing that ToS. Why give advice that no-one asked for and no-one needed? Isn't that pretty much the worst?

kentonv12• 5 years ago

My manager's manager is the CEO. Yes, we all discussed it and everyone agreed the clause should be removed, but at the time of my comment I wasn't sure if I was authorized to announce that publicly. He has since posted in this thread.

zimmerx• 5 years ago

I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.

nix23• 5 years ago

Every benchmark should start with:

For us/me XX works better because of...

rileymat22• 5 years ago

Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two.

https://www.fastly.com/pricing/ does not make a distinction it says that "Any size company looking to give Fastly a try." They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.

flumpcakes2• 5 years ago

They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.

I think this is a non-sequitur. Bandwidth is a fundamental property of the performance of a system, if you can "pay for more bandwidth" then you are literally getting an inferior product when you do not pay.

Consider the implementation for "paying for more bandwidth". I assume that it is the same copper cabling hitting the same switches for fee paying and non-fee paying customers so the only way to "offer more bandwidth" is to implement some kind of quality-of-service metric on the network. This is explicitly giving the non-fee paying customer an inferior product.

judge20202• 5 years ago

Test up to $50 of traffic for free. Pay as you go from there with bandwidth pricing.

Looks like it's just post-use monitoring that'll cut off the account after it exceeds the trial limit or start billing per-GB; it doesn't say 'slower connection', so CF didn't realize that a trial account would affect their benchmarks regarding Compute@Edge

rileymat22• 5 years ago

People have stopped using "bandwidth" as we used to. It now means a total sum of the amount transferred in a lot of cases, not the "width" of the pipe.

It is more evident here: https://www.fastly.com/pricing/#bandwidthpricing

(To be clear, I agree with your definition, but it is not used)

atom_arranger• 5 years ago

As a consumer it’s great that both companies are benchmarking. Even if Cloudflare’s was flawed. Maybe a case of, the best way to get the right answer is to post the wrong answer.

We’re at least moving towards objective comparisons, talking about numbers.

Cloudflare prohibiting benchmarking is not good, obviously.

rkwasny• 5 years ago

Popcorn time ;) Let's wait for @jgrahamc to wake up

Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite.

However TBH benchmarking a distributed system like Cloudflare/Fastly is going to be REALLY HARD. There are 1000s of servers involved and reproducing the results might be impossible.

raggi• 5 years ago

It's a decent use case for planetlab.

ethbr03• 5 years ago

And reproducible benchmarks with cached, publicly accessible results.

At some point, you either have to make free tiers worse, or deal with thousands of people using them to run the same benchmark over and over.

Cloudflare loves alliances: start a distributed benchmarking alliance! :)

ksec4• 5 years ago

Popcorn time ;)

Indeed. We already have comments claiming Fastly are dismissive and evasive while others claiming the fault of Cloudflare. The community is divided, which is quite usual on recent HN.

Let there be another holy war of CDN between Cloudflare and Fastly.

saagarjha4• 5 years ago

Ok, but let’s ideally not have the comments from that war wash up here.

unityByFreedom16• 5 years ago

The community is divided

I don't see any division. It's clear to me who has experience to back up their words and who doesn't.

unityByFreedom16• 5 years ago

Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite.

Why doesn't Fastly just gasp ask Cloudflare about performing a benchmark? If they say no, publish that.

Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. It's absurd to say that contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison,

[FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services
> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;

That is not so different from Fastly's own sign-up process.

[1] https://www.fastly.com/pricing/

[2] https://news.ycombinator.com/item?id=29468455

anaganisk2• 5 years ago

Why are u parading all the comment threads slightly supportive of fastly? Unlike Fastly, cloudflare bars its own customers with TOS while fastly tailors their offering to new customers, I just don’t see your point.

jgrahamc• 5 years ago

Sente then gote

apengwin• 5 years ago

same

technobabbler2• 5 years ago

I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points:

* Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)

* Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing).

They do have two good points:

* Benchmarks between CDNs should be done from the same client locations (fine, but why did you then use your own set)

* Cloudflare should not prohibit benchmarking of their products (fair)

But this whole post read like a feeble attempt at misdirection, and made me distrust Fastly as a company. The complaints about their methodology aren't solved by using your own, even more flawed, methodology that doesn't even use the same language. You could've just emailed them and asked to work together on an apple-to-apples test instead of creating an even more flawed benchmark.

tylerhou3• 5 years ago

* Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)

First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript.

Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.

sroussey• 5 years ago

First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript.

Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs

ksec4• 5 years ago

I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points:

Because it is not meant as a comparison but they are the one defending a benchmarks ran by their competitors?

makeitdouble• 5 years ago

* Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing).

TBF, people who care about benchmarks usually expect high loads and value performance enough to put sufficient money on the table. I'd see the free tier people having different priorities.

Sure, it's still valuable info to know what's the free account limits are, but it shouldn't be headline news (except if it's exceptionally fast...)

acdha3• 5 years ago

Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)

This was also addressed in the post: they don’t consider the JavaScript flavor production ready in part due to performance. Even if you’re an all JavaScript shop which will never consider something else, comparing a mature released service to one under development makes the comparison harder.

It would have been interesting and more honest to compare other language runtimes to see whether there’s a trend for the entire platform or simply one language runtime.

imilk• 5 years ago

This comes across as incredibly defensive in tone from Fastly. They could have just reported the results of their test. But calling their competitors outright liars reflects worse on them than anyone else.

bloodyplonker22• 5 years ago

They do have to punch up because they are the little guy in marketing, but they did it in a very sloppy manner with shaky points.

shah00483• 5 years ago

When you're sucker punched with misleading stats, ya kind have to hit back. And didn't sound defensive at all, btw.

thecompilr• 5 years ago

Awesome, now where do I sign for the free fastly account? Oh right, they only do enterprise, so who the hell cares? Cloudflare OTOH caters for the whole spectrum.

yeskia• 5 years ago

Only semi-related, but why does Cloudflare perform so poorly in Oceania? I've noticed less than stellar performance in Oceania with Cloudflare's CDN, so to see similar stats with workers makes it seem it's not a product-specific issue.

plasma• 5 years ago

I'm based Australia and have used Cloudflare for personal/business; if you aren't on the $200/month plan for your website then you don't get (consistent) access to the Australian-based servers for your visitors (they may be routed to eg Singapore instead during some times of the day) which actually hurts performance if your service is located in (say) AWS/Azure Australia region, at least for the CDN aspect, I assume the same for Workers too.

kiwijamo• 5 years ago

Same here in New Zealand too. Cloudflare peers only with certain ISPs. The ones they don't peer with sees some traffic (e.g. sites on free plans) routed to offshore Cloudflare POPs around the world instead of local Cloudflare POPs. I understand if the website is on a paid plan they will be more likely to route to a local POP (incurring more costs in doing so).

LordAtlas• 5 years ago

I'm in India and if you're on the free CF plan, you often seem to get served from the Singapore POP instead of the Indian ones. I've seen other people on various groups complain about this too.

foton1981• 5 years ago

Seriously, who cares about 5ms difference? Or even 25ms. I don't. And Asia is just used to longer latencies. Edge compute became a commodity even before it was born.

cortesoft• 5 years ago

The whole point of edge workers is to be able to provide very low latency all around the world... once you have that, it opens up whole new classes of workloads. Yes, 25ms doesn't matter for most current use cases.... but that is partly because the use cases that required lower latency weren't possible before.

jlund-molfese• 5 years ago

I’d love to be able to (affordably) spin up a low-latency Windows desktop for a few hours at a time, that I could use for applications which don’t run well on my bottom-tier M1 Mac

ddtaylor2• 5 years ago

That is pretty scummy that Cloudflare won't allow benchmarks.

civilized• 5 years ago

It's been hilarious watching competing companies publish warring blog posts declaring each other full of shit. I mean, I knew, but thanks for airing each others' dirty underpants.

dmw_ng• 5 years ago

Most insightful comment here IMHO.

Vaguely reminiscent of the Glasgow Ice Cream Wars, I'd rather not buy from anyone involved

gary_03• 5 years ago

It reminds me of being on Slashdot 15 years ago. We've even got one of the engineers posting in the comment thread, starting their own little flamewar. The names of the characters have changed, but there's still a bit of the same old Internet here.

unityByFreedom16• 5 years ago

The fact that a lead engineer and CEO took the time to respond here is clarity not a flamewar.

edit maybe I'm wrong,

https://news.ycombinator.com/item?id=29469174

unityByFreedom16• 5 years ago

It's just Fastly's laundry. Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. Yet they claim contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison,

[FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services
> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;

That is not so different from Fastly's own sign-up process.

[1] https://www.fastly.com/pricing/

[2] https://news.ycombinator.com/item?id=29468455

nerdbaggy3• 5 years ago

Most of cloudflare features are locked behind sales, powerpoints, and a couple meetings.

AtNightWeCode• 5 years ago

The off-the-shelf prices from Cloudflare are also negotiable for larger companies.

anaganisk2• 5 years ago

How is a sales deal same as a tech stack, I fail to see your point about fastly’s laundry?

Unlike cloudflare I dont think fastly is in business for solo devs but for enterprises. And each enterprise has its own requirements.

If fastly prefers to know customer requirements before they start using it or tailor them whats wrong?

Meanwhile cloudflare prevents its OWN EXISTSING customers from benchmarking, which is actually draconian.

boynamedsue• 5 years ago

Cloudflare seems to encourage benchmarks of Workers Sites:

https://blog.cloudflare.com/workers-sites/

JediPig• 5 years ago

I am glad I said no to working for both. When I saw both are politically active and showed child like behavior during the interviews, I said no, and decided to dedicate some of my time to dog and cat rescue.

soheil• 5 years ago

I'm regularly surprised why Cloudflare gets so much love specially here on HN. Cloudflare is slow, it man-in-the-middles websites, it shows a redirect page randomly for no apparent reason on websites that it powers, it does questionable things at the DNS level, it randomly shows captchas for no apparent reason, the list goes on. Almost every website nowadays uses this monstrosity. We're handing over the internet on a silver platter to them. I think there are better alternatives and I'm so glad at least Fastly is debunking their bs experiment.

celsoazevedo2• 5 years ago

it man-in-the-middles websites

That's just how reverse proxies work. I'm not saying that this is good, but anyone offering the same service is doing the same thing.

sgtfrankieboy• 5 years ago

no apparent reason

A big reason is making sure the website behind it stays online. Having to deal with weekly/daily DDOS attacks yourself takes a lot of time.

speedgoose• 5 years ago

I also read that Fastly is slower at specific point of times during a week, slower for free test accounts, slower at executing javascript (hence the switch to rust), and slower in locations carefully selected by CloudFlare.

I'm not saying that the benchmark from CloudFlare is fair, but it's a bit funny that their response contain a "let's switch to rust" or "let's benchmark next to our datacenters".

kamray23• 5 years ago

I mean they switched to Rust because a comparison of two fully developed systems is fairer than a production system and one in testing. Besides, I believe the blog post said that they selected nodes at actual random?

tomasreimers• 5 years ago

"A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle."

Are there any widely agreed upon benchmarks on how emscriptened rust compares to untyped js? It feels like a large portion of the reported Delta boils down to that.

chinathrow• 5 years ago

This reads as someone who his losing lots of customers to CF and whines about it.

qwertyuiop_• 5 years ago

Another appendage measuring contest.

seando-vajina• 5 years ago

wgaf

nickdothutton• 5 years ago

The “DeWitt Clause” or sometimes also known as the “Oracle Clause” has been a fixture of ToS for years. The fact that it features in Cloudflare’s docs only serves to remind us that entrenched practices die hard. https://www.brentozar.com/archive/2018/05/the-dewitt-clause-...

Debunking Cloudflare’s recent performance tests

Related Stories