Peering agreements between Backblaze and Cloudflare

https://www.reddit.com/r/backblaze/comments/l2lq42/how_long_will_cloudflare_peering_stay_for/gk6qie0/

by decrypt • 5 years ago

243 60 5 years ago

erentz2• 5 years ago

My brain is a bit fuzzy today but this comparison seems... off? Yes an interconnect inside a colo is going to be dirt cheap. But you can’t compare it to a huge backbone spanning continents that gives you reach across the world. Everyone should peer with everyone they can locally. Use the tier 1s for reach beyond what you can peer with. (This is not a defense of tier 1s and their peering policies or pricing.) Also this is why it’s always amused me when DC guys quote their raw switchport capacity with pride. Gotta multiply that by distance traveled to get a real comparable metric.

wmf• 5 years ago

Yeah, this is a bad take from Backblaze. If ISPs had 99% profit margin you'd see it in their quarterly reports. Spoiler warning: they don't.

Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it.

csnover3• 5 years ago

Frankly, I’ve never seen a good take from Backblaze once I got past their marketing blog posts.

Zero-knowledge encryption? You don’t need that[0][1].

Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3].

A pattern of violating of basic and well-known software security principles[4][5][6]? Bodge in fixes. Don’t tell customers about it. Cancel the public bug bounty programme, claim it’ll be back in an indeterminate period of time[7] and replace it with a private one instead[8].

[0] https://www.reddit.com/r/backblaze/comments/8oczbl/how_do_i_...

[1] https://help.backblaze.com/hc/en-us/articles/217664798-Secur...

[2] https://old.reddit.com/r/backblaze/comments/igaqse/please_al...

[3] https://old.reddit.com/r/backblaze/comments/k764xq/backblaze...

[4] https://twitter.com/zetafleet/status/1304664097989054464, more discussion at https://news.ycombinator.com/item?id=24842871

[5] Hard-coded credentials in deployed apps: https://medium.com/@pig.wig45/from-n-a-to-resolved-for-backb...

[6] Relying on client-side validation only: https://www.youtube.com/watch?v=1LC0aEZFVz8

[7] https://news.ycombinator.com/item?id=24849334

[8] https://www.backblaze.com/security.html

Dylan168075• 5 years ago

What seems to happen when there's insufficient competition for ISPs is that they under-invest in network upgrades. So you wouldn't expect to see crazy profits, it's more a matter of high prices and inefficient operation.

rsync3• 5 years ago

"Also, I remember when there were only half as many tier 1s, so if they're a cartel keeping new members out they aren't doing a very good job of it."

As evidence for your point I would point to Hurricane Electric (he.net) who I have found to be very progressive, cost-competitive and easy to work with.

For historical reasons we[1] are connected with init7 throughout Europe but in the United States and Asia we work with he.net anywhere that we can.

[1] rsync.net has a long-standing (15+ year) relationship with init7 in Zurich.

zamadatix• 5 years ago

Cloudflare is providing the huge worldwide backbone and saying it's cheaper for them to peer where you are and transport it on their backbone as needed rather than pay Tier 1's to do it.

The difference in cost largely comes from being peering not transit though. I.e. they aren't handling traffic for 3rd parties on their backbone (well, in this case, they do offer that though), just traffic between the mutual peering entities. As a result you're never sending them something they didn't already have to pay to get.

Also it's not as simple as distance. A lot of data between 2 world class datacenters is pretty easy and cheap, the fiber is already there. A lot of data to the rest of the world is a PITA and a physical/logistical nightmare.

erentz2• 5 years ago

Edit: ok you expanded your initial comment. So yes they aren’t providing transit and your own comment explains why this isn’t a good comparison, it appears cheaper than it really is because Cloudlfare aren’t paying transit to get your data.

Dylan168075• 5 years ago

Ok, so here is my proof that the "tier 1" networks are over charging: if Backblaze routes a packet over the "tier 1" network is comes down to about 2/10ths of 1 penny per GByte. That's $0.002/GByte.

That roughly matches the information I've been able to find when looking into transit pricing.

And then AWS/Azure/GCP gouge you to an absolutely ridiculous extent even beyond that.

vegardx• 5 years ago

You'll likely see prices in the same range with most providers if you have some volume and are willing to commit to it.

Dylan168075• 5 years ago

By "most" do you mean excluding those three? I know it's not too hard to get a port at a good rate if I have my own server, and it's easy to rent a server with plenty of bandwidth. But if I want to use something like glacier, I'm screwed. Amazon's "contact us" threshold is $28k per month, and anyone under that is paying 30x-40x as much as if it was $.002/GB.

cbg02• 5 years ago

Public internet exchanges are huge in Europe, you can basically reach every network in Europe by connecting to a few large exchanges for a fraction of what you'd pay for tier 1 transit (granted, you still probably need it to reach some eyeballs). I wish we'd see the same concept pop up in the US, as these exchanges are definitely not as popular there.

https://www.de-cix.net/en/locations/germany/frankfurt/statis...

https://www.ams-ix.net/ams/documentation/total-stats

https://portal.linx.net/

sneak4• 5 years ago

A big part of it is that simply connecting your own network to these exchanges is a lot more expensive in the US due to increased land size and corrupt legislation.

Due to the intense regulations (which are bought and paid for at the legislative level, and mostly exist to protect the large incumbents like Verizon, AT&T, Comcast, CenturyLink, et al from any new competitors) you end up having to buy your "last mile" loop (which, due to the US's size and general car-centric design, is much more than just a mile in most cases) that connects your network (in your office, or data center, or whatever, over to the exchange point) from a small number of monopolists.

They're thousands, or tens of thousands of dollars per month.

"We don't care. We don't have to. We're the phone company."

gbil2• 5 years ago

Internet Exchanges doesn't mean that a peering agreement is also in place automatically. For example: https://www.ams-ix.net/ams/documentation/general-terms-and-c... 3.3 Peering arrangements are not covered by these AMS-IX General Terms and Conditions or the AMS-IX Connection Agreement. The Customer is responsible for the negotiation, conclusion and implementation of peering arrangements with other users of the AMS-IX Infrastructure.

cbg02• 5 years ago

While you are correct, many networks connected have in fact an open peering policy: https://www.ams-ix.net/ams/connected-networks

xenadu02• 5 years ago

These used to be more common in the US and some still exist but the big telcos have all stopped upgrading their equipment which deliberately chokes the available bandwidth at those locations. They also deliberately don't on-board any new ones. This is the same strategy they used against Netflix if you recall - deliberately don't upgrade the linecards at choke points to ensure Netflix packets get dropped. Netflix eventually caved (and raised prices) so we're all paying for it on both ends: as ISP customers and Netflix subscribers. Who doesn't like a bit of double-dipping?

They really really want to go back to the long-distance telephone days where people paid according to some kind of metering plan. That's the whole push behind getting rid of Net Neutrality (which is how the internet always worked up to that point - so much so we had to invent a new term to describe what was taken away). Why invest in new plant, fiber, or equipment? Just stop upgrading or upgrade very slowly and let increases in data usage dump money directly into your pocket. A great gig if you can get it.

toast0• 5 years ago

The reasons residential ISPs don't peer is two-fold.

a) They're nearly all Tier-1 networks, and Tier-1 networks have very hard to meet peering requirements; AT&T, Verizon, and Sprint have been Tier-1 since forever, and now they're the dominant wireless carriers. AT&T and Verizon are also large residential ISPs, and CenturyLink is another, and they're a Tier-1 too. Comcast is effectively Tier-1 in the US, but they don't have an international network (yet?).

b) residential connections are setup to be unbalanced flows, and peering policies at networks with restrictive policies generally require roughly balanced flows. Very few networks that do business with consumers are going to get balanced flows. Maybe a consumer oriented backup service can, if they mostly receive backups and don't send a lot of restores; 1-1 messaging without persistent server storage should be pretty much balanced, but nearly everything else is pretty much never going to be balanced. Of course, tier-1 networks also won't peer with you if you're a customer or have been a customer recently; so if you do end up with appropriate flows to your transit provider, you'd need to switch to another provider for some amount of time before coming back to ask for peering, and they may increase requirements in the meantime.

Things were different when we had local/regional ISPs (dial-up and DSL line sharing). As those got big enough to justify multiple transit providers, they would gravitate towards peering places, because you can pickup multiple transit providers easily, and also peer to reduce costs for both networks when connecting to other networks at the peering place. A local ISP doesn't really care about traffic balance, they just want traffic off their transit link; and content networks don't care about traffic balance, they just want traffic off their transit link.

If I had my way, we'd get mandatory line sharing for the last mile (and something for wireless?), so that we could have meaningful competition in IP networking at the local level again. Something where all the last mile infrastructure providers are prohibited from offering retail service would be best; they could keep their tier-1 ISP business, and their last mile business, but someone else would have to at least glue the two parts together.

traceroute66• 5 years ago

you can basically reach every network in Europe by connecting to a few large exchanges for a fraction of what you'd pay for tier 1 transit

Erm.

No. No and, erm, NO !

Members at peering exchanges can be split into two categories "old boys club" and "everyone else".

The former category are mean as mouse shit.

They won't peer with "everyone else", in many cases they won't even peer with you even if you have a decent amount of traffic related to their AS.

They certainly won't publish ANY routes on the exchange route servers.

For example, BT (incumbent UK telco for those who don't know) are famously restrictive about who they will peer with at LINX. There is an old joke between LINX members, "Q. How do you know you've made it as an internet network ? A. BT agreed to peer with you over LINX". Basically the template answer when you ask BT for peering is "NO. Go away now you silly little man, you should be ashamed for asking."

That's not to say European exchanges are worthless. They are great for interconnecting with other mid/small networks. They are great for connecting to CDNs, the hyperscalers, and the Facebooks, Googles etc. of this world.

tiernano3• 5 years ago

in all fairness, the original op did say "basically"... I am peering in a few different IXes in Europe for personal stuff, mostly learning, and i get Apple, Google, Cloudflare, Facebook (even though i dont use them), Netflix and a lot of other places included... I do also agree about the whole "old boys club". My main ISP is Liberty Global (Virgin Media Ireland) and anyone i have talked to about getting a link with tells me its paid peering... would be handy, but too expensive...

Also, they dont peer with Cloudflare direct either... Ahh well...

t0mas88• 5 years ago

Several of the big incumbent ISPs in most European countries are not as eager to peer because they also have a B2B side of their business that sells you the bandwidth you would otherwise need.

But for most smaller end-user networks it's great to be able to peer with Netflix, GCP/Youtube, AWS, Azure easily in the exchanges, because that's where the majority of their user traffic comes from.

wut42• 5 years ago

It's a bit more complicated in reality. Most of the ISP consumers, even if present on the exchanges, aren't openly peering on it. It's great to interconnect with smaller networks and providers, but consumers ISP ? You have to either have a lot of traffic _they_ send to you, otherwise, peering isn't possible, or complicated, or you have to pay.

j16sdiz• 5 years ago

I always thought peer agreements are under NDA. Surprise to see they talk openly.

brianwski4• 5 years ago

Disclaimer: I work at Backblaze.

I always thought peer agreements are under NDA. Surprise to see they talk openly.

Both Backblaze and CloudFlare are very open about this. It is called the "Bandwidth Alliance", you can read about it at these two links:

https://www.backblaze.com/blog/backblaze-and-cloudflare-part...

https://www.cloudflare.com/bandwidth-alliance/backblaze/

For Backblaze, it's part of our business strategy to be open and honest with customers. That feels funny to type, but it's true. We have historically disclosed things that other companies have traditionally kept secret, like our internal drive failure rates. I don't know if this is surprising or not, but it has worked well financially for us. Meaning we explain what is going on, more potential customers hear about us, more customers trust us, our sales increase.

I'm actually curious why other businesses keep their operations so secret?

3pt14159• 5 years ago

Disclaimer: My mum works for AT&T on internet related stuff.

This post has large elements of truth to it, especially for dumb hops over tier 1 which should be much cheaper than they're currently billed, but there's a lot more going on at AT&T than most people realize. Especially across the oceans. It's not quite as simplistic as this take makes it out to be, since the 5%ile case for networking a packet over tier 1 really is expensive to handle, maintain, bill, and provision.

That said, there should be better regulations on tier 1 inclusion. On the other hand, they did make deep investments into infrastructure that took decades to become profitable. Often times with government assurances. It's one of the reasons telephony had such high penetration in Canada (75%) at a time when it was much lower in France (~25%), Bell was given assurances that we're now paying for in more expensive internet. Not saying it's right, but it's like this for understandable reasons, and it's not strictly raw corruption.

2Gkashmiri5• 5 years ago

so why do vps providers provide a "1-3"TB a month transfer when it costs nothing? talking about cheap plans here

exikyut5• 5 years ago

It most definitely does not cost nothing. The whole point here is that the Tier 1s price-gouge everyone else, and from there everything is far more expensive than it needs to be.

Prices only go up from there.

Cheaper plans sold by smaller outfits are likely being sold to end users through 2-3 layers of resellers. This creates a tug-of-war between multiple resale overheads and customer interest in something at a low price point, so the baseline of what is being offered (CPU, RAM, speed, monthly transfer) can get fairly wobbly.

For larger companies, to take Contabo as an example, they basically get to dictate whatever performance metrics they want, because they own the entire stack. So I might have more-or-less-unlimited 100Mbps and 6GB RAM, but CPU performance is... not great, I'm pretty sure my KVM instance winds up almost entirely swapped onto an HDD occasionally, and there are some really harsh QEMU disk I/O quotas.

2Gkashmiri5• 5 years ago

funny that you bring up contabo. i have been thinking about setting up a selfhosted email instance and that could lead to more stuff later on but bandwidth is like night and day. on contabo cheapest vps is like €4.99 / month, 4 vCPU Cores,8 GB RAM, 200 GB SSD, 200 Mbit/s Port, 1 Snapshot and "32 TB of transfer is already included in each VPS package."

while on vultr.com, the ipv4 cheapest is like 25 GB SSD, 1 CPU, 1024 MB RAM, 1 TB transfers for $5/mo.

that is such a huge difference that i dont know what is real.

i want like 10TB storage vps from contabo but apparently they dont have it. do you know anyone like them? i have looked at scalaway and others but they dont say pricing of bandwidth past the allowance. like only for their s3 they say pricing which is exorbitant like the ones mentioned in the article

edit: vultr object storage says "$0.01 per additional GB transferred " https://www.vultr.com/products/object-storage/

and scaleway says Inter-regional* and external outgoing data transfer to other products from a different region and the Internet: 75 GB free every month, then €0.01/GB

same pricing for backblaze b2 for downloads

phonon• 5 years ago

Why doesn't Cloudflare buy GTT? They can easily afford to.

Bkmhfdtyb• 5 years ago

From what I have read, these networking centric companies have a reputation for noncompetitive pay. Is this true? They otherwise sound like great places to work.

2Gkashmiri5• 5 years ago

so i want to start a peertube server, what storage is best, size and ample bandwidth. $0.01/Gb is too much for transfers....

oh, wasabi is not for video, i tried

Peering agreements between Backblaze and Cloudflare

Related Stories