Vodafone Germany is changing the open internet, one peering connection at a time

https://coffee.link/vodafone-germany-is-changing-the-open-internet-one-peering-connection-at-a-time/

by PhilKunz • 9 months ago

479 221 9 months ago

wil421• 9 months ago

I worked with Vodafone years ago to do an integration with ticketing systems. It seems like no one actually worked for Vodafone it was all contractors or contractors of contractors of contractors.

Outsourcing peering to a 3rd party seems like their playbook.

lifestyleguru6• 9 months ago

I used service of Vodafone Germany once. In paperwork fever I only scanned the contract and saw somewhere 6000 but signed off and moved along with remaining paperwork. I thought "it has to be at least 60Mbit/s, right?". Nope. 6Mbit/s DSL, two years contract, cancellation by letter. Fuck you, Vodafone.

egeozcan• 9 months ago

At such organizations, you can usually communicate with any contractor, but you must go through a project manager. These managers, who are often contractors themselves, act as a support center for the other contractors.

benced• 9 months ago

Rare comparative W for American ISPs?

aidenn02• 9 months ago

I think Comcast charges for peering as well (but not through an intermediary).

stego-tech6• 9 months ago

Not really, as US ISPs have been repeatedly trying to game the system into becoming landlords for decades. The difference is, ironically, their own self-imposed monopolies: Comcast may be a T1 ISP, but they’re largely a monopoly in the markets they serve. Same goes for Verizon, Spectrum, Cox, TDS, etc. The end result is a sort of “forced cooperation” with each other, though occasionally one will try to extort the others for cash (I seem to recall L3 and Cogent both engaging in this bullshit around the time streaming video got big).

Companies are extractive by nature, and they will always try to find new ways of squeezing blood from a stone absent regulations saying otherwise (and suitable punishments ensuring anyone caught violating them is crippled in the marketplace, if not outright destroyed). This has been going on for decades and will continue absent regulatory intervention. Just look at how the US Electrical grid bills to see how this could end up (higher prices, bullshit fees, redundant billing).

rsingel• 9 months ago

California's net neutrality law bans these kinds of paid interconnections, but they likely exist as all these deals are wrapped in 15 layers of NDAs

Liftyee• 9 months ago

New selling point for all the VPN sponsored segments... "if you're on Vodafone Germany, make your connection speeds faster with YetAnotherVPN!"

timvisee• 9 months ago

A VPN won't make your route shorter

toast05• 9 months ago

If the shortest route is congested, a longer route can be advantageous if it avoids congested hops.

immibis4• 9 months ago

It actually can, depending on where it is. Paperclip optimizers don't optimize for route length - they optimize for paperclips received. When Vodafone or Deutsche Telekom gets your packet, they try to send it to one of their customers, because their customers pay them to receive traffic, even when it's a longer route.

If you're sending a packet from German Shittytel to German Okaytel, and Okaytel just happens to buy a connection to Singapore from Asiatel to get packets to Asia, and Singapore Internet Corp just happens to buy a connection to German Shittytel to get packets to Europe, they'll be glad to send your packet all the way to Singapore so Asiatel will have to pay them for it. But if you sent your packet to a VPN server in Berlin with a neutral peering with both ISPs, the packet would take a nearly common sense route.

In practice, these situations don't happen, at least not this extreme. Partly because ISPs are trying their best not to be the recipient of this. Okaytel doesn't want their packets to be round-tripped through Singapore - that's a bad user experience and they're ultimately paying for it in money as well. So they might negotiate with Asiatel that Asiatel won't tell Shittytel that it's able to deliver packets to Okaytel - in fact there are often BGP attributes they can set to do this automatically. Business is incredibly cut-throat and incredibly stupid. I guarantee Shittytel has a lot more money than Okaytel because they are better at "extracting value". Not only the ISP business is like this btw.

inemesitaffia17• 9 months ago

It can.

Datacamp, which hosts lots of them has very good peering

undeveloper• 9 months ago

I'm not sure that will not speed anything up, since you face the same peering issue with connecting to XYZ vpn's server

wmf6• 9 months ago

The VPN will be fast if they pay the extortion. It's basically a paid fast lane.

Lio• 9 months ago

Is there anything that Vodafone customers can do legally to punish Vodafone or not delivering on their broadband contracts?

If you're paying for a 1Gbps connection and Netflix is only able to stream to you at 0.93 Mbps because Vodafone or Inter.link are choking off the supply, surely that's breach of contract on Vodafone's part?

I'm sure Cory Doctorow has a word for what's happening here.

tracker12• 9 months ago

Are there competing options, or are they a monopoly?

lifestyleguru6• 9 months ago

In Germany in specific building there is only one provider available in your telephone socket, and one in your cable socket in your apartment. Frequently there is no cable socket.

aidenn02• 9 months ago

Sounds like their largest competitor (DT) is already doing this.

okanat• 9 months ago

Yes and no. There are other providers in Germany. However, with the EU's neoliberal privatization policy the governments privatized many existing infrastructure. Vodafone bought the previous government company that owned all of the the cable TV infrastructure of Germany. So they are a monopoly of a particular type of internet connection. Depending on the place the alternatives could be too slow since Germany also has an aging population that do not {care about, demand} higher internet speeds and didn't upgrade its copper infrastructure due to corruption.

fuzzy23• 9 months ago

No monopoly. Only for cable internet, which may be a possible argument. For landline internet (DSL), there's plenty of alternatives.

growt• 9 months ago

Afaik almost a monopoly: there is Deutsche Telekom which does the same thing and Vodafone. I think apart from some local providers almost everybody else is just a reseller of one of the two.

aktuel• 9 months ago

Depends on the region. Often there are smaller regional companies providing fiber internet. Prices for these fiber connections a still somewhat higher than the cheapest vodafone tier, but you also get better service for your money.

hilbert42• 9 months ago

This is a bit messy but if there are completing options at a given location install multiple ISPs and run them concurrently and log the details—download speeds, etc.

There's nothing as good as hard verifiable data—even if regulators play hardball and favor ISPs then you've the evidence to whip up political action (claim biased decisions, etc.).

rurban2• 9 months ago

There's no monopoly, it is a cartel. Most big businesses in Germany operate in cartels to fuck the customer.

m_gloeckl2• 9 months ago

You can file a complaint with the "Federal ministry for digital transformation" (formed this year). It does actually work, but it's a lengthy process.

I did force my cell phone carrier to grant me proper 4G speeds last year, after spending many hours with their help line and ultimately complaining to the (then) ministry of transportation and digital infrastructure.

afeuerstein• 9 months ago

Can you elaborate on what was wrong with your cellular connection?

lukan• 9 months ago

"If you're paying for a 1Gbps connection"

That's why you are paying for a "up to" 1Gbps connection. (I think it was already a struggle that they had to put the "up to" in the big advertisement)

Telaneo2• 9 months ago

Surely there's a reasonable expectation that Netflix would work at decent speeds, especially given that Netflix's infrastructure, nor the network load as a whole are to blame, but rather the specific ISP bureaucracy? Getting 1/1000 the listed speed does not strike me as something even a 75 year old computer neophyte of a judge would take kindly too, unless it were for very good reasons.

wat100002• 9 months ago

There's always an implicit reasonableness requirement on these things. "Up to" isn't a wildcard that lets you do anything you want. It will protect you if you struggle to maintain top speeds during peak hours, or if a technical fault cuts speeds in half for a few days. But if you provide 1/1000th of what you claim to offer, "up to" won't stop a judge from observing that you're not really providing what you said you would.

dvdkon• 9 months ago

At least here in the Czech Republic, ISPs have to also list a "guaranteed speed", and it can't be less than some fraction of the advertised maximum. I don't know what part of the Internet that speed is supposed to be measured against, though.

toast05• 9 months ago

There's definitely not anything in the contract that promises performance to a 3rd party, especially not in a residential contract. The legal options are switch to a different ISP and/or start a new one. Not always easy or practical, but there you go.

Who is to say where the performance problem is? Certainly not your contract.

Maybe if the last mile is cronically congested, or between the local aggregation switch and their regional exchange points, you might have a legal case. But if the issue is insufficient connectivity between their network and other networks, I would be very surprised if the contract terms covered that at all.

There's a bunch of networks throughout the world where their policies mean you can get more economically acheive better connectivity to their customers by hosting outside the geographic boundaries of the network rather than inside it. Doesn't make sense from a theoretical point of view, but when German ISPs won't interconnect within Germany, serve their customers from Poland or France and the connectivity picture may change significantly. Worst case, serve them from the US (but the latency may be too high)

mystraline• 9 months ago

The play by major content providers is "not to pay" and "block inter.link"

Sure, you lose Vodafone germany. Then you explain clearly why to every major media.

This coukd be stopped fairly quickly.

TulliusCicero2• 9 months ago

Blocking seems overkill. Just put up a banner explaining why the service is slow, warning customers about their ISPs.

Spivak• 9 months ago

Because a banner is ignorable. A block will actually spur action.

preya2k• 9 months ago

Doesn’t seem to work in regards to Deutsche Telekom so far.

noAnswer4• 9 months ago

"The play by major content providers" like Google? Where in order to become a "Verified Peering Provider" you are not permitted to use a IX?!

https://peering.google.com/#/options/verified-peering-provid...

inemesitaffia17• 9 months ago

The restriction is only to Google.

1299, a Google gold partner is at IXes

fuzzy23• 9 months ago

It's important to keep in mind that Deutsche Telekom is basically doing the same, and has been… forever?

I disagree with this move, but it is not without precedent.

sadeshmukh• 9 months ago

They mention it extensively in the article.

kleiba• 9 months ago

I suppose OP is hinting at the fact that Germans are probably already used to having one of the shittiest internet services in the Western world.

fuzzy23• 9 months ago

They certainly do; however, we all know the game: Headlines only. Reading TFA? Meh

lxgr6• 9 months ago

The big difference here is that you're almost never forced to use Deutsche Telekom for wired internet (there are many DSL resellers, and many of them actually provide routing themselves), but in some buildings, there is literally only Vodafone, with no choice of any alternative service provider on top.

cachius3• 9 months ago

Still Telekom has a big market share there.

BoredPositron• 9 months ago

Fucking around with peering is the specialty of German ISPs. Telekom (our biggest provider) is sometimes unusable for YouTube/Netflix/Cloudflare/Steam in big cities because of similar shenanigans.

hylaride3• 9 months ago

I was shocked just how slow and poor the mobile networks were in Germany. When I last visited (circa 2014) I literally switched my prepaid sim from T-Mobile to Vodaphone because the experience was so bad - only to have the same bad experience. I had barely usable LTE and connections dropped to EDGE on the train between Hamburg and Berlin. Google Maps barely loaded in the cities let alone the fact I was playing Ingress at the time and it was pretty much unusable

This was surprising to my Canadian sensibilities. Our mobile networks are expensive, but I generally get solid 4G and now 5G coverage between Toronto and Montreal and had full 4G (at the time) coverage on a road trip between Saskatoon and Calgary.

tracker12• 9 months ago

I hate this line of thinking.. Netflix isn't just sending data to random users, it's data Vodafone users request and want to receive.

phineyes• 9 months ago

This isn't unique to Vodafone. Google has also been slowly withdrawing from IXes globally in favor of PNIs and "VPPs" (verified peering providers). This only makes it harder for smaller networks to establish presence on the internet and feels pretty anti-competitive.

On the flip side, IXes are becoming harder and less desirable to participate in: port fees are going up, useful networks are withdrawing, low quality network participants are joining and widening blast radius. I'm not sure what the answer to this is, but this has not been a great year for the "open" internet.

stroebs• 9 months ago

I thought Google was _always_ like this. At least going back to 2015 when I left the ISP game, peering with them was notoriously difficult if you didn't have the traffic volumes required. Our network suffered from asynchronous routing to Google and Netflix for years because they refused to allow our routes despite checking all the boxes they require. Customers eventually left because other (larger) ISPs didn't have this issue.

I get why the enshittification of IXPs is occurring. Over the years many small and careless ISPs have caused issues for IXPs (and peers) based on what I've seen on mailing lists. It's hard work managing many hundreds or thousands of peers, let alone the equipment cost with multi-100Gbit ports becoming the norm for larger providers.

MichaelZuo3• 9 months ago

Why did your company expect Google to readily accept peering?

If there was such a large difference in volume they would be choosing to intentionally make it more difficult for themselves.

mike_hearn• 9 months ago

What sort of verification are they doing? Is this trend being pushed by the lack of proper security on BGP?

brynx97• 9 months ago

Google gave a presentation on this that I think is helpful context for "why": https://nanog.org/events/nanog-94/content/5452/

kyrra• 9 months ago

Direct YouTube link: https://youtu.be/Yg-qV6Fktjw

techsupporter• 9 months ago

low quality network participants are joining

(Genuinely curious because I truly don't know in this context) What is a low quality network participant? One of the "bulletproof" hosts?

inemesitaffia17• 9 months ago

Malware, flapping, bogons, remote peers, etc

9cb14c1ec0• 9 months ago

IXes are becoming harder and less desirable to participate in

Could this be due to the rise of services like Equinix Fabric and Inter.link? Google doesn't need to peer directly with most anymore because there is always a middleman somewhere who can handle it, and for many businesses the convenience of a point and click web gui outways whatever it costs?

simonebrunozzi• 9 months ago

Notably, Chris Sacca (now a famous VC), when working at Google in the early 2000s, got in trouble for mentioning that telecoms were preventing Internet from being Net Neutral, and instead of being fired, he got promoted by Larry Page and given a huge budget to work on it. I can't find a reference but I'm almost certain this is correct.

hylaride3• 9 months ago

Bell Canada also has had a long-standing policy of refusing to peer with internet exchanges. They'll only truly peer with other direct backbone providers and a handful of one-off peer with other large networks (google, cloud flare, etc), but their historical position as Canada's base backbone (not so much anymore, but it was definitely a thing pre-2005) has meant their policy is most people should pay them to peer. I'm not sure if it's still the case, but IIRC for awhile they also refused to peer with any other domestic backbone providers.

The result has been some funny routes sometimes. I live in Toronto and have seen trace routes bounce over to Chicago to connect to stuff colocated here in Toronto.

It's frustrating as their fibre is my only real high speed option; also their lack of IPv6 on anything but their mobile network is annoying.

jacquesm• 9 months ago

That's insane given that Front St. 151 is probably the best spot on North America when it comes to connectivity.

1over137• 9 months ago

it’s also insane because probably no Canadian wants their traffic going through the US unnecessarily, since all the 51st state takeover crap.

hylaride3• 9 months ago

FWIW, this has being going on for decades. Bell literally doesn't care if there's a chance they can suck up some peering fees.

There have been periodic times where it became an acute problem, like early in the YouTube and Netflix years there was a lot of congestion in their upstream peers and they held out hoping those orgs would pay for the peering. They were also over provisioned in early DSL days where their upstreams became saturated and there were few alternative paths.

subarctic2• 9 months ago

Maybe that's a way to give them bad PR and convince then to change policies? Unfortunately most people probably don't understand this well enough and they have a pretty well oiled PR machine with all their control over tv and radio stations, etc

idatum• 9 months ago

also their lack of IPv6 on anything but their mobile network is annoying.

This gives me even less confidence after BCE took over ZiplyFiber, US PNW provider. There's a long running joke about IPv6 just one more lab test away from deployment.

subarctic2• 9 months ago

It's annoying how they're the only big ISP offering fiber everywhere when they're also the ones that don't support ipv6 and have the shitty peering policy. I've heard you can use another isp though (teksavvy maybe?) that uses Bell's fibre and supports ipv6

deviantintegral• 8 months ago

I’m on teksavvy fibre (via bell) and IPv6 works wonderfully.

inemesitaffia17• 9 months ago

It's not unusual for incumbents to refuse to peer in their own country.

See Telstra(Australia), the Korean Telcos, NTT(in Asia), Globacom (Nigeria) etc

danogentili2• 9 months ago

Seems pretty much a private equivalent of "fair share" govt regulation currently being pushed by ISPs in Europe through lobbying with the Digital Networks Act (https://www.namex.it/toward-the-digital-networks-act-the-fut..., https://stopdna.eu/).

South Korea pioneered fair share govt regulations in 2016 (which caused Twitch to exit the market in 2024 due the exorbitant "fair share" fees).

stego-tech6• 9 months ago

The solution - as always - is regulation. ISPs typically already have very generous business models with widespread monopolies on customers, overwhelming barriers to entry for new companies, and a lack of rate controls allowing then to price arbitrarily - all of which supports immensely profitable businesses without the need for additional extraction of capital from other parties. Regulators and consumers alike should be screaming in rage at the idea that their ISPs are now multi-dipping for revenue, but we’ve done a piss-poor job of explaining how this works to the common man and thus can’t count on them to support the Open Internet as we’d like to see it.

That being said, the threat to the open internet is also more than just ISPs being gigantic assholes: it’s centralization in general. A majority of web traffic passes into or through one of three main cloud compute providers; Cloudflare has such an outsized impact that regional IP blocks can disrupt global traffic; and ISPs have been permitted to consolidate through mergers and acquisitions into expansive monopolies. The internet is fiercely centralized and largely closed already, which is why these ploys by shitty ISPs are likely to work absent Government intervention.

You want to protect the open internet? Regulate the shit out of its major players again. Force them to keep it open, especially when it hinders expanding profit margins.

danogentili2• 9 months ago

The current trend in govt regulation is actually going in the opposite direction, with telecom lobbies in Europe pushing for "fair share" (pretty much an implementation through law of what Deutsche Telekom and Vodafone Germany are doing right now) through the Digital Networks Act.

South Korea pioneered "fair share" govt regulations in 2016 (which caused Twitch to exit the market in 2024 due the exorbitant "fair share" fees).

stego-tech6• 9 months ago

Because western governments (and those whose governments were modeled from western regimes - like post-war South Korea and Japan) have become victim to regulatory capture and corruption. It’s why the FCC has repeatedly killed, blocked, or reversed reforms like net neutrality or “nutrition labels” on ISPs, and why South Korea gave in to “fair share” regulations that deterred further investment. Tech money is hugely influential, and the industry is almost exclusively made up of rent-seeking slumlords at this point, particularly at the top (Oracle, Microsoft, Google, Adobe, Apple, etc). It’s why DMCA reform is blocked, why pirates get jail time while AI grifts get a hand-wave, and why right-to-repair or data privacy remains a fractured and piecemeal reform instead of a national agenda item.

The problem isn’t regulation, but regulatory capture ensuring companies get the regulations they desire and benefit from.

wmf6• 9 months ago

Unfortunately you're going to get regulatory capture and extortion when the "bad guys" are local but the "good guys" are foreign.

kmeisthax3• 9 months ago

Regulation isn't good enough. The government needs to make their own competing ISP.

Hell, at least in the US, there's precedent for this: government builds and maintain all the roads; they run most transit and intercity rail operations; and they run physical mail delivery. At one point they even owned most of the railroads[0]. Communications and travel infrastructure are things government is moderately good at.

For some reason, we just decided not to have a government-sponsored telecom company, even when Ma Bell made it patently obvious that having all the country's telecom infrastructure be privately owned by one company was a bad idea. It's obvious that a government-run ISP is about as crucial to life in 2025 as a government-run postal carrier was in the early 1800s.

[0] In the 1970s, all of America's railroads went bankrupt. First, they discharged their passenger rail mandates into Amtrak, then they went bankrupt anyway, and then they got nationalized.

stego-tech6• 9 months ago

As I’ve stated in other comments, the reason western governments don’t do this more often boils down largely to regulatory capture. Every single time there’s been a large mobilization of efforts to regulate some aspect of tech - municipal broadband expansion, cable box standardization and openness, right to repair, DMCA reform, privacy reforms, mandatory binding arbitration clauses, EULA’s, provider monopolies, etc - tech money floods into regulators and political races to counter the will of the mobilizers and their supporters. Then those same ghouls repeat mantras like “disrupt” and “deregulate” to convince people that actually it’s a good thing you only have three cellular networks, one cable provider, one telephone provider, two operating systems, and four media conglomerates to choose from. At one point these slimeballs claimed anyone who used anything else (like Linux, or GrapheneOS, or FOSS) was obviously a criminal who wanted something for nothing, such was their fear of an open ecosystem.

Regulations get a bad rap because for decades the only ones to really get passed have only entrenched existing players and (rent-seeking) business models while blocking new entrants or competitors. I’m 100% in agreement with you that every single state and country should have an internet network that’s open access and governed solely by that country’s constitutional law - a sort of digital state, if you will, with which they can court business and interest groups alike to represent their interests globally. Instead, we’re presently stuck with a “whoever donates the most money to politicians wins” model, and that means the open internet exists in spite of the interests of Capital, not because of their good graces.

panick21_2• 9 months ago

Hell, at least in the US, there's precedent for this: government builds and maintain all the roads; they run most transit and intercity rail operations; and they run physical mail delivery.

Correct me if I'm wrong but it seems to me the US government is doing a terrible job at all of these.

fweimer2• 9 months ago

Is doing business with Inter.link really structurally different from getting connectivity to an exchange like DE-CIX and doing business there? I know that in theory, you get settlement-free peering at exchanges, but only for those networks that participate.

And who funds Inter.link? Their publicly available balance sheet shows significant, growing debts to a linked company, but it doesn't mention its name.

inemesitaffia17• 9 months ago

Depends what exactly you're buying from them L2 yes. L3 no

bbzylstra2• 9 months ago

I'm surprised to read an obviously AI written article ("This isn't about efficiency—it's about extraction") from a tech/news site. Does anyone else find this weird? It make me question the editors note about how much background research was actually done.

unethical_ban• 9 months ago

It didn't seem AI to me.

oasisbob2• 9 months ago

The easiest way to notice it is its excessive use of what Wikipedia calls "negative parallel construction". [1] It's a super common genai tic

This isn't about efficiency—it's about extraction
The problem isn't your connection to Vodafone—it's Vodafone's restrictive connections to the rest of the internet.
Vodafone's exit from public peering isn't an isolated technical decision—it's part of a broader pattern of large telecoms trying to reshape internet economics in their favor

The more obnoxious signs though is the excessive length, loose structure, repetition, and lack of serious editing. Writing ~3000 words used to take quite a bit of effort, so you'd need to be at least a strong enough writer to organize and structure your thoughts to make it that far. Now it's so easy anyone can put out tons of generated content on whatever topic they want.

[1] https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing

littlestymaar2• 9 months ago

It looks like the website was founded not so long ago, and I guess AI-generated slop is the only way to make a profitable web media nowadays.

bbzylstra2• 9 months ago

Also, am I going crazy or do the "comment" and "share" buttons under the header just tick up but don't allow you to actually do anything? This feels like a vibe-coded website but it could be Firefox being weird.

PhilKunz5• 9 months ago

HeyHey. The website uses Ghost right now, and a lit based web components catalogue. Some features are not yet entirely carved out. The commenting system being one of them. Components are ready, but some integration work has to be done. We also want to enable highlighting stuff directly, so people can comment on specific referenced stuff...

The commenting APIs in ghost are a little obscure.

oasisbob2• 9 months ago

Hate it, but doesn't surprise me one bit.

This article is 2700 words of repetitive slop. It seems that people are adapting to this new world.

littlestymaar2• 9 months ago

The disclaimer at the top of the article is really mind blowing:

We may have failed in some areas to grasp the issue entirely. The reader is advised that not everything might be correct and you should follow the sources and conduct your own research to get an adequate understanding of the subject at hand.

alienbaby• 9 months ago

it sounded to me like an AI generated article covering it's ass, or at least written by someone who did naught but AI generated research as their own 'research'.

bell-cot• 9 months ago

Not to say that Mr. Musk seems popular in Germany, nor that orbital dynamics are friendly to high user densities, but ...

noAnswer4• 9 months ago

https://inter.link "Trusted By: SpaceX"

inemesitaffia17• 9 months ago

SpaceX used to pay AS33891 Core-Backbone GmbH to reach AS3320 now they pay 1299 (German (landed) Starlink users) for that

Interlink is to get to France Telecom/ Orange from German Starlink users

And to get Italian/Bulgarian landed traffic to AS3320

From what I see most of the traffic that flows via inter.link from SpaceX is point to point traffic. Not regular IP transit like what they get from 1299 or GTT

SpaceX is settlement free in Germany, customer elsewhere.

uyzstvqs• 9 months ago

It's very simple. I host my stuff on a network with an open peering policy. If you as an ISP somehow have peering issues with that, then that's a you problem. I will not pay a ransom to some shady middleman that you decide to use because your network admins are too lazy. I will (rightfully) blame you and tell your customers to switch ISPs if they have issues.

Play stupid games, win stupid prices. Just wait until Vodafone Germany customers get slow speeds and an automated warning banner on every other website they visit. "Too big to fail" until it isn't.

LaurensBER2• 9 months ago

This is a very reasonable approach until somehow your competitors service loads twice as fast on the Vodafone Germany network.

As a business, at that point, you're basically extorted to pay the ransom or deal with a loss of revenue. Since the ransom is most likely lower it won't take long for your other competitors to start paying it as well leaving you with an objectively worse product, irrespective of your warning banner (which lefty Linda or Gradma Garry isn't going to understand).

inemesitaffia17• 9 months ago

The network is buying transit from someone

lwn• 9 months ago

I recently moved to a Dutch municipality that runs its own non-profit ISP. They installed a symmetric 1 Gbps fiber connection with a static IP at my house for 40 euros per month.

The service is solid, there’s no upselling or throttling, and hosting things from home just works. I bring this up because when we talk about “open”, “fair” and “monopolies” the model of a local, non-profit ISP backed by the municipality could offer a real alternative. It doesn’t directly solve the peering issues, but it shifts the balance of power (and cost) somewhat.

LaurensBER2• 9 months ago

While this is great when it works it does raise some interesting challenges, what happens if the ISP loses money, should taxes be used to cover the cost since this is a public service? Is it reasonable for this ISP to undercut commercial offerings? Internet is in a weird grey zone where it's almost a utility but not on the same level as water or sewage.

I'm glad this non-profit ISP exists but on a national level I would prefer (strong) net neutrality laws. Probably not an issue in NL but in less developed countries neutrality isn't guaranteed.

redserk• 9 months ago

Why make the assumption that municipalities must treat their internet services as a second-rate utility? Many local governments are competently run.

If the internet is out, it's going to be just as visible and probably will yield as many complaints as losing power, sewer, and water.

linohh• 9 months ago

Usually these ISPs are part of or under the municipal utility provider. So if they lose money, it first gets offset by profits from other utilities and eventually yes, the taxpayer will step in, directly or indirectly. No big deal. No one is complaining about subsidies for water or power lines in rural areas, neither should they when it comes to internet. Remember: These ISPs were founded because the market was already failing to provide decent offers or any at all.

wat100002• 9 months ago

If it loses money, do some combination of raising prices and cutting costs. Aim for an average of zero profit/loss over the long term. Undercutting commercial offerings is perfectly fine if it works out that way.

oersted• 9 months ago

There's voting with your wallet and voting with your, well, vote.

In some sense a democracy is also a market and can lead to efficient allocation of resources, particularly common resources for common good.

This is why public utilities tend to work so well in practice. People, especially in the US, don't seem to realise that such services are also subject to strong market forces, just a different kind of market.

Voters care a lot about good public services, and they also care a lot about not getting taxed much. This can lead to very efficient outcomes in well functioning democracies, often more efficient than those that come out of private enterprise, when it comes to services that most of the population needs.

rcbdev• 9 months ago

In Austria the Internet, like the postal service, is a Universal Service ("Universaldienst"). As such, any completely unserviced citizen can petition the current Finance Minister to decree an ISP of his choice - usually A1, which is the privatized form of our former public office for postal services and telegraphs - to service their area. The costs of facilitating the servicing of this area are covered by all active ISPs of a certain size operating in Austria.

Telecommunications law in Europe is a very interesting thing.

thelaxiankey• 9 months ago

i've wondered for a long time why this isn't a more common solution to these services that are almost inevitably monopolous. power, water, and internet kind of things.

jimbokun• 9 months ago

In the US the governments have actively killed them as a favor to the large corporate Internet providers.

j-bos• 9 months ago

Good explainer vid: https://youtu.be/CIEQPwf9MHY

tldr: one town in the US did it and it became an economic miracle, big telcos noticed and have set up lobbying and advertising infra to ensure it never happens.

port112• 9 months ago

In Europe these were largely publicly-owned, but the neoliberal tendency to privatise everything has slowly dismantled the public corporation.

My home country's formerly public energy provider has a weird share structure: a Chinese company and BlackRock add up to a fourth of the stock. No foreign investor should really be buying up stock in critical infrastructure.

This will always upset me.

radeeyate• 9 months ago

And here I am in the US paying $50+/mo for CenturyLink to give me 20Mb down and 0.5Mb up.

emeril• 9 months ago

shhhh.... don't say any.... the US is the bestest country in the world...

we're also really good at feeding our poor and disabled too

ulfw• 9 months ago

And here I am in Hong Kong paying a too high (there's cheaper offerings) 248HKD (USD32) for 2gbit/s up/down

ProllyInfamous• 9 months ago

Somewhat similar over here in Chattanooga (aka "Gig City"): our city's Electric Power Board has offered synchronous gig fiber for $75/month, a 300mbps "slower option" for $58/month — it's plenty fast — or 25gbps for $300/month... to all electric customers [0].

Of course our lobbied state congress critters passed a law to restrict this, so EPB can only offer internet to a limited geographic area (under the auspices of network monitoring of power delivery) — wouldn't want their Comcast-bros to have any competition! Certain apartment complexes are exempted, which prevents you from using EPB.

Wish more jurisdictions were even allowed to do this; wish politicians weren't such whores.

[0] https://en.wikipedia.org/wiki/EPB

binome• 9 months ago

Interesting take, but generally large, incumbent eyeball networks have refrained from open peering at IX's for decades at this point. They maintained presences, but usually just to grab a few specific peers they wanted, not to peer broadly with everyone across the exchange, and the bulk of traffic from large providers into eyeball networks comes across PNIs or on-net CDN nodes, not IX.

If anything, this move to centralized PNIaaS platforms makes interconnecting with the eyeball networks even easier for smaller providers. The portals allow for straightforward visibility on what they want to charge for paid peering, and instant automated EVCs and turnup, shortcutting the long and windy process of negotiating terms and establishing individual XCs in DCs that you agree to peer in.

meibo• 9 months ago

The Telekom story mentioned in this article is 100% as bad as they make it out to be, most of the users we support with issues reaching our services are with Telekom Germany. Or in an authoritarian nation that blocks access to western services.

amaccuish3• 9 months ago

Can confirm, recently moved to o2, it was insufferable.

For anyone wondering: netzbremse.de/

EN https://netzbremse.de/en/

mft_2• 9 months ago

Would you recommend o2?

ThinkBeat• 9 months ago

Using Starlink as your Internet provider does not protect you in any way when it comes to changes in pricing, speed or connectivity.

Jackson__• 9 months ago

Yeah, the blog post took a bit of a weird turn there + the AI slop headers are also quite off putting. I know this may be conspiracy territory, but it does make me wonder whether this is guerilla marketing to ensure big E meets the sale goals for his $1T payment.

PhilKunz5• 9 months ago

except the only infrastructure to replace is the satellite dish, and not the fiber that goes into your home. Or am I wrong here.

pimeys5• 9 months ago

Awesome. The only internet connection we get IN THE MIDDLE OF BERLIN is Vodafone Cable. Deutsche Telekom wants to build fiber here, but our landlord refuses to open the door to the cellar because he wants to kick everybody out and raise the rent for new tenants.

What a time to be alive.

immibis4• 9 months ago

Deutsche Telekom is just as bad as Vodafone. IIRC the government stepped in and said they had to offer peering to everyone, so now they offer peering to everyone in some random hamlet in the middle of nowhere for 5000€ per month per gigabit, while peering at other locations and prices is still at their sole discretion.

mtmail• 9 months ago

The issue here seems to be the landlord.

socksy• 9 months ago

Part of the issue is that the landlord gets any say whatsoever about a public utility.

pimeys5• 9 months ago

Yes. It's the so called Berlin issue. If you're lucky enough to find an apartment, you have a good chance of getting a slumlord who does everything to make your life miserable.

I'd buy my own place, if there would be anything available. Probably need to move to another city or country.

lxgr6• 9 months ago

our landlord refuses to open the door to the cellar

The simple solution would be to make this illegal, i.e. require landlords to allow at least two competing wired ISPs to connect each household.

No need to make them pay for it; I suspect it would be more than enough to end their very lucrative arrangement of somehow rewarding exclusivity. (I don't have any evidence that landlords are getting paid for it by Vodafone directly, but I highly doubt that there's any above board reason for the status quo.)

sunaookami• 9 months ago

Deutsche Telekom peering is infamous for being bad, they want extremely high fees that not even Meta is willing to pay: https://about.fb.com/de/news/2024/09/warum-wir-unsere-direkt...

inemesitaffia17• 9 months ago

Meta doesn't want to pay anything. Not even Cogent/HE transit rates

naIak• 9 months ago

Refusing to peer directly is not “killing the free internet”. This level of hyperbole doesn’t help your argument.

immibis4• 9 months ago

yes it is? This level of normalcy bias doesn't help your argument, either.

globular-toast• 9 months ago

When can we start doing meshnets again? I was excited about it in '05 long before I could afford networking equipment. 20 years later and we've only regressed.

immibis4• 9 months ago

Right now. There's Freifunk in Germany. But we're all terrified of laws now. There was a time when hackers would dare to sticky-tape cables out their window and across the street. Not any more, everything's all nice and regulated and a strict order is harshly enforced. If I place a fiber anywhere outside of my apartment I expect to be deported.

everfrustrated• 9 months ago

Inter.link is openly peering on IX's tho. So I'm unclear why the angst. https://inter.link/knowledgebase/peering-policy/

inemesitaffia17• 9 months ago

No guarantees of performance to peers, no one's carrying peered traffic to other peers or upstreams.

That's giving free transit/route leaks

metanonsense• 9 months ago

Ironic that I switched to HN from chess.com because Deutsche Telekom‘s peering with Cloudflare (or more its lack of) made the site even more unusable than usual. 5 minutes ago I thought „Maybe it’s time to switch to Vodafone“

IlikeKitties• 9 months ago

Do not switch to Vodafone. Even without their peering it's still a horrible ISP

trvz• 9 months ago

Even the first paragraph is wrong:

There's a reason your internet feels like magic. When you click a YouTube video in Berlin, that data doesn't travel some convoluted path through half of Europe to reach you. It flows through something called an "internet exchange point"—a giant room full of routers where hundreds of networks connect directly, swapping traffic efficiently and, crucially, for free.

When you open a Youtube video page, the video is probably loaded from Google's caching servers located in your ISPs network.

ElijahLynn• 9 months ago

Thank you for writing this article. It is a rare take on what is happening, and not everyone has the expertise to write it. It seems easy enough to share to consumers who may be affected by this. And it does have some action steps too.

vitorsr• 9 months ago

Telefonica does this.

Until I switched, it would only peer with other Tier 1 providers 2000 mi away from my location, even though there is a large IX 5 mi from home co-located with a large regional ISP with several other networks and appliances connected to it.

I filed a complaint but it is impossible to escape the event horizon of the customer service black hole, and customer protection regulation agents fail to appreciate how clownish it is to have 100 ms ping to my university 5 mi away.

So I switched and recommended everyone within earshot to do so as well.

To this day I fail to understand the logic behind not peering locally.

inemesitaffia17• 9 months ago

The logic is money.

Also if something goes wrong with their traffic ratios, Telefonica would have to pay for transit.

weinzierl• 9 months ago

I think the open internet like it existed in the late 90s and early 2000s is never coming back. Luckily tech got cheap enough that I believe we eventually will have continent wide grass roots mesh networks and uncensorable free communication again.

20after4• 9 months ago

Mesh networks don't really scale to continent size and there really isn't a lot of free/unlicensed spectrum to work with, it's already really crowded. WISPs do a lot with what they have to work with but it's still not going to supplant the internet without some major technical breakthroughs.

sunshine-o• 9 months ago

I do not think this is about replacing the current Internet and watching Youtube or Netflix using a mesh network.

It would be more about having a slow, free and backup internet available. Nowadays, we talk a lot of cyber attacks and WW3. I am pretty sure the Internet would be the first thing to go out if things would escalate (at least at the same time as the grid).

I am not sure how long a "modern" society can operate without connectivity. The idea of a mesh network is also very cool.

fortranfiend• 9 months ago

Wow, thanks I hate it.

lxgr6• 9 months ago

Instead, all traffic will flow through a single company called Inter.link, which charges content providers based on how much data they send to Vodafone customers.

As a past customer, I'd like to challenge the implication that it's possible to send any data over Vodafone's network. (My DOCSIS connection with them peaked at fractions of an Mbps for many months during the pandemic, with latency measured in multiple seconds.)

1a527dd5• 9 months ago

Germany is on a weird path; first the nuclear power withdrawal and now this. It's all a bit topsy turvy.

lxgr6• 9 months ago

Germany has had a weird approach when it comes to the Internet for several decades, but sure, it must somehow be related to recent nuclear power policies.

glitchcrab• 9 months ago

It doesn’t make sense to conflate a decision made by the government with that of a private company, just because they happen to be in the same country.

Starlevel004• 9 months ago

When Deutsche Telekom customers want to watch YouTube, that traffic flows directly from Google's network to Deutsche Telekom's network at a Frankfurt exchange point—maybe four or five router hops, minimal latency, no intermediaries. It's elegant. It's efficient. And it's exactly what Vodafone is abandoning.

Tab closed

cachius3• 9 months ago

Why?

Dilettante_• 9 months ago

AI writing cadence I presume.

Edit: https://news.ycombinator.com/item?id=45850683

mft_2• 9 months ago

Is anyone in Germany able to recommend ISPs which aren't Vodafone or Telekom?

rurban2• 9 months ago

No. My company went with a so called better ISP because doing AI means transporting tons of image data up and down. But their peering was horrible. I couldn't update Ubuntu at all compared to fibre Vodafone at home. Took hours. Transporting the harddisc home and back was faster.

We complained a couple of times with traceroutes showing their nonsense routes, and eventually it got a bit better. But not as fast as Vodafone still

inemesitaffia17• 9 months ago

Use the same provider

Nevermark• 9 months ago

Companies claim mergers of competitors are good because they increase efficiency, to get permission to centralize markets to a far greater degree than any cartel could.

Once under more centralized control, new and old efficiencies are moved from customer benefits per charge, to conglomerate revenue per expense.

The centralization enables the change, and defends it from competitive pressure.

And regulators keep falling for it, because industry money has so many ways to push watchdog decisions in the direction they want, under the cover of relentless PR.

It would be a very blunt instrument to require companies that reached 50% market share, or $500m valuation, for more than three years to split into independent companies. In any way they wanted to organizationally and asset-wise, as long as the highest valued component was valued at less than 60% of the original. (Strategically owner/leadership designed breakups often result in a greater sum value. So more than one component may end up worth more than 50% of the original.)

A very very blunt economic instrument, indeed.

But I really think markets would become more dynamic, competition fiercer, technological growth faster, economic growth higher, and customer benefits greater.

Great for the labor market too. Both in job creation and economic mobility. The continual emphasis on developing new leadership talent for success created spinoffs would be significant.

Startups would have fiercer competition in terms of incumbent adaptation and innovation, but lower passive barriers based on scale, brand, etc.

Billionaires would continue to be minted. Warren Buffet adds value to many companies without creating self-serving keiretsu out of them. Other billionaires would tilt more toward the multi-founding pattern, instead of the single-company (or tree of controlled subsidiaries) mogul type.

(I am aware that some markets, especially some utility type markets, "want" to be monopolies due to objectively high costs of duplication. But even those can be made more decentralized and more competitive by increased modularity on functional lines, and similar decompositions, suited to specific economics and practicalities.)

lorenzk• 9 months ago

I have had sporadic problems with Vodafone Germany’s IPv4 routing for months, sent detailed analyses and refused in advance to pay for any technicians they might send because the problem could not be on my property. They sent two technicians who confirmed my view. Then they charged me 160 EUR for them without fixing the issue.

noAnswer4• 9 months ago

You have to go to forum.vodafone.de:

1. State stat the error also occurs with the Vodafone router.

2. that you already have done a factory reset of the Vodafone router.

3. that you already have turned your Vodafone router off for 24 hours and it didn't fix the error.

4. that you already talked to the hotline multiple times.

After that you have to pray that someone with the same problem comes along and endorse your problem. Like, "I have the same problem since...". This sometimes conjures a Vodafone guy who tells you he has informed the technicians. Than you have won and within less than a day the error is gone.

I was six month without IPv6 even though the error message was clear. The forum route finally worked.

0xbs• 9 months ago

They are NATing the IPv4s, i.e. multiple customers are sharing the same IPv4, which can cause problems. I called them and told them that I absolutely need to do port forwarding and asked nicely if I could get an exclusive IPv4. The support said „Of course, let me check the box in the UI for your account, should be active by tomorrow“ and done. You have to get lucky to get someone who understands your problem and is willing to fix it, though.

ExoticPearTree• 9 months ago

Greed. The shortest explanation.

There is this idea in Europe (and I think it is taking shape in other parts of the world) that content providers should also pay the ISPs for the traffic to/from them. Basically ISPs want to double-dip in making money from both sides of the pipe.

And this needs to be put to rest, otherwise we'll pay for the Internet access like we pay for cable TV: Netflix - $5/mo extra, HBO - $3/mo extra, Facebook - $2/mo extra.

I am all for capitalism, but greed needs to have a hard cap at some point.

Vodafone Germany is changing the open internet, one peering connection at a time

Related Stories