Ask HN: Doesn't the Rogers outage prove – network as a single point of failure?

https://news.ycombinator.com/item?id=32028484

by avi_vallarapu • 4 years ago

8 22 4 years ago

news.ycombinator.com

High Availability and Disaster Recovery and Auto-Scaling and a design in place to avoid SPOF (Single Point of Failure) always ignore Network being a SPOF. But, Network can be a Single Point of Failure too.

wmf2• 4 years ago

This has been known for a long time. Almost every data center uses multiple ISPs for this reason.

avi_vallarapu2• 4 years ago

Right. My point is - "How so many Services across Canada went down as they never designed their Services to overcome the network outage ? "

wmf2• 4 years ago

For things like ATMs I think having two network connections is pretty expensive. Companies also like to use "strategic outsourcing" where they use the same ISP for all their locations which results in the systemic outages we're seeing now.

ilrwbwrkhv6• 4 years ago

Again as I wrote elsewhere it's because Canada is a deeply mediocre country just getting by with others help and technology.

Interac the main electronic money transfer system used Rogers as their primary network. Guess what they used for their backup network? Rogers.

There is also very rampant high level corruption leading to anti competitive behaviour, oligarchy, money laundering and I can go on and on.

Point is Canada is not a good example of networks and failures. It is a good example of incompetence and mediocrity.

cloudsec93• 4 years ago

While I'm all for ranting about too much control by too few players...

How is this different then any "big" player having issues? (Think AWS, Big US ISP, large payment provider such as VISA/MC)

I don't think most NEVER thought about network issues ... but rather a calculated risk, as long outages tend to be a rare occurrence. It's honestly cheaper to eat losses on a day like today then to pay for redundancy and availability.

cloudsec93• 4 years ago

Even multiple ISPs might not help.

I remember a DC issue where they had two physically separate ISP attachments from different sides of the building. Cable cut takes out ENTIRE DC.

Why? While the ISPs were different, at one (farther upstream) point they were both leasing fiber that ran through a single conduit -- so you need understanding of the physical structure of the network, not just "different ISPs".

leastonce• 4 years ago

Network is not a SPOF. Poorly implemented network can be though. My team manages a global network. We run outage simulations quarterly ( to check for unintended consequences of changes ).

Part of that process is to down all circuits and services from a vendor.

Could vendors be sharing upstream Concentrations that create SPOF. Sure they could. That’s why we contract a single vendor to create truly diverse services.

We see 100x outages and failures from craplications vs network.

Ask HN: Doesn't the Rogers outage prove – network as a single point of failure?

Related Stories