Heartbreaking: 6509 PSUs Fail After a Brief 22 Years of Service

https://www.jumboframeinternet.com/post/1/

by jrrtokenring • 4 years ago

274 114 4 years ago

RESEARCH TRIANGLE PARK, NC, CISCO SYSTEMS INC 25-Dec-2021

After only 22 years of service, a 6509 chassis deployed at a customer location has reportedly reached its final end of life. Taken from us in its prime, lon1-gandalf01, known simply as “Gandalf” by those close to it, put in its final notice after the power outage at Interexion’s LON1 facility killed off the last working PSU. Network Engineer Rich Ikhanda, the technician tasked with placing gandalf into its final resting box, stated “it’s a real shame to see this router taken out of service while it still had its youth. I mean, it was barely out of its teens! A true tragedy.” Rich reminisced that “once we disabled that bastard protocol, IPv6, we could easily fit the entire v4 table into Gandalf’s TCAM with plenty of room to spare, until everyone else had the same idea and even the “real” internet table grew too large, referencing the IPv4 default free zone. Luckily we were able to slice, dice, and julienne using as-paths, like real network engineers.”.

Gandalf will be replaced by a UniFi USG-Pro nicknamed “Gandalf the White”. When asked if there were any concerns with Gandalf’s replacement, Rich stated that “I am not sure about this whole app thing to run a network, but it has gigabit interfaces, it should be good to go.”

Gandalf is survived by nearly one million IPv4 routes and a 6513-E named Frodo, all of whom will feel the impact from this loss (as they continue to reconverge).

wmf2• 4 years ago

The really sad part is that Cisco keeps selling 10-year-old obsolete chassis when newer systems have so much lower cost per bit.

jrrtokenring8• 4 years ago

Why innovate, when you can stagnate!

aspyct• 4 years ago

Alternate perspective: if it ain't broken, don't fix it.

ikiris2• 4 years ago

That’s a competing business unit.

smarx007• 4 years ago

Cisco stopped selling them all: https://www.cisco.com/c/en/us/obsolete/switches/cisco-cataly... and https://www.cisco.com/c/en/us/support/switches/catalyst-6500... (2006 and 2020 respectively). The post is the "onion".

testrun• 4 years ago

I absolutely love your username.

jrrtokenring8• 4 years ago

Thank you :)

faeyanpiraat• 4 years ago

My friend asks what is it to love about it, he doesn’t get it?

iso16312• 4 years ago

JRR Tolkin wrote Lord of the Rings

Token Ring was an old networking technology (before ethernet became ubiquitous)

barbazoo• 4 years ago

PSU is the power adapter?

jrrtokenring8• 4 years ago

Yes, PSU is the Power Supply Unit. They are known for failing when they are turned back on after a power is lost.

richardfey• 4 years ago

In my case also after turning off and on the building main power switch, after a brief 12 years of service.

tialaramex• 4 years ago

This is why your 6509 should have dual PSUs.

geocrasher• 4 years ago

My first thought was "MAN... that's a lot of PSUs"

krallja• 4 years ago

I was trying to figure out which Commodore products the 6509 was used in.

tssva• 4 years ago

MOS did make a 6509 processor It was a variant of the 6502. Commodore used it in the CBM-II line of computers.

spicybright• 4 years ago

Everyone knows you measure PSU life in dog years. 154 years is a damn good run!

par• 4 years ago

How many servers out there named gandalf? I've got one in my own house.

nine_k• 4 years ago

I had one in our org back in the day.

We had a number of boxes named following this theme. For instance, the firewall box was named pelennor.

k_sze• 4 years ago

All of my network devices have JAV actress names, lol.

vincnetas• 4 years ago

Is this story by any chance about you?

  Every variable and every method name was given a female first name. "Each of these is named for one of Sten's girlfriends." Given the number of names required, it was improbable that these were real girlfriends, but Sten gave no hint about this being fiction.

https://thedailywtf.com/articles/my-many-girlfriends

dendriti• 4 years ago

well, that's gross

na85• 4 years ago

Cringe intensifies.

no_time• 4 years ago

May their filesystems get the fsck'ing they need

dylan6043• 4 years ago

I've seen elven names, I've seen planets, I've seen Hindu deities, I've seen 7 dwarves, and then there's boring corp names that actually provide useful information about purpose/location/etc.

You can definitely tell personalities by system names.

DonHopkins5• 4 years ago

I dialed into a Gandalf Box at the university:

https://en.wikipedia.org/wiki/Gandalf_Technologies

LinuxBender• 4 years ago

None in mine. His plot modules conveniently put him at other houses when my hobbits are in trouble. I did have an attack box named "Sting" however.

CPLX• 4 years ago

I go for Douglas Adams references across the board, which I think may be tied with Tolkien as the global leader for this sort of thing.

kQq9oHeAz6wLLS2• 4 years ago

None here, but I have a trio of OpenBSD servers named Athos (MariaDB), Aramis (http), and Porthos (mail).

TradingPlaces3• 4 years ago

This whole blog is a hoot.

https://www.jumboframeinternet.com/post/2/

jrrtokenring8• 4 years ago

Thank you :)

6LLvveMx2koXfwn3• 4 years ago

Satire is fine but this 'nice' thing is really galling.

TradingPlaces3• 4 years ago

You’re gonna have trouble topping this lede:

"Engineers actively participating in the NANOG email list had a rare breakthrough on a major problem plaguing nearly no one on Wednesday evening."

kingcharles• 4 years ago

Username checks out...

ezconnect• 4 years ago

The home page have redirect problems on Firefox and cant load.

LilBytes• 4 years ago

I found it was my ad blocker

tempodox2• 4 years ago

Once you get the page to load you'll be able to read that “Jumbo frames don't work on the internet”. You may have been the first to actually prove that point :)

mplabelspace3• 4 years ago

I suspect there may be semi-regular updates =)

RantyDave• 4 years ago

Taken so young, the poor parents.

jrrtokenring8• 4 years ago

"The router that routes twice as fast routes half as long"

wmf2• 4 years ago

Has it seen packets you wouldn't believe?

DonHopkins5• 4 years ago

NOTE: This packet is sold by net wait, not by volume. Packed as full as practicable by modern automatic equipment, it was delayed the full net wait indicated. If it does not appear full when opened, it is because contents have been compressed during shipping and handling.

lostlogin3• 4 years ago

If Gandalf the White is anything like my UDM-Pro, I hope the backups are good. Mine self erases the config periodically, most recently 10 hours ago.

jrrtokenring8• 4 years ago

But I mean... gigabit interfaces it must be good!

mise_en_place• 4 years ago

I wish I still had gigabit interfaces on my workstation. dmesg is constantly complaining about my 10 GigE interfaces

lostlogin3• 4 years ago

I use the 10gb, and it’s on a UPS.

Tempted to move back to an edgerouter. Or go Pfsense.

topspin• 4 years ago

Ironically there is a good chance your problem is also power supply related.

lostlogin3• 4 years ago

Maybe, it’s on a UPS, but that doesn’t mean it isn’t the power.

octoberfranklin2• 4 years ago

Ubiquiti's Octeon hardware is amazing -- but only after you expunge the horrid "bonus software" that they throw in there for free.

ssklash• 4 years ago

What's crazy to me is that a massive 6509 could be replaced by a tiny UniFi USG-Pro.

ikiris2• 4 years ago

Are there unicorns in this fantasy world too?

dylan6043• 4 years ago

It sounds like the tiny UniFi USG-Pro is the unicorn you speak of.

belfalas• 4 years ago

Dear OP, my username bows to yours.

rlt• 4 years ago

"J.R.R. TokenRing"

azalemeth• 4 years ago

Fun fact: in the University of Oxford (former home of one JRR Tolkien) each building has a router and switch cabinet where JANET comes into the building and is distributed. What is this called? Well, it's the Front Door to the internet, or Frodo for short...

badrabbit2• 4 years ago

The 6509 and 2811 and other older models are amazing. I use catalyst switches and routers I bought from ebay and craigslist almost 10 years agi that were probably resold several times after production use. I am no longer in networking but I was always impressed when I ssh into some arbitrary switch or router to troubleshoot and see 10+ year uptime. I bet 20+ is common these days.

I am sure others have stories in this regard. Someone told me their remote router died after 12 years uptime but only because of a lightning strike.

From a software perspective, the Cisco IOS has been doing "unikernel" before it was hyped. Can't say that's a factor but having a standalone binary as the entire OS might perhaps make testing and shipping a sturdy stack easier?

netfl0• 4 years ago

Came here looking for reminiscing. You could do wild things with the right 6500 setup. It was like driving a manual transmission.

iso16312• 4 years ago

I'm embarrased to say I have some cisco 3750 edge switches with 10+ year uptimes. They are isolated at a management level but really should have their firmware upgraded. Arranging downtime with 30 different stakeholders who don't subscribe to the concept of multiple services capable of failure is never easy.

manbart• 4 years ago

AFAIK all the newer Cisco IOS boxes are running Linux now, i.e. IOS XE

badrabbit2• 4 years ago

They are. All the Nexus/nxos line as well. You don't hear people rave about their reliability as much as IOS.

octoberfranklin2• 4 years ago

Gandalf will be replaced by a UniFi USG-Pro

Octeons are truly amazing chips.

Get them while you still can.

raverbashing• 4 years ago

I'm surprised no one with a white beard pulled out a soldering iron out of their drawer and tried to fix it. Or maybe they did...

peterkelly• 4 years ago

I love how many of the comments here think this story is real, and not just the first article published by what appears to be a newly-launched satire tech news site inspired by The Onion. Follow the "prev page" links at the bottom for other articles, you won't be disappointed :)

tempodox2• 4 years ago

And we miraculously see that site although “Jumbo frames don't work on the internet!” :)

xbar2• 4 years ago

It's a good site and I'm not disappointed.

But 6509 longevity is real and the best satire mirrors our real experiences.

Was there ever a gandalf 6509? Yeah. Probably 10,000 of them and many are 22 years old. My 22 year old gandalf/bilbo pair were actually VA Linux hosts. But no doubt lots of 6509s were so named.

newaccount742• 4 years ago

Not totally on topic, but can someone explain why switched mode power supplies don't seem to have any type of inrush current limiting?

Often when I turn on power right after a power loss, the circuit breaker immediately trips. I assume it's because all those power supplies have such a big inrush current that they overload the circuit. All these devices use less than an amp in total on average, but the inrush current is enough to immediately trip a 13A circuit breaker.

fake-name• 4 years ago

Because that's another part that adds to the BOM cost and assembly cost.

SMPSes all have input stages that effectively directly rectify the incoming mains to provide the SMPS internal bus voltage. This is also a great way to basically design an inrush current simulator (diode(s) + big capacitor).

Adding stuff like active PFC correction in your common wall-wart is a non-starter. Those things are already massively cost-optimized.

namibj• 4 years ago

No they have enough that they don't vote a breaker when you plug them in the first time. Also toroidal cores are worse than modern active PFC SMPSs.

Also note that there are different breaker classes for different loads, with the more-forgiving ones having tighter demands on the insurance of the downstream wiring, because they have to still reliably trip instantly from a short.

Ballas• 4 years ago

They should have an NTC for inrush limiting.

newaccount742• 4 years ago

How long do those NTCs take to reset? Maybe that explains why the breaker trips after I turn it back on after a power failure -- if those NTCs are still warm, they would not limit the inrush current.

zeroflow• 4 years ago

Regarding the breaker side, there are different characteristics available.

For example, I've switched the breaker of my workspace from B13 to C13, which both trip at above 13A (plus some) continuous, but C13 has a higher limit for short pulsed loads.

If you take a look here, you can see that a "B" breaker accepts 1 second of 5x load, while C allows for 1 second of 10x load, so up to 130A surge and more for shorter periods. https://www.klocknermoeller.com/faz/faz.characteristics.htm

xxs• 4 years ago

Not totally on topic, but can someone explain why switched mode power supplies don't seem to have any type of inrush current limiting?

They absolutely do - NTC, and they even have a relay to bypass the NTC after the PSU has booted up. If you hear a click after booting up, there is a bypass relay.

Often when I turn on power right after a power loss, the circuit breaker immediately trips.

The circuit breakers (domestic) tend to be B or C class, B triggers much easier than C, the latter tend to be used for motors, stuff with high inrush current. If you believer you have too much inrush current, you might wish to replace the breaker. Also, the PSU datasheets specify how many of then can be connected in parallel to the same breaker.

Last but not least - if you have too much inrush current, it's likely there is no power factor correction either, so the high value caps are connected straight after the bridge rectifier.

Zenst2• 4 years ago

Reminds me of the fun of naming hosts before they became all formalized. Recall in the late '90s at a large Medical distributor, how we set the user/client DHCP pool names from a virus database (there are more virus's than stars in the Universe apparently). Was a nice in-house admin chuckle that AFAIK, nobody in upper management ever found out about. Our in-joke was that virus's kill hosts and the thinking was that without users, every server/service ran just 100% fine.

hnarn• 4 years ago

I know servers are supposed to be "cattle" but I can't get away from the feeling that having at least a part of the name being somewhat sentimental is a nice thing. The default naming scheme for many containers, for example, will combine an adjective and a noun to create a unique name. I myself wrote a pretty silly script once that randomly returned the name of a British or German naval warship. It seems to me that "formalized" naming doesn't have to be exclusive with sentimentality, you can just add it at the end, like:

geotag-srvtype[#]-prod-[pokemon_name_or_whatever]

That way, if your environment is completely fluid, it becomes obvious when a node has re-deployed, and if it's not, you can just re-use the old name when re-deploying instead of generating a new one.

You can of course include hash-looking things in hostnames as well, but I'm a strong believer that hashes are for computers, not for humans to read (most of the time).

tomjen3• 4 years ago

You totally should name the container classes after battleships.

Then you can truthfull say you deleted/sunk the Bismark.

Anyway I think there is nothing wrong with naming classes of things after known things - so you are deploying two more Victor class frontend servers or Dreadnaught monitoring services.

The trouble becomes when you care about Dreadnaught-oux3z and treat it as not disposable.

DonHopkins5• 4 years ago

You could name your servers after cuts of beef, beef recipes, and hamburgers, like the WOPR Computer and Big Mac:

https://www.youtube.com/watch?v=iRsycWRQrc8

https://en.wikipedia.org/wiki/Big_Mac_(supercomputer)

GekkePrutser• 4 years ago

Not everyone buys into the cattle not pets thing. It's mainly a thing related to auto deployment strategies. Which are great for the cloud with its quick auto scaling capabilities. The ideal vision there is that you shouldn't even have the ability to log in to a server. But there are other usecases where a known environment you can poke around in is more useful IMO.

As an example, in cloud environments not much time is spent investigating issues. When something fails the solution is often to just wipe and redeploy. I prefer fully understanding the problem and having the tools to troubleshoot. Especially because in my work the servers still have different roles.

estaseuropano• 4 years ago

I'm sure there are many vira, but that comparison can't be true.

Astronomers estimate there are about 100 thousand million stars in the Milky Way alone. Outside that, there are millions upon millions of other galaxies

https://www.esa.int/Science_Exploration/Space_Science/Hersch...

hobs• 4 years ago

I assume its "named" viruses instead of just a number/designation.

Zenst2• 4 years ago

That is a lot less than then number of viruses and article linked "An estimated 10 nonillion (10 to the 31st power) individual viruses exist on our planet—enough to assign one to every star in the universe 100 million times over."

https://www.nationalgeographic.com/science/article/factors-a...

DonHopkins5• 4 years ago

Yeah maybe so, but how many of those 100 thousand millions stars have millions upon millions of their own viruses?

tmoertel• 4 years ago

Ah, the fun of naming hosts. I once named hosts using a long list of trees I found in a scholarly publication. The list also gave botanical names, so for a chuckle I placed them in the corresponding TXT records. You could `dig paperbark-maple` to learn it was "Acer griseum". The good old days.

ChrisMarshallNY• 4 years ago

I like the screen name.

There's probably a whole lot of folks, that have no idea what a "token ring" is.

xbar2• 4 years ago

For reference https://dilbert.com/strip/1996-05-02

franga2000• 4 years ago

I teach computer networking in high school and it's still a part of the curriculum. I have no idea why, but it had absolutely no difference in importance between Token Ring and Ethernet, despite the latest revision of the curriculum being written in 2017.

kloch2• 4 years ago

There is a known hardware bug affecting every PSU and cards in the 6500/7600 series. It manifests as a significant chance of permanent failure on power-up after being on for several years. As long as the chassis stays powered everything is fine. If you reboot or power cycle you may end up with some random dead cards or power supplies.

The use of the "enterprise" 6509 as a backbone ISP router was the worst kept secret in the industry in the 2003-2007 timeframe. During that time you would see about about 60% of the routers in a carrier hotel were 6500/7600's. Most of the rest were Juniper MX or M with very few Cisco 12k, ASR, or CRS. The 12k used an inefficient fixed cell size architecture and focused on ATM line cards and the CRS1 was far too expensive for most operators. The Ethernet-based 6000/6500 series was originally developed by the enterprise vertical at Cisco as an L2 product, but it gradually evolved to full route scale L3 features at a much lower price point than the "real" ISP routers. This price advantage was especially dramatic when 10GE interfaces for the platform came out in 2003. Cisco replaced the 12k with the more popluar Ethernet focused ASR series but the very low cost of the 6500 kept them in use much longer than they should have been.

While dirt cheap due being officially marketed at enterprises and a robust greymarket for parts, the 6509 was far from an ideal platform for ISP's. It generally performed well but there were a number of quirks from an internal architecture that gradually evolved from L2 only->full scale L3. RP cpu/ram was woefully under powered and control plane policing was marginal at best. Any significant packet flow to the RP instead of being forwarded by the ASIC hardware would instantly cripple the router and cause it to drop all BGP/OSPF/IS-IS sessions. In addition to broken control plane policing there were more scenarios where packets could be punted to the RP unexpectedly compared to other platforms. For example if any of the TCAM partitions filled up then ASIC forwarding was immediately disabled and ALL packets were sent to the RP! You also had to watch traffic levels when round-tripping on adjacent 10GE ports on the most popular cards to ensure it didn't exceed about 6 gbps or those ports would start dropping packets. Real ISP routers can often do full line rate in+out on all ports.

The monolithic IOS for the platform was notoriously buggy, partly due to the architecture and partly due to an enterprise feature set that most ISP's didn't even need. It was wise to avoid any version where the release notes claim to have just fixed a catastrophic bug (you wanted to see only boring fixes to cosmetic bugs or features you would never use) and to search the c-nsp mailing list for other's experiences with the version. Otherwise you were playing "IOS Roulette". Because of this it was common to run a known stable version much longer than normal, running the same version for 5+ years in some cases. After the fail-on-power-up bug became known most left them running without even a reboot until they could be replaced.

In more recent years Arista has made a similar play by evolving their switches from originally focused on minimum L2 latency to full route scale L3, and more recently adding full MPLS support at a lower price point than the equivalent C/J products.

exikyut• 4 years ago

This is very interesting insight, and being able to absorb cohesive info as an interested but unfamiliar outsider is appreciated.

I had a bit of a cursory google and probably quite predictably didn't find much beyond cisco issue reports - is there any speculation out there about the actual root cause of the hardware faults you refer to?

kloch2• 4 years ago

Here's the field notice: https://www.cisco.com/c/en/us/support/docs/field-notices/637...

and more general discussion: https://www.cisco.com/c/en/us/support/web/memory-component.h...

The field notice only applies to cards but the power supplies were definitely prone to permanent failure on power on, possibly from a different type of failure. I replaced more dead power supplies (that had died on power up/cycle) than line cards over the years that I used them.

sandos• 4 years ago

Oh, I read that 6502 and thought it was about Commodore 64. Was cofused that they would replace it with a router thing :)

camb416• 4 years ago

I understood some of these words

Heartbreaking: 6509 PSUs Fail After a Brief 22 Years of Service

Related Stories