Linode is down in Fremont, CA

http://forum.linode.com/viewtopic.php?t=7080&postdays=0&postorder=asc&start=105

by jdelsman • 15 years ago

20 29 15 years ago

forum.linode.com

jdelsman2• 15 years ago

More official information: http://status.linode.com/

slyall• 15 years ago

I hate the fact that they don't link to that status page from anywhere. So when something breaks I usually spend 5 minutes looking around the site for that page before goggling or guessing it.

_hnwo2• 15 years ago

Now this is how you do outage responses & notifications.

I'm not even a Linode customer but this kind of clarity and response is exactly what I want to see from my actual service providers.

famousactress2• 15 years ago

No, it's not. A fair bit about this sucked. I found out my linode was down from Cloudkick... tried to ssh, no go. Logged into Linode.. My dashboard says that everything's peachy and my Linode is running. I start freaking out thinking maybe I ran it out of disk space and it's thrashing so I start issuing reboot requests.. safe-mode-reboot requests, etc.. They stack up. Nothing's being processed. Can't console-in either.

Finally I contact support. They were VERY fast in answering, and when they did they pointed me at status post that said Freemont is kinda out...

WELL THEN WHY THE FUCK DOES MY DASHBOARD SAY MY INSTANCE IS RUNNING?

Lame. Oh well. The 45 minutes or whatever they were offline had another 10 minutes of strapped-CPU reboot thrashing to get through before it was back up.

Oh, and this is the third time this year at least Freemont's been down.

So yeah.. cheers for being nice, and whatever.. but I'm not suuper enamored with how much love HN gives folks for ridiculous fuck-ups followed by a "Hey guys, I'm really sorry but here's a long blog post about how our flux capacitor won't ever fail us again (until next month)" blog post.

[Edit: Yeah.. I know. Downvotes were expected. Sorry for the rant, but this little outage of theirs caused a pretty embarrassing situation with an important client. So I'm happy to trade some karma for the opportunity to say 'fuck' in all caps in a public place right now. Cheers.]

[Edit: Oh, the irony! I post a totally obnoxious comment who's valid-points-to-tantrum ratio is way off the tantrum scale... Downvotes immediately. Then I edit a little (heartfelt) apology into it, and upvotes away. This is exactly the sort of psychology that's got me at least a little bit confused and intrigued about HN, and that I was tantruming on about in the original comment.]

pero• 15 years ago

You should have just jumped on IRC prior to panicking--you would have gotten an update instantaneously. Or better yet, you could idle a detached screen in there. When I had a Linode a few months back, there were constantly 500 rather helpful people in their channel.

ConceitedCode• 15 years ago

Your absolutely right about the dashboard, but does it give a notice if the whole data center is having problems instead of parts of it?

_hnwo2• 15 years ago

Yeeah, that bit about where I said I wasn't a customer of them.

So, I'm mostly just talking about the responses I see on the status page and their follow up.

As a systems dude I prefer seeing a detailed response after the incident rather than excuses.

xfax2• 15 years ago

I'm seriously considering moving my instance to another data-center. The Fremont facility has seen some disproportionate down-time these past few months.

mkrecny• 15 years ago

I had the same thought last time the Fremont center went down - and ended up moving my instance to Atlanta - turned out to be a great decision.

xfax2• 15 years ago

I am assuming you get a new static IP when you move datacenters?

idlewords2• 15 years ago

There's no defensible reason to host in a datacenter in an earthquake zone.

staunch• 15 years ago

Seriously. Earthquakes are by far the biggest cause of outages I've seen, far more than power/UPS/cooling/network/disk issues. These silly fools!

mrinterweb2• 15 years ago

The company that runs the data-center, Hurricane Electric, seems to be really dropping the ball when it comes to providing sufficient/consistent electricity.

lsc5• 15 years ago

The power system at the Fremont 1 location is really on it's last legs, and sucks badly. Fremont 2 is better. I'm not saying Fremont 2 is awesome, but it's not as ridiculously bad as 1.

They claimed they were going to buy new power equipment after the last outage; I'm actually not sure if this current outage is a power outage or what; it looks like all my servers at Fremont 1 are still up (and have been since the last outage, some 9 days ago) and the network hasn't been great, but it hasn't been down hard, either.

jdelsman2• 15 years ago

I have already opened a ticket to move to their Dallas location. Many people have told me it is the best. Wish I knew this upon first signing up with them...

soult3• 15 years ago

Hurricane Electric is a horrible ISP.

They offer bandwidth at very cheap prices but their hardware and connectivity is shit. The only reason why so many companies use HE is because they are up to 9 times cheaper than other ISPs. They got most of their peerings because they were an early IPv6 adopter.

I currently use Nearlyfreespeech.net and BuyVM/Frantech.ca for webhosting, and they both have/had problems with HE. NFSN uses HE currently only for IPv6 and BuyVM actually moved all their servers to Coresite. Prgmr.com also decided to move to Coresite, but I do not know if it was because due to problems at their Freemont location or not. (Maybe lsc can clarify this.) EDIT: As clarified by Luke, they did not move from Freemont (see his answer for more details).

lsc5• 15 years ago

I haven't moved from he.net at all yet, and I'm not sure if I'm moving from he to coresite or to svtix. Corsite is not exactly a great data center, either; I ended up with space there mostly 'cause I'm and idiot and stayed on my upstream's IPs, which gave them a lot of leverage over me, which they, of course, used. So now I'm in he.net fremont (around 3.6kw usable, not entirely used) in SVTIX (around 7.2Kw usable, not entirely used) and in coresite (around 7.2kw, not entirely used)

The cost per watt is best at SVTIX, and SVTIX has by far the best power uptime record. (On the other hand, as the reseller who strongarmed me into mpt points out, SVTIX is run by a real-estate guy; it has no right to be as reliable as it is. But, it's a friendly real-estate guy who seems to be willing to listen to me, very non-corporate, so one option is for me to get more involved with managing SVTIX. Hell, I've started packing the BOC staff with my friends already, and my last reseller has burned his bridges with this guy, and burnt those bridges quite thoroughly, so I smell an opportunity for me.)

My K5 review of market post tower, a coresite data center: http://www.kuro5hin.org/story/2011/6/2/54516/27653

Please note, that review was written for a Kuro5hin audience; which is to say, it was written for the sort of people who would have been on 4chan 10 years ago if 4chan was a thing 10 years ago, so the tone is, uh, rather unprofessional. But the point stands, MPT is not a great data center. I mean, it's worlds ahead of he.net fremont 1, but that's not saying a lot.

Now, the big advantage of market post tower, the corsite location, is that it's the center of the world, bandwidth wise, so I should be able to do some peering and get more providers as I level up my networking knowledge, and I have a cheap gigabit line between market post tower and svtix.

As for he.net bandwidth, well, yeah. It's the cheap stuff. But really, it's about as reliable as most tier 1 ISPs; the thing of it is, he.net is a tier 2, and perfectly willing to pay for bandwidth if they can't peer, so peering spats (which are ridiculously common) are not a problem. Tier 1 ISPs? if they get in a peering spat (see the level3/cogent pissing match) you only have access to part of the internet.

Anyhow, in my position, it's absolutely retarded to only have one upstream. He.net and Cogent is a mix I'm looking at, and so long as I don't f*ck the BGP config, between the two of those, I should be both more reliable and cheaper than one of the higher end carriers.

jacques_chester• 15 years ago

Just my luck. I had planned to move to Dallas after last week's outage, but one of my bloggers asked me to put it off for a while. Le sigh.

timewasted• 15 years ago

Interestingly enough my node is in the Fremont datacenter, and it hasn't been down since last week's power outage. I guess I'm one of the lucky people in the zone that didn't lose power.

Linode is down in Fremont, CA

Related Stories