Linode down in Newark data center

http://status.linode.com/

by hendler • 12 years ago

34 34 12 years ago

status.linode.com

dangrossman2• 12 years ago

"We have detected a large incoming DDoS attack to our Newark facility. We already have our administrators working to mitigate the attack. We will have service restored as soon as possible."

    [dan@palladium]# traceroute 66.175.213.170
     1  * * *
     2  ae12.dar02.sr01.wdc01.networklayer.com (208.43.118.150)  0.349 ms  0.351 ms                                                                                                                                     0.325 ms
     3  ae9.bbr02.eq01.wdc02.networklayer.com (173.192.18.188)  0.951 ms  0.966 ms  0.975 ms
     4  * * *
     5  * * *
     6  * * *
     ...

eli2• 12 years ago

Where is that quote from?

dangrossman2• 12 years ago

A support ticket I had opened before there was anything on their status page. That was the response at 11:53 PM Newark time.

j15e5• 12 years ago

Same here, got this reply from support :

"Linode 1 hour ago

Hello there,

We have detected a large incoming DDoS attack to our Newark facility. We already have our administrators working to mitigate the attack. We will have service restored as soon as possible. Please refer to the following page for status updates:

http://status.linode.com

If you have any other questions please let me know.

Thank you.

Nick"

extrapolate2• 12 years ago

Interesting their reply says that, while their status page says it is due to a power outage.

Edit: Nevermind, j15e clarified.

jrockway• 12 years ago

I think this is the first time I've ever noticed my own website being down.

j15e5• 12 years ago

Complete blackout for my nodes, can't reach them at all. Even Linode dashboard stats for my nodes are empty starting 11:51.

Almost 30 minutes now, I hope it get fix soon!

jdprgm• 12 years ago

Is it just me or does it seem like there have been tons of DDoS attacks the past couple months.

colinbartlett• 12 years ago

It sure feels that way, but I don't know if they are just being better reported now. Services to mitigate DDOS have been around for ages, so I imagine the attacks have as well.

akerl_• 12 years ago

It doesn't help that every time a service gets hit and the result is a frontpage HN post, more people notice that DoS attacks work for gaining "attention".

egray2• 12 years ago

DDoS attacks have been on the rise, I work at a hosting provider and last year we only saw somewhere around a thousand attacks, now we're seeing that monthly...

mslider• 12 years ago

We have tens of servers across 4 Linode data centers. We get hit with both data-center-wide and single-server downtime pretty much on a weekly basis.

In the span of 3 months we had something like 25% of our servers go under "emergency maintenance" downtime. Just part of the ebb and flow of "the cloud", I guess.

kevingadd• 12 years ago

Wow, that sounds pretty unlucky. Do certain data centers/servers have more downtime for you than others?

zzzeek• 12 years ago

update: "The Newark datacenter has suffered from a power outage. "

so, no DDoS after all?

extrapolate2• 12 years ago

Seems like it was caused by a power outage.

  1:09AM EDT (UTC -4): The Newark datacenter has suffered from a power outage. We are working on bringing Linodes back up as soon as possible.

bengtan2• 12 years ago

My server auto-rebooted and is back up, and has been up for 30 minutes as of writing (approx 1:00am EDT / UTC -4). Strange that mine is up and fine now, but they haven't updated the status page yet. Maybe mine is an isolated case?

j15e5• 12 years ago

Is your node location "Newark, NJ, USA"?

Mine are still very unreachable.

bengtan2• 12 years ago

Yes, in Newark. I was also affected by this outage earlier. Looks like maybe I'm one of the luckier ones (relatively speaking).

j15e5• 12 years ago

New response from support :

"Based on the information we had at the time we believed it to be a DDoS attack. We are now aware that it was a power problem affecting our Newark data center. The backup system did not work and we are still looking for additional information from the data center staff.

We will be updating our status page with any further information.

Thank you."

j15e5• 12 years ago

Nodes are now being reported as stopped, but can't be started yet.

"An issue affecting the physical hardware this Linode resides on has been detected.

We are working to resolve the issue as quickly as possible and will update you as soon as we have more information.

Your Linode will return to its previous state once the issue is resolved.

Thank you for your patience and understanding."

davidw• 12 years ago

Hanging out on #linode, it appears as if some people's servers are coming back up. This is the second day this week I've woken up to unpleasant computer news :-(

... and mine are back up.

timdev22• 12 years ago

Some of mine back up, others not. Not how I wanted to end my Saturday night.

timdev22• 12 years ago

Evidently DC issues are fixed, since some of my servers are up again. Others still down due to (related?) hardware issue. Linode support proactively opened tickets about those that are still down, and upon friendly questioning, expressed a high level of confidence that they'll be back up once drives are migrated to stand-by hardware.

thaumaturgy2• 12 years ago

They previously had a message that Linodes would be returned to their previous state. Mine was rebooted. Just FYI. Check your DBs and all that usual stuff.

znowi• 12 years ago

There goes my impressive uptime...

someotheridiot2• 12 years ago

Mine was rebooted 2hrs ago but still cant access it.

thaumaturgy2• 12 years ago

That happened to mine, too, though not for as long.

I bet they had some host filesystem corruption and they're dealing with it manually.

ommunist• 12 years ago

I have 12 websites there. zomg. Looks like they are alive now, after 2hrs outage. Did someone cut the cable for scrap metal?

BorisMelnik• 12 years ago

I feel like this should be included in every press release post down-time.

"As issues unfold, information changes and speculations can lead to improper actions based on incorrect information."

So very well said.

mmahemoff2• 12 years ago

As others have mentioned, we've seen reports of network and power outage, and now I've received a notice of hardware failure.

Just when I set up HA load-balancing, the whole data center outage takes it all down for 4 hours and counting.

    Hello,

    The host that your Linode resides on experienced hardware
    failure. As a result, drives are being transplanted to
    separate, known-working standby hardware which we keep
    available in case of events such as this.

    Again, we're very sorry for any inconvenience caused by
    this issue. Please feel free to update this ticket in the
    event that you have any additional questions or concerns.

dan_bk• 12 years ago

Does Linode automatically check for, detect and replace faulty hardware? Or is this something you have to explicitly ask them to carry out?

mmahemoff2• 12 years ago

Normally they just detect it and raise a ticket like this (which happened here).

Linode down in Newark data center

Related Stories