Posterous is being DDoS'd

http://twitter.com/posterous/status/21082191639

by superjared • 16 years ago

84 45 16 years ago

superjared• 16 years ago

This is hot off the heels of an email sent out last night:

As you’re no doubt aware, Posterous has had a rocky six days.

On Wednesday and Friday, our servers were hit by massive Denial of Service (DoS) attacks. We responded quickly and got back online within an hour, but it didn’t matter; the site went down and our users couldn’t post.

On Friday night, our team worked around the clock to move to new data centers, better capable of handling the onslaught. It wasn’t easy. Throughout the weekend we were fixing issues, optimizing the site, some things going smoothly, others less so.

Just at the moments we thought the worst was behind us, we’d run up against another challenge. It tested not only our technical abilities, but our stamina, patience, and we lost more than a few hairs in the process.

I’m happy to report Posterous is at 100% and better than ever. Switching to a new data center will help us avoid the type of attacks we saw last week, and the new, bigger, beefier servers will speed up the site and increase capacity. We were hit pretty hard, but we’ve come out stronger in the end.

While we were certainly frustrated, we know that no one was more frustrated than you. Your website was down, and I humbly apologize for that. Know that throughout these six days, restoring your site and your trust has been our number one priority.

At the same time, I have never been prouder of the Posterous team. Garry led the team in the server migration, Vince worked like a mad man until he passed out on his desk, and Jackson helped the community understand the impacts of some pretty technical issues. This week has been all hands on deck, and we had some pretty great hands.

We have great plans for the future of Posterous. Our team is stronger than ever, and we are laser focused on developing better ways for you to share your content online. Thank you for believing in us, thank you for trusting us, and thank you for sticking with us through the past six days.

--Sachin Agarwal Cofounder and CEO, Posterous

ryanhuff• 16 years ago

This sort of crisis management is always a challenge. How do you communicate this sort of problem to customers without losing credibility later if there is a risk of lingering issues? Based on the tone of the email that you posted, as a customer, I would expect the problem solved, case closed. Certainly, it makes customers feel good that the team has it handled.

The new events, if not handled quickly, could create doubt in customer minds and the team, as the optimistic outlook may not hold. I hope they beat this soon.

zaidf• 16 years ago

Simple rule about DDoSes: there are no guarantees.

codemechanic• 16 years ago

Yes. It is getting DDOS'd now. I just posted a post. But can't access it.

gaiusparx• 16 years ago

So this is the 2nd DDOS attack. Can DDOS attack be fended off?

Mistone• 16 years ago

2nd DDOS in a matter of a week.

bradgessler• 16 years ago

By crushing the enemy.

I hope Posterous does a write-up about this on Axon Flux when this is all over. Poor fellars.

rantfoil5• 16 years ago

Usually it means using a hardware solution like Intruguard or going through a third party vendor that resells Intruguard and/or other proprietary anti-DDOS technology.

We've opted to use Gigenet, and they're holding the line right now quite well.

mrduncan• 16 years ago

Our datacenter is experiencing heavy packet loss. We're on the line with Rackspace now.

I don't see where they stated it was a DDOS attack. Packet loss can occur due to a large number of different issues.

EDIT: They just clarified with the following: The DDoS attackers have returned and evolved their attack around our countermeasures. We expect to be back online ASAP w/ @gigenet antiddos

nroach2• 16 years ago

It's interesting that they're basically saying Rackspace's network couldn't hold up and doesn't have an effective solution for DDOS victims on their network.

Gige's DDOS protection is basically a hosted redirect and filtering system that "leverage[s] the cost of DDoS mitigation amongst a large group of businesses, giving you access to the infrastructure that would normally be out of reach financially." http://www.gigenet.com/ddos-protection.html

But if Posterous is hosted on the Rackspace cloud, why isn't there a mechanism to do this through their existing host? After all, Rackspace has an equivalent service (maybe rebranded?) that would seem to offer protection without needing to go to a 3rd party vendor. http://www.rackspace.com/managed_hosting/services/security/d...

So it's interesting that Posterous needed to go around Rackspace to get a solution.

rantfoil5• 16 years ago

It's tough -- if someone is hitting you at > 1gbps, that exceeds a gigabit ethernet port -- typically on shared services you only pay for that much. Any more than that and it affects all the other customers on the shared network. So yes, Rackspace has the bandwidth for it, but at the same time if you were Rackspace, would you let our DDoS take down the other hosts in the datacenter?

Gigenet / Black Lotus / Prolexic comes in and says -- we have excess bandwidth, we'll take care of it. Then they proxy clean traffic to you. Unfortunately as we are learning, it's also very expensive.

vox• 16 years ago

Why DDOS posterous? What's there to gain from it?

This isn't like the ones who DDOS'ed MS out of their hatred for it or the ones who blackmailed & DDOS'ed a gambling website when they refused to pay up.

shaunxcode• 16 years ago

someone probably has an issue w/ the blog/content of someone who is using posterous rather than posterous themselves.

jackowayed• 16 years ago

That's my guess as well.

Weebly said (I think in their YC Founders at Work interview) that they get multiple DDoSes a day, aimed not at them but at sites they host. Now they have good enough systems in place to deal with it that they said they often just don't notice.

I guess when you host enough people's content, it's only a matter of time before someone wants to DoS something you're hosting.

petercooper• 16 years ago

I severely doubt this line of thought.. but they didn't make any friends with their "switch from X to us" campaign, and it only takes one bruised ego with a lack of ethics to stir up trouble.

That said, if it's like most other DDOS attacks I've seen reported, the target is probably a user of the service with Posterous merely being unlucky enough to host a particular site someone doesn't like. I vaguely recall another platform got DDOSed recently due to some Israel vs Iran type cyberwar.

kiba• 16 years ago

Must be something that they really dislike.

oscardelben2• 16 years ago

Maybe, and take it with a grain of salt, some people got infuriated by their recent switch to posterous campaign. I know this may be a very remote option, but in my opinion it's very possible.

lanstein• 16 years ago

"A grain of salt" is the expression.

mishmash• 16 years ago

Never underestimate the power of lulz when trying to understand a DDoS.

fragmede• 16 years ago

The botnet controller could be extorting websites directly... wire $5000 to my paypal account or your website will go down. Anyone from Posterous care to confirm/deny in this case?

devin• 16 years ago

Do you have any evidence that this has ever been done in the past?

It's not inconceivable, but it seems like a huge leap.

rdl• 16 years ago

This is very common with online gaming (== gambling) sites, and payment systems; anything where there is obviously money being processed.

didip• 16 years ago

Who knew what the attackers are thinking. Not too long ago DnsMadeEasy was DDoS from Korea.

Maybe there's financial incentives?

DotSauce• 16 years ago

Could this have anything to do with Posterous' risqué post claiming superiority over Tumblr?

http://www.blogherald.com/2010/06/24/posterous-slanders-tumb...

rantfoil5• 16 years ago

We're getting about 500k packets per second, 500mbps to 1.5gbps peak, it's a synflood from a botnet. Typically we can IP hop and null-route the old IP's. That usually buys us about a day until the botnet phones home to get the new IP's, at which point we just hop again. Since our DNS TTL is only 5 minutes at most we are down 5 minutes.

TODAY, the attackers hopped IP's to our new IP immediately. So they appear to be learning. But then again, so are we.

Gigenet's anti-DDoS service has helped us a ton here and is now serving as our front door IP to block the synflood. They've been really responsive.

nroach2• 16 years ago

Any suggestions for customers with DNS names pointing to posterous blogs? Are we going to need to follow you on each hop?

rantfoil5• 16 years ago

Custom domains are unaffected, no action needed.

catch23• 16 years ago

I wouldn't think so...

weaksauce• 16 years ago

Do all the caching name servers respect the TTL though?

rantfoil5• 16 years ago

They better. So far so good.

wwortiz• 16 years ago

I really wish setting DNS TTL's to 5 minutes meant that browsers and everything else (OS caching) would respect that but sadly in my experience it doesn't.

Surprising that it works so well for you.

axod• 16 years ago

Gigenet's service looks interesting. They don't provide pricing though. I'm guessing it's not cheap? Is it based on the bandwidth they'd have to use?

avar2• 16 years ago

Relying on a DNS TTL of 5 minutes doesn't work everywhere. Some ISPs will forcefully reset any TTL below 24 hours to 24h.

thaumaturgy• 16 years ago

AT&T/sbcglobal being one.

sabat• 16 years ago

Absolutely true, and completely ridiculous. What, DNS bandwidth is too high? Please.

alphadog2• 16 years ago

I seriously doubt this is the reason but they insulted a lot of people with their "switch to us" targeted attacks.

That was classless.

Malcolm_Lloyd• 16 years ago

It wasn't classless, it was marketing. No one wants their competitors poaching their clients but in business you deal with it. That said, the same thought went through my mind, I wonder if it's related.

alphadog2• 16 years ago

That sounds very clear and professional. Unfortunately, it doesn't always work out that way.

Most of their competitors are in the same situation they are in.

Being the first one to torpedo the other ship and not expecting someone to get upset or God forbid do something awful in return is naive.

I think there is a difference in poaching your competitor's customers and claiming they are "no good".

OoTheNigerian• 16 years ago

I could claim with nearly all certainty that this is a result of their campaign. The attacked several communities. Not just startups, but communities with developers (wordpress, tumblr etc). Their marketing was bold; but looking back they might have been more subtle.

paolomaffei2• 16 years ago

I wonder if you guys know who's doing it, and why.

paolomaffei2• 16 years ago

Can someone estimate the cost of this attack? It shouldn't be cheap to lose all these bots for a massive 1gbps attack.

Posterous is being DDoS'd

Related Stories