Pastie.org host pulls hosting after DDoS attack

http://pastie.org/

by yareally • 14 years ago

128 83 14 years ago

pastie.org

veeti• 14 years ago

Sounds like an opportunity for the other managed Rails hosts.

callmeed• 14 years ago

Right, especially considering EngineYard and Blue Box are here at RailsConf

seanp2k23• 14 years ago

>"Sounds like an opportunity for the other managed Rails hosts."

They can play it up, but if you throw enough bandwidth at /any/ host, they'll null route you. Other hosts (at least, the not-stupid ones who have been in this position before) know this and (hopefully) wouldn't sling mud at them over this.

sha903• 14 years ago

Nobody is slinging mud for taking the site down temporarily. At least, without knowing the details of the attack, I don't think anyone is suggesting that there should have been zero downtime. The mud is being slung by those who don't appreciate a host dropping a client (paid or not) because of a single incident. That shows a complete lack of loyalty to your clients, again, paid or not. RM could have found a more reasonable way to drop pastie, if it really could handle the mix of takedown notices and network disruptions, but they didn't need to do it in the middle of the night with zero notice to the owner. Certainly they could have at least given him some time to migrate his stack and not have to leave all of pastie's users out in the cold --- not that it's such an essential service, but still...

petercooper2• 14 years ago

Actually, I think it might be a better opportunity for GitHub, given they have Gist. Or maybe the audiences are already too similar, dunno.

jackolas• 14 years ago

I really enjoyed this, the user experience is very nice and its a well designed site.

I tend to use Gist now because its improved and I have a client for it, but this is a shame to see.

bryanl• 14 years ago

This is a good chance to move over to http://gist.github.com for all your pasting needs.

petercooper2• 14 years ago

I use both Gist and pastie.org. While I prefer Gist and wish it would just add support for disposable gists (without making me log out!), pastie.org is awesome for the disposable stuff on IRC, etc.

res0nat0r• 14 years ago

Hastebin is pretty awesome too. http://hastebin.com/

postfuturist2• 14 years ago

DDOS attacks are a fact of life, nice to know that Rails Machine will throw you under the bus when one happens. Doesn't seem to fit their homepage description: "You write Rails apps. We deploy, manage, support, monitor, and scale them. Done."

jsprinkles16• 14 years ago

So if someone throws multiple tens of gigabits at your customer, and your upstream threatens to turn off your entire hosting company, you would respond "no way, we're going the extra mile for our customer"?

Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. Your idealism will fall down quickly when one annoying customer threatens service for every last one of your clients: it becomes a "do I continue to get paid or do I fight for this customer" equation.

Price out mitigation equipment and the multiple high-level engineers you will need to administer it before you respond telling me how wrong I am, by the way. The gear alone is an engineer's salary just to get started.

Edit: Fine, I dropped the Amazon example.

jayzee• 14 years ago

Amazon did not drop Wikileaks fearing DDOS attacks. They dropped Wikileaks fearing Lieberman.

saurik4• 14 years ago

I don't remember it being about either DDoS (if anything, I thought they went to Amazon to avoid the DDoS, seemingly somewhat successfully) or "ToS" (which to me has an implication that Amazon decided they didn't like the service, as opposed to caving under the pressure of other people not liking the service).

http://arstechnica.com/security/news/2010/12/wikileaks-kicke...

> The Wikileaks website migrated to Amazon's cloud hosting service yesterday after being hit by a distributed denial of service (DDoS) attack. Amazon decided to discontinue serving the controversial website this morning in response to pressure from critics, including prominent members of Congress. ... Senator Joe Lieberman (I-CT), chairman of the Homeland Security and Governmental Affairs Committee, was among the congressmen who pressured Amazon to stop hosting Wikileaks. He told AFP this morning that he plans to question Amazon about its relationship with Wikileaks.

postfuturist2• 14 years ago

Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly communicate that to customers.

I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.

Granted, not everyone knows how to do that in a pinch, but it shouldn't be a big deal for a hosting company. You only have to figure out how to do it once and when you detect a DDoS attack, flip the switch for that customer.

kijin2• 14 years ago

> paying customers that do not attract DoS attacks

A little off topic, but I've always felt a slightly uneasy about the concept of "attracting" DDoS attacks. Sure, if you knowingly piss off a bunch of script kiddies, you're attracting attacks. But it seems that nowadays, any site that hosts user-generated content is at risk of being attacked for any random reason. And yet, a lot of people talk about "customers who attract attacks" as if those customers are to blame. It almost sounds like blaming women who wear certain types of clothes for attracting sex crime.

Of course, the fact that you didn't do anything to provoke an attack might be irrelevant when your upstream faces a choice between cutting you loose and eating hundreds of thousands of dollars. People need to do what they need to do to protect their networks. Nonetheless, I'm curious what you guys think about the concept of "attracting attacks". If you blame the victim even a little bit, does that affect your judgment about what should be done in the case of an attack?

pestaa2• 14 years ago

I can not understand these attacks. Why block a service that is free of charge, useful and did no harm? Unless of course this DDoS was not targeted, which makes even less sense to me.

Also, why did Rails Machine throw out the site so quickly? If I choose to sponsor someone out of my free will, I'd do so without distinction from paying customers.

0x02• 14 years ago

Just guessing, but maybe someone didn't like the content of a stored public "pastie"?

pestaa2• 14 years ago

... and had too much time on their hands. Seriously, when something is offending me, I contact the author and webmaster first. Not take the whole cluster down.

gojomo• 14 years ago

In such a case, those who launch the DDoS may have no legal/legitimate grounds to take some disfavored content down. (Perhaps it is embarassing to their prophet, government, guru, criminal enterprise, or movement.) They may have asked for material to be taken down, and the hoster said, "no, it's legal content" – at which point the DDoS is launched as both primary censorship (disabling access to the whole site) and censorship-by-blackmail ("take it down or your whole site stays down").

On the other hand, it's also possible that the originators of the DDoS are angry about something the hoster did take down. Perhaps they'd put up material against the rules, Pastie took it down, and in a tantrum after-the-fact, they decided to retaliate with a "well if you won't host our stuff we'll block eveyone". (This is a bit more like the Anonymous DDoS against payment services they didn't like.)

Without a statement from those involved, it can be hard to determine DDoSer motivations, but those are some common patterns.

seanp2k23• 14 years ago

@everyone who doesn't work in hosting....

>"why did Rails Machine throw out the site so quickly?" If you run a datacenter, you pay for an uplink. That uplink has limited capacity. 4gbit, 10gbit...whatever. A big attack can saturate that link completely, so even with the biggest most expensive "mitigation device" on the market (some of this gear can get into the hundreds-of-thousands-of-dollars for /one/ device, mind you), if a DDoS is overloading your upstream bandwidth providers, you can either have your entire DC brought to a crawl, or null route the site.

With that said, how did CloudFlare keep lulzsec up? Anycast, lots of iron, lots of smart technicians, and probably tens to hundreds of thousands of dollars in bandwidth fees. TL;DR it was a publicity stunt that they very smartly played up.

DDoS is pretty misunderstood, and lots of clients think that there is some magical box that can take all the traffic. Again, if your link is saturated, a "mitigation" device can only filter the traffic; your upstream providers can and will take you offline if you don't fix it. Failing that, you get a massive overage bill and every other client at the facility is crawling. It's not really a good solution (mitigation devices /can/ help with smaller attacks, but for the big stuff, null routing is the best solution unless you have something like CloudFlare -- and even they will pull the plug if the attack gets too heavy, because it's simply not worth the expense to them to keep your site online.)

jsprinkles16• 14 years ago

People who haven't worked in hosting don't realize that not only is the gear $100,000+, the administrators that understand it are just as expensive. Annually. (Mitigation is a relatively rare skill for network administrators.)

Edit: Let's say you run Joe's Web Hosting. Joe's has three facilities, and you run redundant ten gigabit uplinks at each. Last I priced a device that could handle ten gigabit at line rate, it was ~$120,000, so figure:

   $120,000 x 2 uplinks x 3 facilities = $720,000

Just for the gear.

(I honestly don't remember if that figure was for the gigabit device or the ten gigabit device. I think the ten.)

saurik4• 14 years ago

As someone who does not work in hosting, I am somewhat surprised that "upstream bandwidth providers" don't have mediation strategies for DDoS attacks. It is my understanding that most of the asshats out there aren't sitting on Stuxnet: they have well under a hundred machines that they are able to flood traffic at you with (which is certainly more than enough). It seems like there are network-level mechanisms you can use to just block such an attack at the ingress connections. Is anyone willing to explain more about the issues here? (I, at least, would find it fascinating.)

adrianpike• 14 years ago

A question about null routing - would the upstream provider or the datacenter do that?

Wouldn't one have to be dropping the packets at the upstream level in order to keep the pipe clear? At larger scales, is it common for the datacenter to own/control a router upstream, or would that be something that they would have to get in touch with their upstream provider to make happen?

heliostatic2• 14 years ago

Perhaps an opportunity for Cloudflare to offer support?

dknecht• 14 years ago

We would love to help, and have offered.

heliostatic2• 14 years ago

Awesome; I hope Josh is able to get things back online. I'm curious how much support or assistance Rails Machine offered prior to pulling hosting.

jsprinkles16• 14 years ago

I'm looking at the "Compare Plans" page; is DDoS Mitigation part of another feature that I missed, or is it implicit in the service (i.e., a side effect of having a good CDN)? I can ask that an easier way: if my site takes a 10 Gbps hit, which plan would mitigate it? Enterprise? Do you bill for the transfer that you eat?

Genuinely curious and haven't really looked around the site much, and you guys hit my radar when you took on Lulzsec. Can't imagine the plethora of nasties directed at their gear.

knewter• 14 years ago

Clearly the gist.github engineers DDoSed them - it's all so clear!

datums• 14 years ago

This behavior of unplugging the destination of the DDoS is common with smaller hosts. They don't have the capital to spend on expensive mitigation devices. There are times when these attacks affect their entire network (bad design), so their quick and fast solution is to null route you at their cores.

sha903• 14 years ago

The issue isn't unplugging, I don't think anybody here thinks that it is unreasonable if they were to do this. The issue is that they kept the site unplugged for good because a single DDoS attack. My analysis based on the official response is that they used this incident as an excuse to drop the site because it was too much of a hassle to deal with the takedown notices.

codexon3• 14 years ago

It isn't an issue of expensive mitigation devices.

Even expensive and large hosts like Softlayer will null route you.

They just don't want to pay bandwidth or go through the hassle of asking their upstream providers to filter the attack for them.

rickmb2• 14 years ago

"Smaller hosts" like this are imho companies that sell a service they are fundamentally unable of actually maintaining. And they are usually not very transparent about it.

mattmanser• 14 years ago

I don't know about the scale of the DDoS attacks but have you tried cloudflare?

tristanoneil2• 14 years ago

If you're looking for more of a minimal snippet experience there's always http://marked.cc

tvh2k• 14 years ago

Unless you're pasting code. Check out my trivial example: http://www.marked.cc/4f9766f05e69190001000017

tristanoneil2• 14 years ago

It does support code snippets you just have to use fenced code blocks ala Github style.

``` x = [] ```

nicksergeant2• 14 years ago

FWIW, apparently Rails Machine's CEO thinks some of Pastie's users are not-so-great: https://twitter.com/bradleyktaylor/status/194937146153508864

avree• 14 years ago

Uh, he's not talking about all of Pastie's users, he's stating that Pastie has some bad users who use it maliciously and threaten the provider.

nicksergeant2• 14 years ago

Sorry, worded that wrong. Fixed.

wrecked3• 14 years ago

I'd like to apologize to those who have been negatively impacted by my decision to pull support for Pastie (especially Josh). To understand why I made the decision to pull our support after 9 hours of multiple DDOS attacks, I'd like to share some background and our ops philosophy.

It is important to understand that I put our existing customers that pay us to manage and scale their high growth revenue-generating web applications before all else. This is the core of our business and what they trust us to do. As we are seeing now, this means that I will protect them at the expense of making some non-customers and "risky" customers upset. Let me explain further...

Rails Machine at this time is 6 people. Through a lot of tools like Moonshine, experience, and process we manage 100+ web applications. Please note that I did not say "host". Hosting is only part of the package. We commit to do whatever it takes to keep our customers' applications available and growing their business.

Everyone in the organization is a developer on a varying scale of dev to ops including myself who started with Rails in 2005 and have been a professional dev for 20 years.

Although not as quiet as we would like, in general the workload of responding to outages, bugs, scaling problems, and traffic bursts are managed by the team. We've been doing this for 6 years focused specifically on Rails and have seen most problems with Rails applications in production. This makes us fairly efficient in identifying and resolving issues.

We've been hosting Pastie pretty much since the beginning and free of charge for several years. In the past two years, Pastie began to attract a lot of users intending to use it to do illegal things. This includes sharing stolen credit cards, stolen passwords, phishing schemes, copyrighted content, virus/trojan horses, hacker scripts, confidential corporate info, etc, etc.

Please know that the overwhelming majority of Pastie's user base are well meaning folks who kindly follow Josh's basic rule of "using Pastie for good". A tiny minority however attract a lot of attention through their public pastes that ruin the experience for everyone else.

Aside from the obvious problems, the public existence of this stuff makes a lot of people upset who in turn threaten us. This includes but not limited to criminals, giant corporations, angry individuals, trolls, and more importantly our data center/upstream provider. These upset people then send us nastygrams requiring us to take action or else. "Or else" includes suing us, arresting us, DDoSing us, and more importantly terminating our service.

To avoid bad things happening to us and all of our other customers, we have to take action immediately. Every now then other customers get a spam notice or a DMCA notice but in general it happens once and not a huge deal. Pastie on the other hand generates 100s of abuse complaints. Abuse complaints that we can not ignore and require us to investigate and follow up on. While we are doing this, we are not helping the 100+ other customers that generate zero abuse complaints and most likely never will.

Enter the DDos. Over the past 6 years, we've handled multiple DDoS attacks on different applications. Given that 95% of our customers are running revenue-generating business applications, we deal with a DDoS about once or maybe twice per year. They are really annoying and consume a lot of time and a lot of concurrent team members. Although we wish they never happened, as some have pointed out DDoS attacks are a fact of life. Many high profile sites with much larger teams and budgets have struggled for multiple days fighting waves of attacks. We accept this as part of our job.

The problem in this particular instance is that these DDoS attacks on Pastie were a continuation of the stream of operational disruptions already being generated by the site. After handling the first attack with four of us covering all of the angles around 10pm and with the help of Internap's network team, we halted the attack.

Within a few hours, it began again in the wee hours of the morning. At the same time, alerts for another customer who had entrusted us with their business came in. So a decision was made to halt the second attack as quickly as possible and focus on doing our job as we promised to the rest of our customers. We could have chosen to ride out multiple other attacks and engage in a lot of time consuming and expensive behavior to preserve a site that was already a source of ops disruption. Making that choice would have been inconsistent with our values and commitments.

This decision was purely ops motivated to protect our team members ability to serve our core customers.

Some of you are upset about this decision and I am sorry for that. I know 100+ customers that would approve. I put my customers and team first.

Please feel free to reach out to me directly (email or twitter) if you would like to discuss this further.

@bradleyktaylor, Founder, Rails Machine

sauteedbiscuits• 14 years ago

You should edit your post to remove the 6th paragraph. Not your place to be calling pastie a increasing malware/virii distribution site. In fact, Not only is it libelous, but its not even appropriate to share with us. There is a reason companies dont get to comment on every bad thing that gets pointed their way.

IMO, your response does more damage to you than trying to explain it away. Doesnt matter if the guy is not paying you or not. In fact, your privacy policy says you wont do this and you just did.

I am not sure whats worse when you get hosted with you:

Is it when they disconnect you for having a incoming attack, or the public post afterwards where they air your dirty laundry?

Wow, its even worse on twitter:

https://twitter.com/#!/bradleyktaylor/status/194937146153508...

wrecked3• 14 years ago

The pastes that we receive complaints about are publicly viewable and searchable by anyone. The increase in these kinds of pastes is also publicly observable.

If a YouTube employee said that some naughty YouTube users post copyrighted videos, it wouldn't be private information or libellous as such videos are publicly viewable.

I'll edit to clarify that the overwhelming number of Pastie users are "using it for good" as Josh politely requests on the site.

noahc• 14 years ago

Bradley,

Keep on keeping on. You made the right choice and in the long run dealing with the take down notices and legal wrangling would have been a full time job.

What most people probably wont understand is that no sane business is going to go to bat for a non-customer who is costing them signifigant time and money as well as putting their whole business at risk.

jsprinkles16• 14 years ago

You made the absolute right decision, and anybody that is saying otherwise hasn't been in your position. The good of the many over the good of the one; that's hosting.

The operational awareness in this comment makes me wish everybody would print it out as an example of a tough call to make in defense of your brand.

rickmb2• 14 years ago

Throwing the victim under the bus is standard procedure for third rate hosters. Doing it without notice is even worse.

Only if the brand is "we're el cheapo hosting cowboys" it's the right decision.

roy-t• 14 years ago

I think this was a hard decision and I really believe that you're trying to protect your business, but I also believe this is the wrong decision.

- Not being able to handle 100s abusive requests is an indication that you can't handle rapid growth. You're going to (want to) host other websites that can grow big and will catch a lot of wind. Although there is no direct revenue for you Pastie.Org was good practice for this it seems also this wasn't a sudden problem (you yourself indicate here that the problem was slowly growing) so this should be no factor in the decision to terminate the hosting so suddenly. Of course if it's more hassle than it's worth you and Pastie can come to an agreement (like stopping next month) but it now sounds like that decision should've been made months earlier or not yet at all.

- Although it's extremely hard to protect yourself from DDOS attacks you've now openly indicated that you're vulnerable and that you will drop things you are committed to do if there is some pressure in order to save the rest of your customers. This will make other customers nervous since they can now be easily threatened and might even go down for the fun of it (I personally don't think it's fun but some people apparently do)

- Even though pastie.org was not generating any revenue and was not a 'customer' you still had a commitment to them and tbh it looks pretty bad that you've one-sidedly decided in an instant to break that commitment, this is really costing you cred.

Of course what's done is done now, but I just wanted to voice my opinion in this debate. Say I'm a devil's advocate here because by the positive comments here mostly you guys seem to be doing mostly good!

lotides• 14 years ago

I know I'll get a lot of shit for this but good riddance. I hope it never comes back. I've felt this way since I found my stolen accounts posted on Pastie and they ignored requests to take it down.

jmah• 14 years ago

Once they're on pastie, it's already too late.

Pastie.org host pulls hosting after DDoS attack

Related Stories