Ask HN: How to deal with SYN FLOOD and extortion

https://news.ycombinator.com/item?id=530973

by xal • 17 years ago

40 50 17 years ago

news.ycombinator.com

We made a big mistake in our server setup. We bought beefy as app and db servers but went for ultra reliable sun ultrasparks and openbsd for firewalls. This was fine until the DDOS started. After 3 days of many different attacks we now how much better firewall hardware installed thanks to our fantastic colocation which went the extra mile and helped us when we needed their help most but we are now surviving on total brute force.

Our firewalls are faster then the SYN Floods that hit us. This is an arms race that we cannot win in the long run. Yes we can buy more hardware but it's much easier to infect more machines with bots over time.

How do people protect themselves against extortion and malicious ddos attacks? What software / hardware protects the bigger sites on the net?

Brushfire2• 17 years ago

Move yourself into the cloud and let someone else deal with it. Seriously.

I'm a big fan of outsourcing anything you arent 'great' at. In this case, its running firewalls. Unless of course your app is something related to server maintenance or firewalls or security, etc.

Most software companies really shouldnt try to deal with this stuff -- I had to learn the hard way. You should only spend your time on your applications, not anything else that doesnt directly benefit you. You can get lost in the maintenance.

Tichy• 17 years ago

Just curious, but how would "the cloud" help with such a problem? Sure, supposedly you could summon infinite computing power to handle the DDOS requests. But the computing power would not come for free, so the DDOS would still do serious financial damage.

Brushfire2• 17 years ago

You are paying either way. Thats just the way it works.

However, under one option, you pay and Amazon deals with the actual load, downtime, server stress, hardware improvements, etc.

Under the other, you get to deal with all of that AND get to be pissed off AND accomplish nothing for your business at the same time.

irrelative• 17 years ago

There are several companies that create DDoS mitigation hardware/software. I have good knowledge that Amazon (et al) use several of these companies for solutions.

tptacek9• 17 years ago

Any large ISP should have a process and gear in house to mitigate DDOS attacks; you should escalate, first to your hosting provider and then to their upstream provider.

Every ISP does something a little bit different for large-scale attacks. Some of them off-ramp traffic to scrubbers, some of them have inline devices that can characterize and block SYN floods. My previous employer, Arbor Networks, now has gear deployed at something like 90% of the worldwide tier 1 and tier 2 ISPs that will detect any major flooding event and generate a report which might be helpful to you.

What I'd watch out for is the dozen odd fly-by-night operations that are promising you that, for a monthly fee or a one-time purchase of some $100k box, you can block these attacks on your own network.

lsc13• 17 years ago

I'd extend that warning... watch out for any company, even if it's large and reputable and the salesmen have nice suits, that promises to solve your DoS problem by selling you an expensive firewall. If your traffic is below 100Mbps, there's nothing the fancy box can do that your OpenBSD box can't. (this changes as your traffic climbs significantly above 100Mbps, but most of us little guys are below that line.)

_b8r0• 17 years ago

I'd go a bit further and say that even over 100Mbps you're not going to get much protection over an OpenBSD box or a Sun system running a decent Firewall.

The key thing is to try to block it upstream. Your firewall (and indeed anyone's firewall) is only really designed to restrict IP-layer access to specific source and destination hosts, ports and datagram types. Unless it was built by Cisco, in which case it was probably designed to do a whole load of other stuff, none of which will help you right now.

There are limited things you can do to the IP stacks of most OSes these days that can help discard unsolicited packets, and reduce resource consumption on the local boxes (if that's high) but an upstream solution is likely to be more optimal in most cases.

lsc13• 17 years ago

your provider (or your network guy, possibly you) needs to blackhole the source of these attacks (preferably notifying the ISPs in question.) most of the better co-location providers will give you a way to blackhole traffic from certain IPs. Really, you should be able to do that from your OpenBSD firewalls without help from anyone else, assuming the problem is too many syn packets rather than too much traffic. dropping all packets from a particular source IP is not expensive, even when you multiply that by a large number of source IPs.

asmithmd1• 17 years ago

Black listing the source IP's is NOT going to help. The attacker can just spoof any IP, he doesn't care about actually connecting he just wants to tie-up your server

http://en.wikipedia.org/wiki/SYN_flood

axod2• 17 years ago

As far as I'm aware (Correct me if I'm wrong anyone in the know), the process for any spoofed source address flood would be:

  * Determine which upstream provider it's coming from
  * Contact that provider, and ask them to investigate
  * They do the same thing
  * If a provider finds out the source is local, then they block it / contact the person(s) responsible etc

Of course it gets harder if it's a botnet or some other system that is distributed over a large number of sources.

lsc13• 17 years ago

depends on how smart your attacker is. the source IPs must not respond or the attack won't work, so potentially you could block enough bogons. Also the first syn flood attacks came from a fixed source IP.

You have OpenBSD firewalls... have you checked out OpenBSD SynProxy? I haven't used it, but it was designed for just your case.

http://www.openbsd.org/faq/pf/filter.html#synproxy

daveungerer5• 17 years ago

Makes you wonder why ISPs are not discarding packets with spoofed addresses, instead of letting them escape into the greater internet. Is it too expensive to do? My knowledge is a bit rusty, but isn't this a common router function?

tptacek9• 17 years ago

That's not strictly true. On the very high end, in products sold pretty much only to ISPs, you can get source address filtering for hundreds of thousands of sources for established connections, and you can get SYN proxying to have the head-end complete the 3WH before your downstream connection ever sees those SYNs.

mpk2• 17 years ago

This works fine when there are tens or even a few hundred source IPs. However, malicious DDoS attacks are often orchestrated using zombie machines (maybe your parent's unpatched Windows box, for example). This way the perpetrators remain out of reach and you end up swamped with thousands upon thousands of source IPs, none (or few) of which share a net block.

This is really not a problem you can solve down at the firewall-before-webserver level.

lsc13• 17 years ago

the response to my post was saying I was wrong (and is right in saying I was wrong) because the attacker is spoofing the source address. This makes it a harder problem to solve than zombied PCs (the case of zombied PCs sending unwanted traffic can be blocked by a firewall or other means, you just need an automated method of identifying the boxes you want to block.)

lsc13• 17 years ago

axod2• 17 years ago

I've had a few DDOS attacks in the past, first when hosting with slicehost, and then when at linode. Running IRC related stuff, these attacks are pretty much expected from time to time.

Both dealt with them fantastically, keeping me updated on progress etc.

As far as I remember, there was only one instance when they decided (slicehost) to take my server 'offline' and just wait for it to pass/upstream providers to deal etc.

The other times they were quickly able to escalate to upstream providers with minimal impact. I've had a few at linode, and whilst it does use up some bandwidth allowance, it hasn't been a big deal at all.

Obviously if you're able to, get more IPs, use a few data centers, etc. distribute yourself so that an attack on any one part doesn't have a big impact.

patrickg-zill• 17 years ago

If they are never connecting, then why not set up a SYN "honeypot"? As soon as a valid user connects, redirect them to a different site or IP address that you do not advertise, that actually hosts your app and db servers.

Thus you could rent or colo a small system on a different network, maybe even in a different part of the country and then redirect valid users to an entirely different location with a cookie in the URL. No cookie no service.

moe• 17 years ago

The attacker would simply move the syn-flood to where the candy is.

mpk2• 17 years ago

The big boys seem to be a little reluctant to publish information on their DDoS strategies. (If anyone has information that says otherwise, it's much appreciated).

I spent some time googling around and in case anyone's interested, these links seem to cover most of the mitigation patterns.

A slideshow covering common mitigation techniques, http://www.slideshare.net/intruguard/10-ddos-mitigation-tech...

A Cisco whitepaper (which, as usual, has en emphasis on Cisco kit and a long URL), http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps...

A fairly comprehensive PDF document - A Survey of Active and Passive Defence Mechanisms against DDoS Attacks

http://www.fbi.cqu.edu.au/FCWViewer/getFile.do?id=17921

lacker• 17 years ago

The big boys seem to be a little reluctant to publish information on their DDoS strategies.

Makes sense, the bad guys are just as likely (if not more) to read up about them.

tptacek9• 17 years ago

I don't know how secret this stuff is; I think if you poke around NANOG, you'll find a lot of material. If there's something that makes DDoS mitigation mysterious, it's how ad hoc all this stuff is.

What I've had firsthand experience with is:

* Tier-1's profile all their traffic, either directly or with flow export, and will get alerts if they see spikes to certain netblocks, or spikes to specific /32s with specific characteristics.

* Inside a Tier-1, the third-tier support people usually have additional monitoring they can enable for customers if an incident has been escalated to them. They get hundreds of these calls a week.

* A lot of Tier-1's can quickly reroute traffic to a specific /32 to run through special-purpose filtering setups, which might be a DDoS box like a Cisco/Riverhead, or an IPS like TippingPoint, or even just a Cat with lots of TCAM space set aside for filtering.

* A lot of Tier-1's --- maybe all at this point --- have some mechanism set up to share signatures of attacks so they can push filtering further upstream. When I watched that stuff happening, the sense I got of it was that this was really reserved for things like global botnet C&C.

What I'd add to the discussion on products is, most of what's built to combat DDoS is really only useful if you sell transit. If you're a Fortune 500, I know there are ISPs where you can get Cisco or Arbor gear deployed on the head end specially for you. But that's a Fortune 500, not 37 Signals.

datums• 17 years ago

If they have already attempted to extort $$. You should keep the mitigation device in place. There are a few devices that do packet inspection and can handle multi gigabyte size attacks.But like many have mentioned contact the upstream provider

Ask HN: How to deal with SYN FLOOD and extortion

Related Stories