When your IP traffic in AWS disappears into a black hole

http://engineering.clever.com/2014/12/10/when-your-ip-traffic-in-aws-disappears-into-a-black-hole/

by schimmy_changa • 12 years ago

185 52 12 years ago

engineering.clever.com

maerF0x0• 12 years ago

I wonder if this is a problem for any cloud provider, I also wonder if ipv6 could help mitigate? Then the IP collusions would be rarer.

schimmy_changa6• 12 years ago

I think you are definitely right that IPv6 would mitigate, but I think that the real way to avoid this issue is for the kernel to make the tradeoff for you - it's so cheap to do an arp ping, you might as well on maybe every 100th packet or so. Or similar: timing out and clearing the arp entries more frequently.

As for other cloud providers, I'd be really interested to hear from Digital Ocean, Rackspace, and Google!

pyvpx• 12 years ago

cheap for one server...definitely not cheap at AWS scale.

jsolson3• 12 years ago

Answered parallel to you in this thread: https://news.ycombinator.com/item?id=8732356

pquerna• 12 years ago

It's common on these provider "isolated" networks to use a small /24 subnet by default.

I'm more surprised that AWS isn't sending a gratuitous ARP when IPs are re-cycled to mitigate issues like this.

windexh8er• 12 years ago

That would mean that one of two things would need to occur:

1) Amazon sends GARP on behalf of the guest OS, from outside the kernel 2) Amazon has hooks into the guest OS to instruct the GARP to occur

Both are very bad things with regard to customer / Amazon separation of privileges and control.

I think every network engineer was probably yelling ARP at the screen early on in this write up and is likely one of the first things those seasoned in L2/L3 would look at first.

Also, as others have stated IPv6 removes this problem altogether. With the complete gutting of the concept of a broadcast and replaced by announcements within multicast groups which are much more efficient when you're talking about "small" prefixes that house 18,446,744,073,709,552,000 hosts (/64). ARP tables wouldn't be able to scale or work efficiently.

Overall this makes me think that it would be interesting to build a best practices guide around IPv4 and IPv6 networking within cloud provider environments. I think the gratuitous ARP on boot is a relatively safe practice, especially with regard to environments that are in continual flux.

$0.02.

justincormack3• 12 years ago

ipv6 doesnt actually use arp, so it is not going to be exactly the same anyway.

jsolson3• 12 years ago

This particular problem would not crop up on Google Compute Engine. See this cryptic snippet from /sbin/ifconfig on a production GCE VM:

eth0 Link encap:Ethernet HWaddr 42:01:0a:f0:40:dd inet addr:10.240.64.221 Bcast:10.240.64.221 Mask:255.255.255.255

This isn't an intentional feature, it's just a property of how our Andromeda SDN[0] is wired up. In particular, we lift the business end of figuring out where the other hosts on your LAN are up into the SDN rather than relying on the guest to cache ARP entries (hence the /32 netmask).

[0] http://googlecloudplatform.blogspot.com/2014/04/enter-androm...

jbeda• 12 years ago

Actually, this is intentional. Pre-andromeda, we hit ARP caching issues when testing scale for GCE. The model to let the SDN figure out the other end (and stub out things like ARP) was carried forward with Andromeda.

(I'm the original TL for GCE)

schimmy_changa6• 12 years ago

I think the biggest thing I was surprised by with this investigation was the lack of documentation about data-layer tools. At one point I was looking through the source of the 'ip' command to try to find out exactly which conditions caused a 'STALE' entry in the ARP table...

ChuckMcM• 12 years ago

Interesting trace down to stale ARP entries. It gets worse when the switches are running mac address filtering and they get out of date. We had that issue with some Blade G8052 top of rack switches with their upstream 10G ports. They sometimes "forget" which upstream port has the MAC address that they are switching too, and those packets just spew out messily into the data center leaving a mess. The "fix" it to force the switch to ping up through a specific upstream port periodically to the center switch's management IP address. Sigh.

p8952• 12 years ago

I've seen the exact same issue in some Dell PowerConnect switches.

We only saw the issue when using the "iSCSI Auto-Config" mode. Manually configuring the switches with the same config but entered by hand resolved the issue.

danesparza• 12 years ago

Am I weird because I actually muttered 'ARP caching issue' halfway through your article? :-)

Love the technical write-up -- thanks!

schimmy_changa6• 12 years ago

I don't think you're weird, I think you're probably a great candidate to work as a sysadmin / SRE, or at Clever, we're hiring: https://clever.com/about/jobs#engineer-distributed-systems !

justincormack3• 12 years ago

Your job descriptions list a lot of technologies but dont mention anything to do with operating systems or networking. It seems odd to explicitly call out Node and redis but not anything down the stack.

josegonzalez• 12 years ago

A ops friend of mine also saw this exact issue last week - same conditions as well - so this was also fresh in my mind.

He ended up disabling arp caching though, rather than clearing it.

daveloyall3• 12 years ago

So his machines issue a WHO HAS ...? request and wait for a response before every outgoing packet? ...

invertedohm• 12 years ago

The "it works after traceroute" line was the moment I was sure it was ARP caching. Then I was just wondering if it had anything to do with AWS specifically, which could have pointed towards a serious VPC bug. That doesn't seem to be the case though. You're not weird. Just burnt by real experience.

Nice write-up. Too bad I'm not in SF. :P

valarauca12• 12 years ago

I've reached the point that flushing the ARP cache is part of my network troubleshooting routine. Always gives you the weirdest hardest to trace errors.

js22• 12 years ago

Until that time when you flush the cache on Solaris[1] and remove the entries for its own interfaces. Then it stops responding to ARP requests for itself. Now you have an even more fun issue to troubleshoot!

1. something silly like: "arp -a | awk '{print $2}' | xargs -n 1 arp -d". It's been a while and I don't think I was the original culprit.

daveloyall3• 12 years ago

Let this be a lesson: teach your children about hubs before you teach them about switches! :) So then, that inevitable question will come up: "Well, how does it know WHICH port?"...

justincormack3• 12 years ago

Kids now, never seen a hub!

X-Istence2• 12 years ago

I guessed this as soon as he mentioned spinning up/down a lot of machines using scripting. I've seen the same thing happen within an OpenStack environment I used to maintain.

As part of the cloud-init scripts on new machines we would install arping and have the newly provisioned VM automatically send gratuitous arps so that arp caches would get updated.

I haven't seen the issue on my RHEL 7 VM's though, nor in my current environment, but we don't spin up/down as many VM's as we used to.

kelnos• 12 years ago

Don't feel bad; as soon as I read the bit about them using a small amount of IP space, and new instances getting assigned IPs that were recently used by terminated hosts, I thought the same thing.

mobiplayer• 12 years ago

Same here... I´ve been seing things like this everyday, for years (Network / NetSec Support Engineer here :))

Just today: Servers with a static IP address getting it changed to an APIPA just because some other 3rd party device sent a unicast ARP request to it that should've been a broadcast as per RFC4436. I personally find ARP to be one of the funniest protocols out there and I love the faces people make when they understand how ARP glues L2 and L3 and suddenly everything makes sense.

It is funny to see articles like this, because the network seems to be the "last frontier" for IT companies/workers. Only a handful seem to be brave enough to work on it (and actually like it). It is supposed to work like the lights when we switch them on and any interaction with a network engineer is just to tell them about an issue :)

In my opinion, the network guys are the geeks among the geeks. Don't get me wrong, it is easy to understand networking, it just takes dedication.

falcolas• 12 years ago

Try an arping from the new workers on first startup? Ran into this quite a bit when using VIPs for DB failover, and an arping fixed the caching issue in most cases.

wahnfrieden• 12 years ago

FYI, Clever: I click "Engineering Blog" at the top, and all links to blog posts on that page 404.

rgarcia• 12 years ago

sorry about that. It's fixed now!

spectre2562• 12 years ago

This reminds me of a time at a previous company years ago, where we experienced an issue that felt similar, although the root cause was quite different.

Basically, we had multiple teams all launching/terminating web servers. Unfortunately, they were all in the same EC2 deployment, and more often than not our load balancers from one team would send traffic to the web servers of another team. Furthermore, our setups were similar enough that this would sometimes cause bad results for users. We fixed it by making sure that our web servers on every team spoke on different ports. Not elegant, but effective (until two teams accidentally picked the same ports).

These days we have good enough infrastructure tools that this problem should never happen. But in 2009, at a company that was overwhelmed with growth, those sort of things happen.

vosper• 12 years ago

Would a better fix have been to have a load balancer per team, and then have the EC2 instance bind to the correct load balancer?

spectre2562• 12 years ago

That's what we had, but because the IPs were often recycled, traffic sometimes went to the wrong place if the load balancers has stale data.

perlgeek• 12 years ago

Wouldn't it be a better solution to not reuse IP addresses quickly? If I understood it correctly, they are in a private network anyway, so they could afford it.

daveloyall3• 12 years ago

I thought there was a mechanism that would allow some sort of failure notification packet to make it back to the origin, so that it would then perform a proper ARP lookup WHO HAS 1.2.3.4? TELL 4.3.2.1!.

Perhaps AWS is blocking some class of packet?

X-Istence2• 12 years ago

The issue is that if it is on-network the only real "failure" notification is a time-out. No ICMP packets, nothing are sent back to indicate you are sending to a non-existent IP address.

jcollins• 12 years ago

We saw this earlier this year after upgrading to a new Linux kernel.

The solution for us was to set this in sysctl.conf:

net.ipv4.neigh.default.gc_thresh1=0

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150... https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150...

kiyoto• 12 years ago

Looking at the port number, it looks like Clever is a MongoDB user =)

akx• 12 years ago

You, uh, don't even have to just recognize the port number, reading the article is enough...

[...] These workers all connect to a MongoDB replicaset to update and read data [...] 27017 = port mongod is listening on

girvo• 12 years ago

We had a fascinating bug on EC2 -- we could connect to the instance, but no network traffic made it out. It wasn't security group problems, it was literally a really weird bug in EC2's network that we somehow triggered, the engineer over at Amazon that looked at it was really excited when he came across our case as it was so weird, heh. They fixed it, I can't remember exactly what was done on their end, but it was one of the weirder problems I've attempted to debug. Nothing I tried worked!

zenocon• 12 years ago

I just experienced this early this week. Very frustrating. I also posted to AWS forums and got zero assistance; am currently not paying for AWS support plan. This article came at an opportune moment -- it makes sense and removes the shroud of mystery around why it "works sometimes" which leaves me with an uneasy feeling for a production setup.

schimmy_changa6• 12 years ago

awesome, I was hoping it could help out someone else :)

legohead• 12 years ago

I was going to respond with my little story, but I see your article already linked it! ;)

When your IP traffic in AWS disappears into a black hole

Related Stories