EC2 Users Should be Cautious When Booting Ubuntu 10.04 AMIs

https://silverline.librato.com/blog/main/EC2_Users_Should_be_Cautious_When_Booting_Ubuntu_10_04_AMIs

by mike_heffner • 15 years ago

89 43 15 years ago

silverline.librato.com

mihasya6• 15 years ago

We ran into what we believe to be this very problem at SimpleGeo and spent a long time figuring out what the cause of the JVM locking up for tens of minutes at a time were; we believe this bug (or general class of bugs) to be responsible. Upgrading to 10.10 has caused the majority of the symptoms to disappear.

My personal hunch is that there are probably startups and developers that dumped Cassandra due to "stability issues," which were in reality symptoms of this bug. There's obviously no way of confirming or denying this, so I'll reiterate that it's just a hunch.

gary4gar• 15 years ago

Instead of 10.10, I would suggest switching to Debian Or CentOS.

Due its short support cycle, Maverick isn't suited for servers.

imbriaco• 15 years ago

The patch in the bug report seems solves the problem and allows you to stay on 10.04 LTS. It's a tiny patch and has been stable for us under production load.

mihasya6• 15 years ago

You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run. With the amount of work going into stabilizing the kernel when running in a virtualized environment, chaining yourself to a slow-moving distro will cause the exact opposite of stability when running on Xen. See: this blog post.

mmalone4• 15 years ago

Or even FreeBSD, which works on EC2 now. It uses a different technique for scheduling, so this issue will not affect it.

dialtone2• 15 years ago

The importance of the short release cycle has decreased a lot for us since we moved to EC2. It's really simple to re-deploy the entire infrastructure on newer and supported images. Using puppet and cloudformation or whatever it is that you prefer you can upgrade with very little trouble.

saurik• 15 years ago

I would be careful with Maverick (or Lenny; Karmic was unaffected): there is a serious issue with it dropping interrupts on EC2. If you are going yo use Ubuntu on EC2, you /really/ want to be using Natty.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/666211

dialtone2• 15 years ago

This is even more serious than the one in 10.04. While in 10.04 the bug would slow down the machine for a while in 10.10 with 2.6.35 kernel will simply hang the machine and requires a reboot (which can only be achieved by rebooting from the AWS console multiple times) of the instance to be "fixed".

I'm in the process of upgrading all of our instances to Natty from 10.04 or younger. It's actually weird that this issue didn't get any attention whatsoever.

mikeyk• 15 years ago

Seconded. We had our 10.04 machines regularly, and become totally unresponsive. Then, we had even worse luck with Maverick--starting with the issue where it didn't recognize half the RAM on the 68GB instance type. Natty seems to be holding up the best so far.

dreamdu5t2• 15 years ago

This applies to any AMI, Ubuntu or not.

Ubuntu shouldn't be used for web servers to begin with. You should be using a distribution with a long and thorough stable release cycle with minimal packages, such as Debian.

krobertson2• 15 years ago

Can you clarify? I wouldn't use non-LTS on servers, but the LTS ones have a stable and long enough shelf life.

I've personally found Ubuntu more bare bones out of the box than CentOS.

dreamdu5t2• 15 years ago

The main difference between Ubuntu-LTS and Debian is that Debian's stable release cycle is longer and more thorough. This means that Ubuntu-LTS will have slightly newer software, but Debian will likely have less bugs.

Debian's primary concern with its release cycle is stability. Other distributions like Ubuntu or CentOS trade a little stability for newer software.

My method: Use Debian but when you must have newer versions just add an unofficial repository to your apt sources (assuming you're prepared to deal with the complexity and inconsistency this might introduce).

derobert• 15 years ago

There are two (sometimes more) versions of Ubuntu: Server and Destkop. Desktop is definitely inappropriate for running production web servers (AMI or otherwise), for the reasons you indicate.

Server, however, is perfectly appropriate, especially the LTS release (which is supported for five years). Ubuntu LTS will probably actually be supported longer than (for example) Debian Stable.

aaronblohowiak• 15 years ago

librato has an amazing product and continues to have a great blog. I love how they dig into the root cause (linking to the typo in the patch that introduced the bug) and also have easy graphs to take in the impact.

RyanKearney3• 15 years ago

Honestly if you're using Ubuntu for a server then performance is not your priority anyway.

mihasya6• 15 years ago

What is your recommendation?

hiroprot• 15 years ago

FWIW, we're using the Amazon Linux AMI (from Amazon themselves) and are quite happy with it.

It is a pretty minimalist AMI, which is what we wanted. One caveat: it is in beta.

RyanKearney3• 15 years ago

CentOS if you can't use RHEL.

mey• 15 years ago

Is this strictly a Ubuntu/Xen bug, or does this have impacts on other VM environments (Virtual Box/VMWare)?

EC2 Users Should be Cautious When Booting Ubuntu 10.04 AMIs

Related Stories