Auto Recovery for Amazon EC2

https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+AmazonWebServicesBlog+%28Amazon+Web+Services+Blog%29

by tshtf • 12 years ago

167 49 12 years ago

aws.amazon.com

fletchowns• 12 years ago

An important note if you want to use this right away:

This feature is currently available for the C3, C4, M3, R3, and T2 instance types running in the US East (Northern Virginia) region; we plan to make it available in other regions as quickly as possible. The instances must be running within a VPC, must use EBS-backed storage, but cannot be Dedicated Instances.

waskosky• 12 years ago

For people not yet on one of the newer instance types (HVM VPC), it is possible to migrate directly assuming you can tolerate a few minutes of downtime to make the final switch. This was the guide that helped me: https://forums.aws.amazon.com/thread.jspa?threadID=155526

wahnfrieden2• 12 years ago

VPC-only sounds like a giant caveat, and it is, but this is a good opportunity to note that this is the trend now with AWS and the direction they're heading - (non-VPC) "EC2 Classic" is being gradually phased out, VPC is now the default for new accounts, and most new features are being added only to VPC. So, time for everyone to start thinking about migrating.

mbell2• 12 years ago

this is the trend now with AWS and the direction they're heading - (non-VPC) "EC2 Classic" is being gradually phased out

New AWS accounts can't even use EC2 Classic, it's effectively deprecated at this point.

pjl• 12 years ago

AWS recently announced ClassicLink [1], which helps with migration: "In order to allow EC2-Classic instances to communicate with these resources, we are introducing a new feature known as ClassicLink. You can now enable this feature for any or all of your VPCs and then put your existing Classic instances in to VPC security groups."

[1] https://aws.amazon.com/blogs/aws/classiclink-private-communi...

saryant3• 12 years ago

I've been having a lot of issues with r3.large instances becoming unreachable lately. Hoping this can serve as a stopgap.

jeffbarr2• 12 years ago

Have you noted your problem in the EC2 Forum or consulted AWS Support?

saryant3• 12 years ago

edit: I was wrong. Ignore what was here before, AWS did respond and I just didn't notice. Apologies to AWS for speaking out of turn.

bmurphy1976• 12 years ago

Please note that this is for EBS backed instances only.

If you want something similar for ephemeral instances, do what we do: min 1 max 1 auto scaling groups. We've found that Amazon is pretty good at catching bad instances and terminating them, although on occasion we do have to terminate an instance manually. The autoscaling group takes care of the rest.

biot2• 12 years ago

Any reason why this isn't automatic? From the "Recover your instance" docs:

  Examples of problems that cause system status checks to
  fail include:

   * Loss of network connectivity
   * Loss of system power
   * Software issues on the physical host
   * Hardware issues on the physical host

All of these are on the physical host, which end users cannot control. So if AWS has an issue that kills your VM, if you don't have this setup then your instance is effectively dead?

perlgeek• 12 years ago

Loss of network connectivity sounds like it could be temporary. If you have long-running calculations and want to wait for the result, it might make sense to wait a bit longer and see if the network comes up eventually.

And there's no indication that the hardware and software "issues" are permanent or even fatal.

biot2• 12 years ago

The strategy of "wait a bit longer" can be configured with an exact value for how long to wait via this recovery feature. However, given that you don't have any information about the failure's permanence or whether it can be resolved by AWS staff without taking the host down I don't see why waiting indefinitely is a particularly good option.

In some ways, I guess this answers my own question. Amazon doesn't know how long you might want to wait or if you have a VM that you would even want to have recovered, so configuring this lets you tell Amazon what your parameters around recovery should be.

moe2• 12 years ago

Any reason why this isn't automatic?

Perhaps it's a co-promotion for CloudWatch. I would guess quite a few of their users had never heard of or seen a use for CloudWatch. Some of them might now enable "detailed monitoring" for $3.50/mo per instance while they're at it.

larrymcp• 12 years ago

You know, it's funny but reading the EC2 forums you see several occasions where Amazon does reboot instances automatically after a failure.

I recall several posts where a customer asked a question along the lines of "Why did my instance reboot" and then someone from Amazon replied something to the effect of "Sorry there was an issue with the underlying hardware but we did restart your server on new hardware".

CarlHoerberg• 12 years ago

Finally, it's crazy that they've haven't implemented this earlier, and why isn't it enabled by default, like on GCE? We've had for a long time an app that just polls the ec2 api and looks for impaired instances and then automatically restarts them. We have about 2-10 impaired/scheduled-for-reboot/on-deprecated hardware-instance per month so that app is quite a time-saver.

alrs2• 12 years ago

The ugly caveat isn't VPC, it's EBS.

This lands on the wrong side of pets-versus-cattle. AWS has been moving towards giving people what they want, but it's still best practice to use ephemeral storage and architect accordingly.

dice• 12 years ago

If you've drunk the pets-vs-cattle koolaid to the point where your instances are all (or mostly) ephermal then you're probably also using auto-scaling groups and don't care if a particular single instance falls over.

peteretep• 12 years ago

    > If you've drunk the pets-vs-cattle koolaid

Generally drinking the kool-aid is a bad thing - any reason to favour pets?

toomuchtodo• 12 years ago

but it's still best practice to use ephemeral storage and architect accordingly

Its not worth the engineer time. Use EBS volumes, clean them up when they're no longer in use after termination. The only time you need local/ephemeral storage is swap or scratch space, or throughput you can't get from general or provisioned EBS.

Plus, you get auto recovery now without having to have architected for it ;)

gtaylor• 12 years ago

After I say this, they'll probably have a huge EBS-caused outage, but it feels like EBS is much more capable now than it was even a year or two ago.

The General Purpose SSDs and the Provisioned IOPS have more than handled our performance concerns. Since that last awful multi-AZ outage a year or two ago, there hasn't been much to deal with for us.

gst• 12 years ago

EBS is great if it works for requirements, but unfortunately the maximum size of an EBS volume is limited to only 1 TB. With SSD instance storage you can get up to 6.4 TB.

Rapzid2• 12 years ago

Here is a fun story. Amazon's RDS services are backed by EBS. They also restart the instances automatically if they encounter issues.

But sure, "Best PracticesTM".

alrs2• 12 years ago

You're right, RDS is backed by EBS.

oellegaard3• 12 years ago

Heavy EC2 user here. This doesn't solve your problems, if you want to do this right, setup an EC2 Auto Scaling group and build an image each time you need to change your server. That is the proven way most large deployments work, including Netflix.

oellegaard3• 12 years ago

I have a semi-old blog post about how we do it here: http://blog.kristian.io/aws/2014/06/28/how-we-moved-from-her...

jontro• 12 years ago

Is there any guide on how to integrate this in a deployment workflow? I would be very much interested in reading up on how to do this the best way.

lebski88• 12 years ago

We use this approach at MixRadio, you can read about it on our blog: http://dev.mixrad.io/blog/2014/10/31/How-we-deploy-at-MixRad...

There is also a video of a talk about it: https://skillsmatter.com/skillscasts/6057-herding-cattle-wit...

We based our approach on Netflix but ended up building our own tools which we've now open sourced.

oellegaard3• 12 years ago

If you're interested, shoot me an email and I will be happy to go through our setup. You can find my contact info at kristian.io

discodave• 12 years ago

Yes but most people don't have large deployments like Netflix. The main reason Neftlix and co build entire images is because if you're scaling up and down by thousands of instances then the extra time for those instances to do all the build, configuration and so on is material. So you're better to do it once and essentially CTRL+C CTRL+V.

If you're only scaling up a few instances at a time then your number one consideration is probably making life easy for yourself.

bkirkbri• 12 years ago

We tend to take an in-between approach, baking base AMIs as needed but configuring and installing code with Ansible at boot time. We've got scripts to cut tarballs with an Ansible playbook plus support files from any tag in our repo. Then it's just a matter of specifying the tag in user-data when starting an instance. We do that through CloudFormation.

edit: grammar

bkeroack2• 12 years ago

At the risk of being down voted, let me say that this is yet another AWS "feature" that is primarily a workaround for deficiencies in the platform.

waitwaitwhay• 12 years ago

Can you explain that for me? I use Microsoft Azure a lot and there things like this just happen automatically. No configurations, alarms, etc.

regularfry2• 12 years ago

If by "the platform" you mean "available computing hardware", then yes. I'm not sure that's a useful data point.

bkeroack2• 12 years ago

In the narrowest sense your sentence is correct. Perhaps you mean that you think this can be expected on any computing hardware, which is far from correct. If all you've ever used has been public cloud services you can be forgiven for having this misconception.

tedunangst• 12 years ago

Wouldn't transparent migration to new hardware be even better? Isn't one of the advantages of virtualization the ability to move a running image from one machine to another?

chacham15• 12 years ago

In many cases that is not possible. If the machine loses network connectivity, has a disk error, is infinite looping, is out of memory, etc. how is another machine supposed to access its data?

kolev2• 12 years ago

If you rely on something like this, you rely on nothing. This is like crutches for your broken architecture. For singleton roles, you could do an autoscaling group of one and do better.

kolev2• 12 years ago

Not sure why the downvotes - at least ASGs can be expanded later unlike the one-offs. Plus, you should never have SPOFs anyway. There was a comment below about not all projects being of Netflix' scale - well, there's Digital Ocean for the smaller one - EC2 is not the most cost-effective solution for small projects anyway.

halayli• 12 years ago

This makes me so happy.

j-kidd• 12 years ago

This shall be a great fit for the NAT/Bastion instance, since the high-availability setup has a few drawbacks: https://aws.amazon.com/articles/2781451301784570

andr• 12 years ago

I think CodeDeploy is quite an undervalued AWS tool. It's a combination of Puppet for server config and Heroku-style deploys. Together with AutoScaling it makes it trivial to set up any number of identical servers, without relying on custom AMIs or recovery.

Auto Recovery for Amazon EC2

Related Stories