Hurricane Sandy and AWS: Migrating Octopart Out of us-east-1 in a Hurry

http://octopart.com/blog/archives/2012/11/hurricane-sandy-and-amazon-web-services%253A-migrating-out-of-us%252Deast%252D1-in-a-hurry

by sam • 14 years ago

53 22 14 years ago

octopart.com

ChuckMcM• 14 years ago

Ok, if you're looking for a startup idea, here you go a free one on the house :-)

"EjectorSeat" - EjectorSeat is the hip cool way to create a current, usable, set of scripts which can move any AWS/EC2/Linode/Heroku/... (start with AWS/EC2) instance from where it is, to somewhere else. By using EjectorSeat all of the changes and specs you have on your install are automatically documented into its configuration and data location databases so that when the time comes to pull the yellow and black handles, you will know that your install is being migrated, and better yet, when its done migrating one push of a button and blam! its up and running.

-----------

It seems like this should be possible to package up (it will take a bit of work of course) but its the 'copper' option (of the silver/bronze/gold nomenclature) for disaster preparedness. Basically you don't have the funds to maintain two instances all the time, this lets you move the one you have when you need to.

Of course like real ejector seats it will have failure conditions (like flying inverted at 50', bad time to eject) but it could provide an ops guy with a bit of piece of mind.

lr• 14 years ago

Most of the hard work is already done for this: Puppet, Chef, etc.

jeffbarr• 14 years ago

CloudFormation should also be helpful here.

eli• 14 years ago

I wonder what percentage of AWS clients are not using Puppet, Chef or similar. I bet more than half.

If I were going to build this, I'd take a look at Blueprint, since it (at least in theory) can reverse engineer what's installed on a running server.

pwelch• 14 years ago

Agreed. We created a backup database in a west region and it only took a few minutes to provision with Chef.

I would be curious if there was an open source tool that copies your current security groups and writes them to other regions.

neurotech1• 14 years ago

LOL on Ejection Seat reference. Can we add NATOPS procedures for emergency APU start in flight? That would maintain power to systems in case of double flame-out.

I had a similar idea for a while. The memories of trying to get EC2 nodes manually configured, and it isn't easy at first. Also some painful memories migrating clustered servers (Linux running H-Sphere Control Panel) which didn't exactly fail gracefully.

Even with puppet/chef and other tools, migration takes time and planning. EC2 node architecture is not persistent, and should factor in failure of a node at any time.

dotBen• 14 years ago

Linode makes it easy to move instances around to different regions - at the hypervisor level it's just a command to move a Xen instance within their network of DCs.

Moving between vendors is always going to require more testing/configuring/etc then just a script to move the assets over.

sehugg• 14 years ago

RDS/EBS snapshots being bound to their region is a big stumbling block. Especially for RDS, because the only solution is mysqldump (or similar) and that not only blocks the production database but takes hours for databases above 10 GB. So at the end of this you have a very out-of-date backup that you can transfer to the other region.

Cross-region snapshot mirroring would be a great enhancement feature for AWS.

blantonl• 14 years ago

You might just consider setting up a dedicated slave MySQL database on a smaller instance in the west region just for the eventuality that you need to spin up a master there. Then you can just snapshot, copy to ephemeral, and go from there.

rhizome• 14 years ago

Given that the alternative is being down for days, turning a site read-only while the dump-move is happening seems like a reasonable compromise.

davidjgraph3• 14 years ago

"EBS snapshots cannot be moved between regions"

Not true. I've done this using Ylastic. See http://ylastic.com/features.html:

"Migrate EBS linux snapshots between regions."

"Migrate EBS windows snapshots between regions."

Takes a few minutes to kick off and does it all for you.

sam2• 14 years ago

Interesting. I wonder if they use an undocumented AWS feature or are snapshotting the volumes and copying them. My guess is the latter.

davidjgraph3• 14 years ago

The OA comment of "EBS snapshots cannot be moved between regions" should have "easily" inserted in there. I've seen quite a few full length processes for doing it, e.g:

http://elastic-security.com/2011/02/10/how-to-copy-an-ebs-ba...

http://serverfault.com/questions/336321/how-do-i-migrate-ama...

I'm 95% sure they are just putting a UI over that lengthy process. But as I say, I've tried it and it worked perfectly for me. This is their process:

http://blog.ylastic.com/migrating-snapshots-between-ec2-regi...

nmcfarl3• 14 years ago

As you’ve used it - how long does it take? I read up, and I decided they where probably just making a new EBS volume in the new region, `netcat`ing the data across, and taking a snapshot of that new volume, with the `netcat`ed data.

So I did it by hand - that way. It it’d be pretty easy to tell if this was the case, because it really wouldn’t be fast….

davidjgraph3• 14 years ago

It's not fast, they send you an email when it's done. It was a while ago now, but I recall it taking about 30 minutes before I got the email.

hubmo2• 14 years ago

I know this is a little late for your case but you can migrate snapshots from 1 region to another. We spend sunday moving from US-East 1 to US-West 1 and took 2 hours to setup our entire infrastructure consisting of 20 servers & about 180 GB database

There are several options

1. Manually - http://alestic.com/2010/10/ec2-ami-copy

2. Automatically via Scripts a) migrate-ebs-image.pl - http://search.cpan.org/~lds/VM-EC2/bin/migrate-ebs-image.pl b) CloudyScript Migrate SnapShot - https://cloudyscripts.com/tool/show/4 c) CloudyScript Migrate AMI - https://cloudyscripts.com/tool/show/5

3. Commercial Services a) http://ylastic.com/ b) RightScale

* ylastic is really cheap at $25 / month * I used the perl script migrate-ebs-image.pl, super simple to install and use. * CloudyScripts has open sourced their Ruby gem so you can build on top of it. They also have a free web form you can just use (but may not be secure enough). But you can launch their AMI in your own instance which should be secure

Hope that helps for next time!

sam2• 14 years ago

Thanks, these will be useful for next time. Hopefully AWS will add the capability to migrate the EBS snapshots to the console soon.

hubmo2• 14 years ago

would be nice for sure. luckily AWS builds the API first then builds it into the console, so many you can still get the job done just takes a little digging. feel free to reach out if you ever run into another situation like this

Hurricane Sandy and AWS: Migrating Octopart Out of us-east-1 in a Hurry

Related Stories