AWS issues unavoidable reboot schedules with short notice on many EC2 instances

https://forums.aws.amazon.com/thread.jspa?threadID=161544&tstart=0

by eknkc • 12 years ago

269 152 12 years ago

forums.aws.amazon.com

robszumski• 12 years ago

I'm assuming this is due to fallout from the recent bash security issue?

vetrom• 12 years ago

Swapping bash should not require reboots. The pending XSA-108 (http://xenbits.xen.org/xsa/) however....

jagger27• 12 years ago

How often do Xen vulnerabilities get embargoed?

adrr• 12 years ago

It has to be that. Can't think of another reason why you need to mass reboot the servers the instances are running on and serious enough they are patching through the weekend. Wonder if its being exploited in the wild. Also none of our C3.2Xlarges are getting rebooted, those use SR-IOV which can virtualize PCIE devices at the hardware level so it probably a network boundary exploit allow you to pull other instance traffic.

viraptor• 12 years ago

That's unlikely. You can update bash without any restarts. This sounds like an architecture change / global hardware (firmware?) update / kernel upgrade which doesn't allow nice migration to/from the hosts.

takeda• 12 years ago

There is Xen exploit that will be disclosed on October 1st. Amazon most likely already know what it is.

Corrado2• 12 years ago

They say that its fine to reboot but that there is no guarantee that you will land on an updated host. However, AWS does provide a script to run on Windows machines which should tell you if that particular machine has the issue. I took a quick look at the script and deduced that it is indeed a Xen issue.

akerl_4• 12 years ago

It seems like providing a script that identifies if your system is vulnerable to an embargoed XSA would be a violation of the predisclosure list, since it would basically be pointing at what the issue was?

wmf• 12 years ago

How does one bit of information (vulnerable / not vulnerable) tell you what the vulnerability is?

nraynaud3• 12 years ago

maybe just checking a version number or a compilation date? Indirect check?

rands33115• 12 years ago

This is not related to the scheduled reboot maintenance. The script that you are referring to has to do with a separate Windows networking driver and Xen.

Corrado2• 12 years ago

Indeed, it appears as if you are correct [1]. This issue is with Windows Server 2012 R2 network drivers as is from Sep. 17, 2014. If you're interested in the script you can download it from that page.

[1] http://aws.amazon.com/windows/2012r2-network-drivers/

devicenull• 12 years ago

Post the script?

It seems they could easily hit the metadata service to determine if a machine is patched or not.

nraynaud3• 12 years ago

I love how the amazon support person is absolutely not answering the very precise question.

vinceguidry• 12 years ago

Heh, I thought the answer was very clear: no.

nraynaud3• 12 years ago

I didn't read "no" in the answer, just circonvolutions around: "it's no but we're mangling it, plus we're not changing our course after your feedback". A clear answer would be: "fuck you".

quink• 12 years ago

EC2 screenshot: http://i.imgur.com/OdCehey.png

And copy of the email that was sent out to us...

----

Dear Amazon EC2 Customer,

One or more of your Amazon EC2 instances are scheduled to be rebooted for required host maintenance. The maintenance will occur sometime during the window provided for each instance. Each instance will experience a clean reboot and will be unavailable while the updates are applied to the underlying host. This generally takes no more than a few minutes to complete.

Each instance will return to normal operation after the reboot, and all instance configuration and data will be retained. If you have startup procedures that aren’t automated during your instance boot process, please remember that you will need to log in and run them. We will need to do this maintenance update in the window provided. You will not be able to stop/start or re-launch instances in order to avoid this maintenance update.

If you are using Windows Server 2012 R2, please follow the instructions found here: http://aws.amazon.com/windows/2012r2-network-drivers/ to ensure that your instance continues to have network connectivity after reboot. This requires that you run a remediation script in order to ensure continued access to your instance.

Additional information about Amazon EC2 maintenance can be found at: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/mo...

If you have any questions or concerns, you can contact the AWS Support Team on the community forums and via AWS Premium Support at: http://aws.amazon.com/support.

Sincerely,

Amazon Web Services

To view your instances that are scheduled for reboot, please visit the 'Events' page on the EC2 console: https://console.aws.amazon.com/ec2

This message was produced and distributed by Amazon Web Services LLC, 410 Terry Avenue North, Seattle, Washington 98109-5210.

ecaron2• 12 years ago

So far I'm 3 for 3 on stopped/started machines that come back no longer having the scheduled reboot event - maybe the problem isn't as annoying as we're thinking?

(Edited because I said "reboot" and meant "stop/start")

thezilch• 12 years ago

The post claims that it could take several hours before their scans identify hosts that were claimed during restarts.

rands33115• 12 years ago

A reboot won't address it at all (it also keeps you on the same host).. no point in doing that. From the e-mail notification: "You will not be able to stop/start or re-launch instances in order to avoid this maintenance update."

ecaron2• 12 years ago

Sorry, I should have phrased better. I issued a "stop and start". Per the AWS thread:

While executing a stop/start is absolutely fine to do, there is not a guarantee that you will land on an updated host. We are periodically polling for new instances and those impacted will receive new maintenance notifications accordingly.

Which I take to mean there's a chance you could land on an updated host. I'll update if any of those machines come back to the Events view.

akh• 12 years ago

This isn't a direct answer, but for others who are coming to this page and need help on where to start, here is some info on what to do now from RightScale's CTO, it also has a little comparison to what happened in Dec 2011 reboot too (just an interesting side note): http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...

vacri2• 12 years ago

There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues.

Anecdata, but a possible explanation.

chillericed• 12 years ago

this has been true for us so far as well. But I wouldn't rule out that they have more scheduled maintenances that haven't been put out yet.

nkozyra• 12 years ago

Concur, though also anecdotal.

pjl• 12 years ago

I don't think the issue is related to HVM. I have instances scheduled for system reboot that are not HVM.

Xorlev3• 12 years ago

We have a lot of non-HVM scheduled for reboot.

One of them was our VPN server, scheduled for reboot during our tech demo.

rurounijones2• 12 years ago

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not.

I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option.

Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

iaw• 12 years ago

IMO, The problem is that the manager of the cloud product isn't the person that is ordering the reboot. The manager of the cloud product considers communication with customers to mitigate their issues a primary concern. Person ordering the reboot is more 'technically' focused and representative of why engineers have a hard time as managers (eg ignoring the potential ill will this type of action can create).

toomuchtodo7• 12 years ago

I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option.

I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining.

Not cool Amazon.

rurounijones2• 12 years ago

Good point, I wonder if they didn't because that would have crippled capacity due to people rebooting, using up the patched hosts and leaving others without the ability to spin up VMs as it sounds like a significant % of hosts are bad at this point..

rands33115• 12 years ago

Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?

res0nat0r• 12 years ago

This is normally what they do when they roll out forced reboots like this.

Note the forum posting says there isn't guarantee of being on an updated host...that is because the patching isn't complete across the region yet.

astral303• 12 years ago

There is not enough capacity. I think Amazon did not make this decision lightly.

pas2563• 12 years ago

Stop relying on long running instances. Design your infrastructure for failure. This is not Amazon's responsibility.

x0x02• 12 years ago

And having dealt with ec2 support, I would have expected amazon to half-ass this just the way they did.

Another reason to consider owned & operated hardware. Or at least something like monthly rentals.

kulids772• 12 years ago

How would owned & operated hardware or monthly rentals be better?

omni3• 12 years ago

I just got notified that my Elasticache instances are getting restarted, too.

contingencies2• 12 years ago

While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no solid tools for doing it right now.

What will the future look like? I believe that a standard, git-like command line tool for infrastructure, sort of a 'brandless ec2' or 'P-abstracted IaaS' version of heroku, will replace all of the current-era providers with a free market for infrastructure based on transparency, real uptime and performance analysis and incident observations by multiple third parties with cryptographic reputation management. Two angles converging on that at http://stani.sh/walter/pfcts and http://ifex-project.org/

irollboozers• 12 years ago

Curious, what do large scale companies in the cloud currently do for this?

btown• 12 years ago

Is Rightscale's multi-cloud offering a step in the direction you're advocating?

http://assets.rightscale.com/uploads/pdfs/RightScale-Technic...

contingencies2• 12 years ago

Kind of. At a glance, it's new, commercial and they gloss over the complexities... therefore I'm skeptical it really works as well as they say it does, and is leaning toward my 'untrustworthy as a long term platform' basket. Though they may have great tools, I believe history shows us that open source is the real way to resolve these very reasonable types of architectural concerns.

lucian1900• 12 years ago

OpenStack has some of those tools and some even have AWS backend shims.

I believe Rackspace are one of those involved in its development.

fletchowns3• 12 years ago

On AWS you should be prepared for an instance to disappear at any time, for any reason. Why is a scheduled reboot such a big deal?

omni3• 12 years ago

It's a big deal because about half of my 100 instances are all going down at roughly the same time. Distribution and replication save you if you have 3 boxes and 1 dies. If all 3 die at the same time, you're still screwed.

all_usernames• 12 years ago

Each Availability Zone is being rebooted on a different day. Best practices dictate HA clusters with >=1 instance in each AZ. So, in theory well-designed EC2 systems can withstand this without interruption.

But yeah, this still sucks.

reedlaw2• 12 years ago

That's true, but how many services get this right in practice? It's not easy to ensure all instances will perform well in case of failover. A lot of apps are running on a single instance and depend on Amazon's fairly good track record.

fletchowns3• 12 years ago

Obviously these individuals have never visited the EC2 forums: https://forums.aws.amazon.com/search.jspa?objID=f30&q=unresp...

If you are relying on a single instance and depending on Amazon's track record, you're doing it wrong.

akurilin• 12 years ago

Will new instances created now not have to be rebooted? A lot of people on EC2 can probably re-create their vms now instead of having to wait for the reboot, no?

alanning• 12 years ago

At the time of writing, no. See the linked discussion. The real issue here, as the RightScale guys point out, is that there is no way to reliably provision a patched instance. Although the comment in the discussion forum by the EC2 fleet manager says they are working on a tool that will let us know whether an instance is fixed or not.

gpjonesii• 12 years ago

A couple of points:

Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up.

Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to update/deprecate/etc. The longer your instances are running (or the more specialized the type of instance is), the more likely you'll see an eviction notice. It should be part of a good practice to launch new instances often as new AMIs become available, or as private AMIs are updated for security patches, etc. - at least monthly! Autoscaling and solid config management simplifies this practice greatly.

Good Luck!

spullara• 12 years ago

They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.

coolbeans012• 12 years ago

Could you please confirm or provide evidence for such speculation?

I don't see anything about this on google.

I heard it is because they are having power issues within their datacenter.

Xorlev3• 12 years ago

This helps services, but as some point you have to run a database layer too ;) Cassandra helps, but it's not the whole story during large-scale close together reboots like these.

It ends up being a decent bit of manual operator time spent when security patches force AWS to reboot. You have to be pretty careful about any service that can't lose members as quickly due to bootstrap times or technology limitations, e.g. RDBMSes.

For things running in an ASG, it's trivial to let it die or just kill it.

pas2563• 12 years ago

Each AZ looks to me like it is a day apart. Surely that is enough time.

madaxe_again3• 12 years ago

Yes, isn't it wonderful that you have to completely remake your autoscaling groups every time you add/remove/modify an ELB. Thank Cthulhu (or opscode...) for chef.

kennu2• 12 years ago

Autoscaling is your friend, but you can also use "auto-healing" if your stack is built on Amazon's AWS OpsWorks and you just want to keep a single instance alive. It will automatically spawn a replacement instance and reattach and mount any EBS volumes.

eropple10• 12 years ago

My understanding is that that doesn't work correctly in the case of AZ failure; the EBS data isn't duped to another AZ and so your instance will fail to come up. So it's not really a solution.

rrggrr• 12 years ago

This feels like a critical security or stability update.

ianlevesque• 12 years ago

XSA-108 perhaps? http://xenbits.xen.org/xsa/

AWS is certainly large enough to have access to an embargoed update.

akerl_4• 12 years ago

The list of people receiving pre-disclosure access is public:

http://www.xenproject.org/security-policy.html

monjinan3• 12 years ago

I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public.

The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyone else and keeps getting better.

These reboots are going to save a lot of peoples butts, they wouldn't give a 48 hour notice if they had a better option.

That being said, I'm very curious to see what info is released next week.

dvidsilva• 12 years ago

yeah, we had a few minutes downtime but nothing too bad; definitely rather have some few time off than a security threat.

fleitz• 12 years ago

Yup, fundamentally it's not following best practices vs. a few minutes of downtime. It's a far better call than not forcing the update.

Of course all the people who built shoddy cloud presences are going to be screaming blue murder that their 'oh so successful' but 'couldn't be fucked to run in multiple AZs' businesses are going to be down for a few minutes.

It reminds me of doing tech support during the dotBomb, every daytrader would phone about how they were losing thousand per second while their internet was down, I'd ask them to switch to their backup connection, and then explain that they had the cheapest residential connection available that's inappropriate for day trading and that if they are making thousands per second they should have a backup connection in place.

Then I'd upsell them on a business package and have a guy there in 24 hours.

madaxe_again3• 12 years ago

You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic.

We spend $30k a month with AWS and they treat us like crap. The tech is ok, but their communications and customer service are mindblowingly atrocious.

If I can give one bit of advice - don't pay for either their support or an account manager. They don't provide either, and their staff turnover is severe, so your account manager will be a noob to their business every few weeks.

npinguy2• 12 years ago

I apologize for your poor experience with Amazon, and have no doubt that many mistakes were made with your account.

But you should recognize that your experience is the exception not the rule. The vast majority of customers have nothing but good things to say, and Amazon works tirelessly to maintain that reputation.

nicobn2• 12 years ago

When a script mistakingly provisioned a ridiculous amount of read/write capacity on hundreds of DynamoDB tables, we were presented with a bill in the thousands of dollars. We got reimbursed after talking with our account manager for 15 minutes.

Maybe my experience is the exception, but my account manager has always been extremely helpful at resolving whatever issues we had.

monjinan3• 12 years ago

Never said I got reimbursed for AWS. As far as I know the SLA is on their infrastructure, was your problem tied to a specific OS?

wahnfrieden• 12 years ago

I've been reimbursed after an issue that led to an unknown malicious attacker spinning up many maximally-sized EC2 instances for a day. We just had to ask support and comply with cleaning up our instances (which we had done anyway...).

james-at-aws2• 12 years ago

Hi madaxe_again,

I work at AWS (npinguy does not to the best of my knowledge). If you've spent $8,000 on maintaining instances you wouldn't have running otherwise to reproduce a bug on our behest, we don't want you to have to pay for that. I'd like to look into getting a refund for you. Is there an email address I can reach you at to get more details on the situation?

barkingcat• 12 years ago

this might be the xen vulnerability that is as yet undisclosed to the public.

emanuelmedina• 12 years ago

http://www.cvedetails.com/vulnerability-list/vendor_id-6276/...

tszming2• 12 years ago

I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window.

Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have no choice.

jsolson• 12 years ago

Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmainte...

(note: I work on GCE, more or less)

tszming2• 12 years ago

Yes you are right, but GCE does not offer the flexibility that allow user to reboot themselves in their own convenient time.

Note: I am not saying the 'live migration' in GCE does not work, I am just saying I've more confident (well...peace of mind, IMO) to shutdown the my own database manually (which is automated and tested), ensure all data are flushed to disk, clients are disconnected gracefully and slave has promoted to master.. and things like that, rather than some black magic like 'live migration'

Of course, I am not saying 'live migration' is wrong, but it is just my preference.

gazarsgo• 12 years ago

VMWare has this feature too, they call it vMotion. I haven't found anyone who could explain the resilience of vMotion in production use cases though.

vacri2• 12 years ago

I think you have to stop/start (from EC2) to move to a new host, not just reboot (from within the instance).

sounds• 12 years ago

These instance types are not affected:

T1, T2, M2, R3, and HS1 instance types are not affected.

http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...

eropple10• 12 years ago

I'd pour one out for the people with the gigantic I2's that they'll have to reprovision, but I'm pretty sure they can afford their own, they don't need me to pour one out for them.

mdellabitta3• 12 years ago

Are you kidding? I have to run i2s. I can't afford anything else! :)

fjordan2• 12 years ago

CVE-2014-6271 was posted on the AWS Security Bulletin earlier today:

http://aws.amazon.com/security/security-bulletins/CVE_2014_6...

My guess is that they cannot rely on all users to perform these updates.

fjordan2• 12 years ago

can someone explain the downvotes?

rands33115• 12 years ago

Already confirmed by AWS its not related. Confirmed in this thread its not related. That issue is an OS level problem, AWS won't touch your OS.

mustafab• 12 years ago

Does anyone think it might be related to the recent bash bug? It even affects dhcp clients which is a case with aws. https://securityblog.redhat.com/2014/09/24/bash-specially-cr...

eropple10• 12 years ago

It is unrelated. This is a Xen patch.

bradhe• 12 years ago

Does anyone think it might be related to the recent bash bug

That would be...pathetic, at best. There's a Xen bug that's being embargoed 'till 2014-10-01. I think that's more likely.

_asummers• 12 years ago

The bash bug does not require a system restart. In fact, it can be updated without shutting down the running bash processes, as the vulnerability only affects new shells.

allegory2• 12 years ago

This is one of the reasons why we fished out for 3x full racks at different DCs and bought our own kit. It's always our schedule.

ceejayoz• 12 years ago

Yes, but Amazon's schedule involves them having advance, non-public knowledge of major security flaws. In some ways, you've got a false sense of security.

allegory2• 12 years ago

Well the bash vulnerability and then immediate disclose of another vulnerability makes that statement pretty moot.

We also have deeply layered non-hetereogenous kit so you have to punch hard to get through it all to something useful.

angrybits• 12 years ago

Everything else is yours too, like getting an ROI out of the capital your spending. It's nice you have that kind of money laying around to spend on iron instead of advertising (as an example), but not everyone does.

RossM• 12 years ago

I have an instance listed as scheduled for reboot in 2 days but haven't received an email. Without this post I'd have been caught out on Saturday wondering why I'm getting a ton of message queue connection errors.

Downtime I can live with, but unreported planned downtime? Not impressed.

kordless• 12 years ago

Check your spam folder. A few of their mail servers occasionally have blacklisted IPs.

frozenport• 12 years ago

What if Amazon's infrastructure fails during the reboot?

scottlinux• 12 years ago

Not only EC2 instances, but also RDS instances.

I received various maintenance email notifications for RDS reboots.

AngelaT• 12 years ago

AWS Reboot FAQs available here: http://www.rightscale.com/blog/rightscale-news/aws-reboot-fa...

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

Related Stories