AWS Nuke – delete all resources associated with AWS account

https://github.com/rebuy-de/aws-nuke

by fortran77 • 4 years ago

222 119 4 years ago

github.com

Centigonal• 4 years ago

This is a fantastic utility to have in my back pocket, and the attention to safety is commendable. nice!

pcthrowaway2• 4 years ago

Needs a flag to run non-interactively

danw1979• 4 years ago

I nearly took this bait

code_duck• 4 years ago

I can barely imagine what is in my AWS account. The last time I used it was 2011.

OJFord6• 4 years ago

I would love a `terraform <plan|apply> --nuke`.

Not `destroy` - the opposite - destroy/nuke what I don't have in my config.

iasay2• 4 years ago

Oh that would be glorious. Not sure how it'd be possible though.

mrg2k8• 4 years ago

May be possible using driftctl and some API calls.

OJFord6• 4 years ago

I've hitherto assumed it didn't because it wasn't, short of calling every single `*:Describe*` API anyway.

But the existence of aws-nuke makes me think (I haven't looked into what it's doing yet) there must be a better way of discovering used services/resources. Through billing perhaps?

thayne4• 4 years ago

That's one of my big complaints with terraform, and it's state-based system. It doesn't inform you if there are resources you don't know about.

inkyoto2• 4 years ago

It is a feature and a good one (with a caveat though).

terraform is best used on a per solution basis: one solution, one dedicated terraform project that will manage its state and its own state only. Multiple projects and people work carry out work in the same cloud account in parallel, and their terraform projects are not meant to interfere with each other. It works best for solutions that make use of fully managed cloud services.

Then there are also platform or connectivity level cloud resources (e.g. Transit Gateway and subnets that are mapped into the internal organisational network address space in AWS) that a random terraform project ought not to manage.

Lastly, if there is an actual need, a resource that terraform does not know about can be manually imported into the terraform's project state. This works best when infrastructure level resources were manually created a while ago and now have to be refactored into and managed by a terraform project. It is a tedious process that has to proceed with a lot of caution.

The caveat. It gets somewhat tricky when a non-serverless cloud resource requires an explicit subnet range allocation within an existing and managed CIDR or similar. There is no one solution to fit it all but containing such projects to their own dedicated VPC and setting up the VPC peering between a solution specific VPC and the main account VPC usually works satisfactory. That is, for example, how Kafka (AWS MSK) can be introduced into an AWS account without affecting existing CIDR mapping.

lvh2• 4 years ago

I've used aws-nuke a bunch but the use case seems significantly diminished now that AWS Organizations has the ability to delete entire accounts.

wielebny• 4 years ago

Last time I've checked it was a lengthy process involving attaching a credit card, leaving the organization and then deleting the account. Has it been changed?

bonobocop• 4 years ago

https://docs.aws.amazon.com/organizations/latest/APIReferenc...

Now has an API (with some caveats)

dhess• 4 years ago

I'm in the process of doing this now. You can close accounts from Control Tower without needing to log in as root to each separate account, adding a credit card, removing it from the org, and then closing it manually.

However, you can only close them from Control Tower at the rate of 2 to 3 per month, due to a hard limit quota which cannot be changed, even if you request it. Needless to say, this sucks when you've followed AWS's own best practices and created lots of accounts using Control Tower's "vending machine."

AWS's archaic account model is one reason we've switched to GCP.

lvh2• 4 years ago

It used to be, but there's an API now: https://aws.amazon.com/blogs/mt/aws-organizations-now-provid...

evanfarrar• 4 years ago

https://github.com/genevieve/leftovers

A co-worker made this when we worked together on a project that ran a large number of terraform configurations in CI against real IaaSes. Each account was a sandbox, so we would run this at the end of a pipeline to clean up any failures or faulty teardowns.

bilekas3• 4 years ago

Wow! This is actually some great work!

crazysim• 4 years ago

Unlike aws-nuke, there seem to be actual automated tests here for this tool. That's really neat!

Though, I do feel the project reaches out to quite a lot of IAAS's for its level of current maintenance, which seems to be somewhat zero.

aws-nukes gets contributions here and there and maybe that's because it's quite focused.

That's a real shame, the implementations for each provider look to be great, standalone by themselves.

jftuga• 4 years ago

This fork is 36 commits ahead. It aims to stabilize deletion (so far for AWS) and add extended regex filtering support.

https://github.com/notrepo05/leftovers

hk1337• 4 years ago

I would think they should probably change the name so it doesn’t come across as an official AWS product but seems nice.

orf8• 4 years ago

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete.

Found this out after using AWS nuke

OJFord6• 4 years ago

That's a weird one, but surely an aws-nuke bug? It must already use a deliberate order - there's plenty of resources that need anything linked/constituent deleted first - so that order is/was just not correct for those?

bilekas3• 4 years ago

Nah I've seen this too. It is considered an expected behavior. In a way it makes sense. But it's not well documented.

Marazan3• 4 years ago

No, Batch's interaction with IAM roles and permission is super weird and not at all documented.

It is easy to screw it up.

bilekas3• 4 years ago

Not sure why but I laughed out loud reading this.

Edit : I've ran into this but it's considered a feature not a bug!

iasay2• 4 years ago

There's loads of stuff like this.

If you detach an EC2 instance from a virtual tape library and destroy it you can't delete the tape library any more. Even AWS support couldn't delete it. This is fine until you have 60TB of tapes online and are paying for it.

Fortunately we found an EBS snapshot of the VM.

TedDoesntTalk3• 4 years ago

This is my worst nightmare. I had a situation like this where I was able to essentially ignore the entire AWS account and cancel the attached credit card. Amazon somehow linked this AWS account to my retail Amazon.com account and started booking my personal credit card.

Marazan3• 4 years ago

Even better if you have your cloud formation with the iam role in the same stack and there is something wrong the the rollback can happen such that the iam role gets deleted standing the compute environment and failing the rollback.

zxcvbn40383• 4 years ago

Update the stack to use a different role with cloudformation in its trust policy, then you can delete the stack with the new policy. Has to be two operations to work.

zxcvbn40383• 4 years ago

Try logging into the console as root, that lets you delete anything.

You can also try updating the cluster to have a new role.

orf8• 4 years ago

That’s… not the issue. The cluster gets stuck in a deleting state forever. And at which point you’re unable to update it.

jiggawatts3• 4 years ago

This is because neither AWS nor Azure use referential integrity in any of their "cloud scale" databases. For example, Azure uses some hideous JavaScript-based document DB where things like renames, moves, and deletes are hit & miss at best. A never-ending whack-a-mole of bugs and issues.

Remember boys and girls: Being "cloud scale" means data corruption and referential integrity violation!

orf8• 4 years ago

I don’t think it is - the issue is that the Batch service needs to assume your role in order to clean up associated cluster resources like auto scaling groups.

If the role is deleted, it can’t do this.

efxhoy• 4 years ago

Not even redshift enforces uniqueness on their primary keys or referential integrity on their foreign keys. That was a fun finding out…

deathanatos3• 4 years ago

Not only referential integrity, but also not supporting Read-Your-Writes[1]. Cloud scale! Nothing like,

1. Create resource. Success.

2. Attempt to use/reference first resource in another resource/call: Failure: Referenced resource does not exist. Odd.

3. Create resource again? Failure! Resource already exists.

Our scripts have so many retry loops and arbitrary pauses mixed in to account for garbage like this. Distinguishing "did the call fail?" from "or is the system just lost in the land of eventual inconsistency?" ugh.

And yeah. shout out to Azure AAD where I can have a role assignment that is granting permissions to "unknown". We call them ghosts.

[1] https://jepsen.io/consistency/models/read-your-writes

bergenty4• 4 years ago

Sounds like it’s a bug in AWS nuke as well then.

suifbwish• 4 years ago

The idea that we can actually delete anything from a non physical system the hyper visor of which we do not control is absolute nonsense. Unless you have control over the physical system and the hypervisor all you can do is destroy your ability to access the information. We can never have confidence of how it truly works back end.

yieldcrv• 4 years ago

this will shave 20% off of Amazon's earnings!

jrockway• 4 years ago

I sure wish cloud providers had a "fake" implementation of their APIs that I could use in tests, for cases like the one that this program was written for. There are so many cross-dependencies that can be checked ("sorry can't create that VM, you need a service account first" or "VM names can only be 69 characters and can't contain '!'") without creating actual resources. You can test all the edge cases that resolve to failure in a unit test, and then only need one integration test to get that end-to-end "when I run 800 Terraform files, can I talk to my webserver at test239847abcf123.domain-name.com?".

I've found that the alternative to this is either a CI run that takes hours, or simply not testing the edge cases and hoping for the best. The former is a nightmare for velocity, the latter is a nightmare for having more than one person working on the team ever.

Lately, I've been fortunate to not be working in the "general cloud provider" space (where everything is a black box that can change at any moment, and documentation is an afterthought of afterthoughts) and have only focused on things going on in Kubernetes. To facilitate fast tests, I forked Kubernetes, made some internal testing infrastructure public, and implemented a kubelet-alike that runs "containers" in the same process as the test. For the application I work on at work, we are basically a data-driven job management system that runs on K8s. People have already written the easy tests; build the code into a container, create a Kubernetes cluster, start the app running, poke at it over the API. These take for-fucking-ever to run. (They do give you good confidence that Linux's sleep system call works well, though! Boy can it sleep.) With my in-process Kubernetes cluster, most of that time goes away, and you can still test a lot of stuff. If you want to test "what happens when a very restrictive AppArmor policy is applied to all pods in the cluster", yeah, the "fake" doesn't work. If you want to test "when my worker starts up, will it start processing work", it works great. (And, since the 100% legit "api machinery" is up and running, you can still test things like "if my user's spec contains an invalid pod patch, will they get a good error message?") Most bugs (and mistakes that people make while adding new features) are in that second category, and so you end up spending milliseconds instead of minutes testing the parts of your application that are most hurtful to users when they break. (And, nobody is saying not to run SOME live integration tests. You should always start with, and keep, integration tests against real environments running real workloads. When they pass, you get some confidence that there are no major showstoppers. When they fail, you want the lighter-weight tests to point you with precision to the faulty assumption or bug.)

Anyway... it makes me sad that things like aws-nuke are the kind of tooling you need to produce reliable software focused on deployment on cloud providers. I'd certainly pay $0.01 more per VM hour to be able to delete 99% of my slow tests. But I think I'm the only person in the world that thinks tests should be thorough and fast, so I'm on my own here. Sad.

time0ut• 4 years ago

I’ve contributed to and used this tool extensively. I have accounts where I run this thing out of a CodeBuild job on a cron schedule to dejunk after a day of terraform development. Fantastic tool.

thedougd• 4 years ago

Are Azure resource groups still pretty comprehensive? You use to be able to create all your stuff in a resource group. Delete the group and it deletes all the stuff with it. Super convenient.

bbkane• 4 years ago

Yeah they seem to be. Except Azure AD stuff, which kinda makes sense (still wish that part were more easily automated)

qwertyuiop_• 4 years ago

Project sponsored by Azure ?

svenwltr2• 4 years ago

No, we use it to clean up after our Terraform integration-tests. It is not always possible to use `terraform destroy`, because it might happen that the state does not allow it (at least that was our experience a couple of years ago).

ezekiel11• 4 years ago

so i have this weird monthly charge I cannot for the love of god find out what is causing it. I cancelled my credit card and now my credit score has taken a hit.

All because some unknown AWS service that is in some zone that I cannot find at all and neither can AWS support.

___8___• 4 years ago

Is the charge the same every month?

yownie• 4 years ago

What does the account statement say?

GeorgeWBasic• 4 years ago

That seems like the sort of thing chargebacks are intended to solve. Your credit shouldn't be taking a hit for that.

aaaaaaaaata3• 4 years ago

I love how everyone wants to help you/figure out the mystery bug, because, after all, you just need knowledge and experience to maybe avoid getting screwed around by AWS.

arkadiyt• 4 years ago

99% of our AWS resources are terraformed, but developers constantly push back / want to use the console to create stuff to test or play around with. So we setup a separate "hack" AWS account and give them admin access in there, and have an automated job using AWS Nuke to delete everything in there once a quarter.

Frajedo5• 4 years ago

We give each one of our developers their very own aws account managed through AWS organizations service. They are full administrators and responsible for resources and cost.

So far we haven’t had any issues or bad surprises, although we have setup some aws billing alerts just in case.

Feel free to make them responsible for cost and resources and you’ll be surprised how well they can manage their own account.

vageli• 4 years ago

We give each one of our developers their very own aws account managed through AWS organizations service. They are full administrators and responsible for resources and cost.

How many developers work at your organization?

cure• 4 years ago

although we have setup some aws billing alerts just in case.

My experience with these has been decidedly mixed. As in, you define them and never, ever see an alert.

101008• 4 years ago

Feel free to make them responsible for cost and resources and you’ll be surprised how well they can manage their own account.

Wow, this is horrible. I understand responsability but this is too much. Are other employees responsible if the company loses money for their actions?

thayne4• 4 years ago

I'm more worried about someone inexperienced with AWS accidentally doing something really expensive than any kind of intentional abuse.

devonkim• 4 years ago

I think it really depends upon a number of factors but even pretty smart people can make stupid mistakes, especially when it comes to security in AWS. I’m familiar with several cases where engineers fired up old AMIs and got the instances compromised within an hour because they were running old, vulnerable software and ran it in a publicly routable subnet. There’s some basic rules to follow that can help avoid issues like those though that as organizations scale need to enforce to a greater degree eventually. Disallowing provisioning their own VPCs, disallowing publicly routed subnets, and establishing some decent auth infrastructure is all a good start that will work for a long time and have minimal friction for users. I’m a strong believer in security as a UX problem where doing the Right Thing should be easier than doing the Lazy / Bad Thing so I feel if people are having issues doing things the right way I’ve messed up and need to improve usability and meet my users where they are to achieve my own goals of a secured infrastructure.

Giving people responsibility and autonomy also comes with some responsibilities by the providers in a shared responsibility model is all I’m saying and every policy works out fine until it doesn’t.

inkyoto2• 4 years ago

We have done something similar: a sandbox account that developers and solution designers can play around in, experiment and manually create resource in as long as resources are properly tagged (we have devised internal naming conventions). They are also responsible for the clean-up; a resource that is not properly tagged is purged automatically after a 8 hour time lapse.

Other accounts and environments (including dev) require everyone to follow a streamlined process: read only access to the account, a fully documented solution design and a corresponding terraform project in GitHub. terraform project checkin triggers a pull request for a review and an approval. Once the pull request has been scrutinised and merged, CI/CD runs terraform to provision resources in the account.

AtomicOrbital• 4 years ago

to me its obvious amazon managers intentionally avoid giving us such a delete all resources button simply to maximize profits

adamw2k• 4 years ago

Funny enough, on the completely opposite end of the spectrum, I was once surprised that after you close an account (post M&A in this case), you can typically restore the resources for ~90 days if you decide you want it back and don't nuke before you request the closure. Can be useful or scary depending on the contents of the account...

master_crab• 4 years ago

Yeah it’s actually a selling point in case a disgruntled employee or hacker gets credentials and asks for the account to be shutdown.

fideloper• 4 years ago

This is also a useful tool just to find resources hiding within an account. It defaults to performing a dry run (it’s a PITA to actually delete stuff, on purpose).

tpmx3• 4 years ago

I noticed that this GCP article about Terraform best practices linked to a similar tool: cloud-nuke (https://github.com/gruntwork-io/cloud-nuke)

https://cloud.google.com/docs/terraform/best-practices-for-t...

After you run the terraform destroy command, also run additional clean-up procedures to remove any resources that Terraform failed to destroy. Do this by deleting any projects used for test execution or by using a tool like cloud-nuke.
Warning: Don't use such tools in a production environment.

It seems like this tool only supports AWS though, at least nowadays.

I kind of feel bad for the GCP people here. Damned if you do (link to an external project, which might change), damned if you don't.

Besides this thing, I really enjoyed reading this article though! AWS is missing this kind of content.

thematrixturtle2• 4 years ago

In GCP, the tool is not necessary, because you can simply delete the containing project to "nuke" any resources attached to it.

tpmx3• 4 years ago

There is a 30 day "marked for deletion period" after you do this. The resources related to the project aren't immediately nuked.

https://cloud.google.com/resource-manager/docs/creating-mana...

emersonrsantos• 4 years ago

Namespaces could solve all of that. Why AWS haven't added that yet? I understand that they rarely change the service, but c'mon, we're in 2022.

aaaaaaaaata3• 4 years ago

How does that help them make more money?

thayne4• 4 years ago

It astounds me that aws doesn't have a way to do this built in. Or even just a way to list all the resources you have, across all services in an account, without having to make thousands of API calls and hope you didn't miss a resource type.

anothernewdude• 4 years ago

Shout out to my Azure homies enjoying their Resource Groups.

brundolf• 4 years ago

The fact that this is hard enough to do it requires a dedicated third-party tool

44gg44gg• 4 years ago

I deleted my AWS account because SES refused to take me out of sandbox due to a security violation that could not be disclosed to me. Interesting that almost all forms of SMTP are blocked once you are on some secret black list.

aporetics• 4 years ago

What’s the fallout?

svenwltr2• 4 years ago

Hello. I am one of the maintainers. Sorry that we did not spend much time for aws-nuke lately.

Feel free to ask any questions.

AWS Nuke – delete all resources associated with AWS account

Related Stories