AWS in 2025: Stuff you think you know that's now wrong

https://www.lastweekinaws.com/blog/aws-in-2025-the-stuff-you-think-you-know-thats-now-wrong/

by keithly • 11 months ago

387 279 11 months ago

lastweekinaws.com

cldcntrl3• 11 months ago

You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.

Not strictly true.

rthnbgrredf2• 11 months ago

Elaborate.

cldcntrl3• 11 months ago

The whole auto-balancing thing isn't instant. If you have a burst of writes with the same key prefix, you'll get throttled.

hnlmorg12• 11 months ago

Not the OP but I’ve had AWS-staff recommend different prefixes even as recently as last year.

If key prefixes don’t matter much any more, then it’s a very recent change that I’ve missed.

QuinnyPig• 11 months ago

I should have been more clear. You still need to partition, but randomizing the prefixes hasn't been needed since 2018: https://web.archive.org/web/20240227073321/https://aws.amazo...

ed_g• 11 months ago

Generally speaking this isn't something Amazon S3 customers need to worry about - as others have said, S3 will automatically scale index performance over time based on load. The challenge primarily comes when customers need large bursts of requests within a namespace that hasn't had a chance to scale - that's when balancing your workload over randomized prefixes is helpful.

Please see the documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...

This 2024 re:Invent session "Optimizing storage performance with Amazon S3 (STG328)" which goes very deep on the subject: https://www.youtube.com/watch?v=2DSVjJTRsz8

And this blog that discusses Iceberg's new base-2 hash file layout which helps optimize request scaling performance of large-scale Iceberg workloads running on S3: https://aws.amazon.com/blogs/storage/how-amazon-ads-uses-ice...

vvoyer• 11 months ago

This 2024 re:Invent session says exactly the opposite:

"If you want to partition your data even better, you can introduce some randomness in your key names": https://youtu.be/2DSVjJTRsz8?t=2206

FWIW The optimal way we were told was to partition our data was to do this: 010111/some/file.jpg.

Where `010111/` is a random binary string which will please both the automatic partitioning (503s => partition) and manual partitioning you could ask AWS. Please as in the cardinality of partitions grows slower at each characters vs prefixes like `az9trm/`.

We were told that the later version makes manual partitioning a challenge because as soon as you reach two characters you've already created 36x36 partitions (1,296).

The issue with that: your keys are no more meaningful if you're relying on S3 to have "folders" by tenants for example (customer1/..).

SOLAR_FIELDS6• 11 months ago

You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small.

afandian2• 11 months ago

Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd.

They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature.

In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong).

It's just baffling.

time0ut2• 11 months ago

You can provision gateway endpoints for S3 and DynamoDB. They are free and considered best practice. They are opt-in though, but easy to enable.

tux3• 11 months ago

That is price segmentation. People who are price insensitive will not invest the time to fix it

People who are probably shouldn't be on aws - but they usually have to for unrelated reasons, and they will work to reduce their bill.

nbngeorcjhe• 11 months ago

People who are price insensitive will not invest the time to fix it

This just sounds like a polite way of saying "we're taking peoples' money in exchange for nothing of value, and we can get away with it because they don't know any better".

happytoexplain2• 11 months ago

People who are price insensitive will not invest the time to fix it

Hideous.

dmart• 11 months ago

VPC endpoints in general should be free and enabled by default. That you need to pay extra to reach AWS' own API endpoints from your VPC feels egregious.

otterley7• 11 months ago

Gateway endpoints are free. Network endpoints (which are basically AWS-managed ENIs that can tunnel through VPC boundaries) are not free.

S3 can use either, and we recommend establishing VPC Gateway endpoints by default whenever you need S3 access.

(Disclaimer: I work for AWS, opinions are my own.)

kbolino6• 11 months ago

The problem is that VPC endpoints aren't free.

They should be, of course, at least when the destination is an AWS service in the same region.

[edit: I'm speaking about interface endpoints, but S3 and DynamoDB can use gateway endpoints, which are free to the same region]

paulddraper3• 11 months ago

Well yeah that's the point....why route through the public internet.

otterley7• 11 months ago

Gateway endpoints are free. Network endpoints (which are basically AWS-managed ENIs that can tunnel through VPC boundaries) are not free.

S3 can use either, and we recommend establishing VPC Gateway endpoints by default whenever you need S3 access.

(Disclaimer: I work for AWS, opinions are my own.)

otterley7• 11 months ago

This is the intended use case for S3 VPC Gateway Endpoints, which are free of charge.

https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpo...

(Disclaimer: I work for AWS, opinions are my own.)

SOLAR_FIELDS6• 11 months ago

The problem is that the default behavior for this is opt-in, rather than opt-out. No one prefers opt-in. So why is it opt-in?

darkwater• 11 months ago

I think they know it. They are complaining it's not enabled by default (and so do I).

hinkley7• 11 months ago

Your job depends upon you misunderstanding the problem.

solatic3• 11 months ago

It's a design that is secure by default. If you have no NAT gateway and no VPC Gateway Endpoint for S3 (and no other means of Internet egress) then workloads cannot access S3. Networking should be closed by default, and it is. If the user sets up things they don't understand (like NAT gateways), that's on them. Managed NAT gateways are not the only option for Internet egress and users are responsible for the networks they build on top of AWS's primitives (and yes, it is indeed important to remember that they are primitives, this is an IaaS, not a PaaS).

bilalq2• 11 months ago

Fine for when you have no NAT gateway and have a subnet with truly no egress allowed. But if you're adding a NAT gateway, it's crazy that you need to setup the gateway endpoint for S3/DDB separately. And even crazier that you have to pay for private links per AWS service endpoint.

immibis2• 11 months ago

A company making revenue is not stupid.

torginus• 11 months ago

If you had an ALB inside the VPC that routed the requests to something that lives inside the VPC, which called the AWS PutObject api on the bucket, would that still be the case?

simonw4• 11 months ago

S3: "Block Public Access is now enabled by default on new buckets."

On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous.

But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!

SOLAR_FIELDS6• 11 months ago

I honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug

simonw4• 11 months ago

Sure... but last time I needed to jump through those hurdles I lost nearly an hour to them!

I'm still not sure I know how to do it if I need to again.

dghlsakjg• 11 months ago

I think the OPs objection is not that hurdles exist but that they move them every time you try and run the track.

crinkly2• 11 months ago

I just stick CloudFront in front of those buckets. You don't need to expose the bucket at all then and can point it at a canonical hostname in your DNS.

hnlmorg12• 11 months ago

That’s definitely the “correct” way of doing things if you’re writing infra professionally. But I do also get that more casual users might prefer not to incur the additional costs nor complexity of having CloudFront in front. Though at that point, one could reasonably ask if S3 is the right choice for causal users.

herpderperator• 11 months ago

For the sake of understanding, can you explain why putting CloudFront in front of the buckets helps?

dcminter5• 11 months ago

Not always that simple - for example if you want to automatically load /foo/index.html when the browser requests /foo/ you'll need to either use the web serving feature of S3 (bucket can't be private) or set up some lambda at edge or similar fiddly shenanigans.

andrewmcwatters3• 11 months ago

This sort of thing drives me nuts in interviews, when people are like, are you familiar with such-and-such technology?

Yeah, what month?

tester756• 11 months ago

If you're aware of changes, then explain that there were changes over time, that's it

sylens• 11 months ago

The thing to keep in mind with the "Block Public Access" setting is that is a redundancy built in to save people from making really big mistakes.

Even if you have a terrible and permissive bucket policy or ACLs (legacy but still around) configured for the S3 bucket, if you have Block Public Access turned on - it won't matter. It still won't allow public access to the objects within.

If you turn it off but you have a well scoped and ironclad bucket policy - you're still good! The bucket policy will dictate who, if anyone, has access. Of course, you have to make sure nobody inadvertantly modifies that bucket policy over time, or adds an IAM role with access, or modifies the trust policy for an existing IAM role that has access, and so on.

simonw4• 11 months ago

I think this is the key of why I find it confusing: I need a very clear diagram showing which rules override which other rules.

awongh3• 11 months ago

This is exactly what I use LLMs for. To just read the docs for me and pull out the base level demo code that's buried in all the AWS documentation.

Once I have that I can also ask it for the custom tweaks I need.

dcminter5• 11 months ago

This could not possibly go wrong...

You're braver than me if you're willing to trust the LLM here - fine if you're ready to properly review all the relevant docs once you have code in hand, but there are some very expensive risks otherwise.

jiggawatts• 11 months ago

Back when GPT4 was the new hotness, I dumped the markdown text from the Azure documentation GitHub repo into a vector index and wrapped a chatbot around it. That way, I got answers based on the latest documentation instead of a year-old LLM model's fuzzy memory.

I now have the daunting challenge of deploying an Azure Kubernetes cluster with... shudder... Windows Server containers on top. There's a mile-long list of deprecations and missing features that were fixed just "last week" (or whatever). That is just too much work to keep up with for mere humans.

I'm thinking of doing the same kind of customised chatbot but with a scheduled daily script that pulls the latest doco commits, and the Azure blogs, and the open GitHub issue tickets in the relevant projects and dumps all of that directly into the chat context.

I'm going to roll up my sleeves next week and actually do that.

Then, then, I'm going to ask the wizard in the machine how to make this madness work.

Pray for me.

reactordev3• 11 months ago

They'll teach you how for $250 and a certification test...

cedws• 11 months ago

I’m getting deja vu, didn’t they already do this like 10 years ago because people kept leaving their buckets wide open?

aaronblohowiak3• 11 months ago

VPC peering used to be annoying; now there are better options like Transit Gateway, VPC sharing between accounts, resource sharing between accounts, and Cloud WAN.

TGW is... twice as expensive as vpc peering?

alFReD-NSH• 11 months ago

And vpc sharing is free. Cost and architecture are tied.

Hikikomori4• 11 months ago

More than twice as same AZ is free with peering. But if you're big enough you can get better deals on cost.

But unlike peering TGW traffic flows through an additional compute layer so it has additional cost.

klysm2• 11 months ago

VPC sharing is the sleeper here. You can do cross account networking all in the same VPC and skip all the expensive stuff.

aaronblohowiak3• 11 months ago

as long as your VPCs aren't too big, yea.

Ayesh• 11 months ago

CloudFront also has 1TB of free data transfer a month under the forever-free perks.

csours• 11 months ago

Strictly off topic:

Everything you know is wrong.

Weird Al. https://www.youtube.com/watch?v=W8tRDv9fZ_c

Firesign Theatre. https://www.youtube.com/watch?v=dAcHfymgh4Y

gurjeet• 11 months ago

It would've been nice if each of those claims in the article also linked to either the relevant announcement or to the documentation. If I'm interested in any of these headline items, I'd like to learn more.

mdaniel13• 11 months ago

I don't believe AWS offers permalinks, so it would only help until they rolled over the next documentation release :-(

They actually used to have the upstream docs in GitHub, and that was super nice for giving permalinks but also building the docs locally in a non-pdf-single-file setup. Pour one out, I guess

bob10293• 11 months ago

Glacier restores are also no longer painfully slow.

Wouldn't this always depend on the length of the queue to access the robotic tape library? Once your tape is loaded it should move really quickly:

https://www.ibm.com/docs/en/ts4500-tape-library?topic=perfor...

hinkley7• 11 months ago

Once upon a time Glacier was its own service that had nothing to do with S3. If you look closely (hi, billing data!) you can see vestiges of how this used to be, before the S3 team absorbed it as a series of storage classes.

Your assumption holds if they still use tape. But this paragraph hints at it not being tape anymore. The eternal battle between tape versus drive backup takes another turn.

bob10293• 11 months ago

I am also assuming that Amazon intends for the Deep Archive tier to be a profitable offering. At $0.00099/gb-month, I don't see how it could be anything other than tape.

chisleu3• 11 months ago

You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.

As of when? According to internal support, this is still required as of 1.5 years ago.

laurent_du• 11 months ago

He's not talking about the prefix, just the beginning of the object key.

viccis2• 11 months ago

The prefix is not separate from the object key. It's part of it. There's no randomization that needs to be done on either anymore.

arpinum• 11 months ago

I think there is some nuance needed here. If you ask support to partition your bucket then they will be a bit annoying if you ask for specific partition points and the first part of the prefix is not randomised. They tried to push me to refactor the bucket first to randomise the beginning of the prefix, but eventually they did it.

The auto partitioning is different. It can isolate hot prefixes on its own and can intelligently pick the partition points. Problem is the process is slow and you can be throttled for more than a day before it kicks in.

chisleu3• 11 months ago

but eventually they did it

They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads.

you can be throttled for more than a day before it kicks in

I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.

scubbo• 11 months ago

I've had two people tell me in the last week that SQS doesn't support FIFO queues.

TheP10002• 11 months ago

API gateway timeout increase has been nice.

coredog64• 11 months ago

It was always there but it required much more activity to get it done (document your use case & traffic levels and then work with your TAM to get the limit changed).

quesera• 11 months ago

I don't see that in this post.

I just started working with a vendor who has a service behind API Gateway. It is a bit slow(!) and times out at 30 seconds. I've since modified my requests to chunk subsets of the whole dataset, to keep things under the timeout.

Has this changed? Is 30 secs the new or the old timeout?

TheP10002• 11 months ago

https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-ap...

stevejb• 11 months ago

I just saw Weird Al in concert, and one of my favorite songs of his is "Everything You Know is Wrong." This is the AWS version of that song! Nice work Corey!

havefunbesafe• 11 months ago

Weird AL or Weird A.I.?

jeffbarr• 11 months ago

I also saw that concert.

Oh yeah, we were in the same row!

nodesocket• 11 months ago

I'll add: When doing instance to instance communication (in the same AZ) always use private ips. If you use public ip routing (even the same AZ) this is charged as regional data transfer.

Even worse, if you run self hosted NAT instance(s) don't use a EIP attached to them. Just use a auto-assigned public IP (no EIP).

  NAT instance with EIP
    - AWS routes it through the public AWS network infrastructure (hairpinning).
    - You get charged $0.01/GB regional data transfer, even if in the same AZ.

  NAT instance with auto-assigned public IP (no EIP)
    - Traffic routes through the NAT instance’s private IP, not its public IP.
    - No regional data transfer fee — because all traffic stays within the private VPC network.
    - auto-assigned public IP may change if the instance is shutdown or re-created so have automations to handle that. Though you should be using the network interface ID reference in your VPC routing tables.

themafia• 11 months ago

You get charged $0.01/GB regional data transfer, even if in the same AZ.

My understanding is that transfer gets charged on both sides as well. So if you own both sides you'll pay $0.02/GB.

pcthrowaway2• 11 months ago

Is transfer into the instance not always free anymore?

digianarchist• 11 months ago

Would love an AWS equivalent to Cloud Run but the lambda changes are welcome nonetheless.

aranelsurion• 11 months ago

Isn't Fargate the AWS equivalent of Cloud Run?

mdaniel13• 11 months ago

It seems Cloud Run comes "batteries included" for a lot more things that one must provision separate AWS resources to achieve https://cloud.google.com/architecture/migrate-aws-lambda-to-...

topher200• 11 months ago

I have a preempt-able workload for which I could use Spot instances or Savings Plans.

Does anyone have experience running Spot in 2025? If you were to start over, would you keep using Spot?

  - I observe with pricing that Spot is cheaper
  - I am running on three different architectures, which should limit Spot unavailability
  - I've been running about 50 Spot EC2 instances for a month without issue. I'm debating turning it on for many more instances

erulabs• 11 months ago

In terms of cost, from cheapest to most expensive:

1. Spot with autoscaling to adjust to demand and a savings plan that covers the ~75th percentile scale

2. On-demand with RIs (RIs will definitely die some day)

3. On-demand with savings-plans (More flexible but more expensive than RIs)

3. Spot

4. On-demand

I definitely recommend spot instances. If you're greenfielding a new service and you're not tied to AWS, some other providers have hilariously cheap spot markets - see http://spot.rackspace.com/. If you're using AWS, definitely auto-scaling spot with savings plans are the way to go. If you're using Kubernetes, the AWS Karpenter project (https://karpenter.sh/) has mechanisms for determining the cheapest spot price among a set of requirements.

Overall tho, in my experience, ec2 is always pretty far down the list of AWS costs. S3, RDS, Redshift, etc wind up being a bigger bill in almost all past-early-stage startups.

mdaniel13• 11 months ago

To "me, too" this, it's not like that AWS spot instance just go "poof," they do actually warn you (my recollection is 60s in advance of the TerminateInstance call), and so a resiliency plane on top of the workloads (such as the cited Kubernetes) can make that a decided "non-event". Shout out to the reverse uptime crew, a subset of Chaos Engineering

general1726• 11 months ago

I think there is more of us who kind of degenerated from doing it the AWS way - API Gateway, serverless lambdas mess around with IAM roles until it works, ... - to - Give me EC2 / LightSail VPS instance maybe an S3 bucket let's set domain through Route53 and go away with the rest of your orchestrion AWS.

calmbonsai• 11 months ago

There are entire industries that have largely de-volved their clouds primarily for footprint flexibility (not all AWS services are in all regions) and billing consistency.

regularfry2• 11 months ago

Honestly just having to manage IAM is such a time-suck that the way I've explained it to people is that we've traded the time we used to spend administering systems for time spent just managing permissions, and IAM is so obtuse that it comes out as a net loss.

There's a sweet spot somewhere in between raw VPSes and insanely detailed least-privilege serverless setups that I'm trying to revert to. Fargate isn't unmanageable as a candidate, not sure it's The One yet but I'm going to try moving more workloads to it to find out.

sklarsa• 11 months ago

Usually I write some IaC to automate this tedium so I only have to go through the IAM setup pain once. Now if requirements change, that's an entirely different story...

no_wizard2• 11 months ago

At what point is AWS worth using over other compute competitors when you’re using them as a storage bucket + VPS. They’re wholly more expensive at that point. Why not go with a more traditional but rock solid VPS provider?

I have the opposite philosophy for what it’s worth: if we are going to pay for AWS I want to use it correctly, but maximally. So for instance if I can offload N thing to Amazon and it’s appropriate to do so, it’s preferable. Step Functions, lambda, DynamoDB etc, over time, have come to supplant their alternatives and its overall more efficient and cost effective.

That said, I strongly believe developers don’t do enough consideration as to how to maximize vendor usage in an optimal way

nitwit005• 11 months ago

Your management will frequently be strangely happier to waste money on AWS, unfortunately.

Truly a marketing success.

Spivak2• 11 months ago

Because the compartmentalization of business duties means that devs are fighting uphill against the wind to sign a deal with a new vendor for something. It's business bikeshedding, as soon as you open the door to a new vendor everyone, especially finance, has opinions and you might end up stuck with a vendor you didn't want. Or you can use the pre-approved money furnace and just ship.

austinshea4• 11 months ago

Most work isn’t greenfield.

AWS can be used in a different, cost effective, way.

It can be used as a middle-ground capable of serving the existing business, while building towards a cloud agnostic future.

The good AWS services (s3, ec2, acm, ssm, r53, RDS, metadata, IAM, and E/A/NLBs) are actually good, even if they are a concern in terms of tracking their billing changes.

If you architect with these primitives, you are not beholden to any cloud provider, and can cut over traffic to a non AWS provider as soon as you’re done with your work.

wiether2• 11 months ago

I agree that using them as a VPS provider is a mistake.

If you don't use the E(lasticity) of EC2, you're burning cash.

For prod workloads, if you can go from 1 to 10 instances during an average day, that's interesting. If you have 3 instances running 24/7/365, go somewhere else.

For dev workloads, being able to spin instances in a matter of seconds is a bliss. I installed the wrong version of a package on my instance? I just terminate it, wait for the auto-scaling group to pop a fresh new one a start again. No need to waste my time trying to clean my mess on the previous instance.

You speak about Step Functions as an efficient and cost effective service from AWS, and I must admit that it's one that I avoid as much as I can... Given the absolute mess that it is to setup/maintain, and that you completely lock yourself in AWS with this, I never pick it to do anything. I'd rather have a containerized workflow engine running on ECS, even though I miss on the few nice features that SF offers within AWS.

The approach I try to have is:

- business logic should be cloud agnostic

- infra should swallow all the provider's pills it needs to be as efficient as possible

benterix• 11 months ago

That said, I strongly believe developers don’t do enough consideration as to how to maximize vendor usage in an optimal way

Because it's not straightforward. 1) You need to have general knowledge of AWS services and their strong and weak points to be able to choose the optimal one for the task, 2) you need to have good knowledge of the chosen service (like DynamoDB or Step Functions) to be able to use it optimally; being mediocre at it is often not enough, 3) local testing is often a challenge or plain impossible, you often have to do all testing on a dev account on AWS infra.

JCM9• 11 months ago

Some good stuff here. I wish AWS would just focus on these boring, but ultimately important, things that they’re good at instead of all the current distractions trying to play catch up on “AI.” AWS leadership missed the boat there big time, but that’s OK.

Ultimately AWS doesn’t have the right leadership or talent to be good at GenAI, but they do (or at least used to) have decent core engineers. I’d like to see them get back to basics and focus there. Right now leadership seems panicked about GenAI and is just throwing random stuff at the wall desperately trying to get something to stick. Thats really annoying to customers.

PartiallyTyped• 11 months ago

Leadership is looking to provide infrastructure for anyone to just pick a model and get on doing things without the hassle of setting things up.

anon7000• 11 months ago

They continue to have large teams working on core stuff. It’s just that they’re working at such a low level (like high perf virtualized networking on their custom network cards) that most people don’t hear about it or care that much.

causal• 11 months ago

This is super helpful. I would read a yearly summary like this.

beaviskhan• 11 months ago

Also S3 related: the bucket owner can now be configured as the object owner no matter where the object originated. In the past this was exceedingly painful if you wanted to allow one account contribute objects to a bucket in another account. You could do the initial contribution, but the contributor always owned the object, and you couldn't delegate access to a third account.

viccis2• 11 months ago

In EC2, you can now change security groups and IAM roles without shutting the instance down to do it.

Hasn't it been this way for many years?

Spot instances used to be much more of a bidding war / marketplace.

Yeah because there's no bidding any more at all, which is great because you don't get those super high spikes as availability drops and only the ones who bid super high to ensure they wouldn't be priced out are able to get them.

You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.

This one was a nightmare and it took ages to convince some of my more pig headed coworkers in the past that they didn't need to do it any more. The funniest part is that they were storing their data as millions and millions of 10-100kb files, so the S3 backend scaling wasn't the thing bottlenecking performance anyway!

Originally Lambda had a 5 minute timeout and didn’t support container images. Now you can run them for up to 15 minutes, use Docker images, use shared storage with EFS, give them up to 10GB of RAM (for which CPU scales accordingly and invisibly), and give /tmp up to 10GB of storage instead of just half a gig.

This was/is killer. It used to be such a pain to have to manage pyarrow's package size if I wanted a Python Lambda function that used it. One thing I'll add that took me an embarrassingly long time to realize is that your Python global scope is actually persisted, not just the /tmp directory.

indigodaddy• 11 months ago

Re: SG, yeah I wasnt doing any cloud stuff when that was the case. Never had to restart anything for an SG change and this must be at least 5-6 years..

buzzdenver• 11 months ago

IAM Role change is more recent though

Inufu• 11 months ago

You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.

Sorry, this is absolutely still the case if you want to scale throughput beyond the few thousand IOPS a single shard can serve. S3 will automatically reshard your key space, but if your keys are sequential (eg leading timestamp) all your writes will still hit the same shard.

Source: direct conversations with AWS teams.

PaulDavisThe1st2• 11 months ago

Can no longer login to my AWS account, because I never set up MFA.

Want to set up MFA ... login required to request device.

Yes, I know, they warned us far ahead of time. But not being able to request one of their MFA devices without a login is ... sucky.

berlesi• 11 months ago

Looks like something that you could solve easily through their support, no?

raffraffraff2• 11 months ago

Support don't talk to you unless you pay for support

austinshea4• 11 months ago

Talk to support: https://support.aws.amazon.com/#/contacts/one-support?formId...

PaulDavisThe1st2• 11 months ago

I did. That's just an AI, which says this:

I understand your situation is a bit unique, where you are unable to log in to your AWS account without an MFA device, but you also can't order an MFA device without being able to log in. This is a scenario that is not directly covered in our standard operating procedures.

The best course of action would be for you to contact AWS Support directly. They will be able to review your specific case and provide guidance on how to obtain an MFA device to regain access to your account. The support team may have alternative options or processes they can walk you through to resolve this issue.

Please submit a support request, and one of our agents will be happy to assist you further. You can access the support request form here: https://console.aws.amazon.com/support/home

That last URL? You need to login to use it ...

jp572• 11 months ago

Glacier restores are also no longer painfully slow.

I had a theory (based on no evidence I'm aware of except knowing how Amazon operates) that the original Glacier service operated out of an Amazon fulfillment center somewhere. When you put it a request for your data, a picker would go to a shelf, pick up some removable media, take it back, and slot it into a drive in a rack.

This, BTW, is how tape backups on timesharing machines used to work once upon a time. You'd put in a request for a tape and the operator in the machine room would have to go get it from a shelf and mount it on the tape drive.

browningstreet• 11 months ago

Yeah, but they've been robotic for decades since.

danudey3• 11 months ago

The most likely explanation is that they used a tape robot, such as the one seen here:

https://www.reddit.com/r/DataHoarder/comments/12um0ga/the_ro...

Which is basically exactly what you described but the picker is a robot.

Data requests go into a queue; when your request comes up, the robot looks up the data you requested, finds the tape and the offset, fetches the tape and inserts it into the drive, fast-forwards it to the offset, reads the file to temporary storage, rewinds the tape, ejects it, and puts it back. The latency of offline storage is in fetching/replacing the casette and in forwarding/rewinding the tape, plus waiting for an available drive.

Realistically, the systems probably fetch the next request from the queue, look up the tape it's on, and then process every request from that tape so they're not swapping the same tape in and out twenty times for twenty requests.

philistine• 11 months ago

I've read very definitive discussions on here that Glacier never used tape. It has always been powered off hard disks.

christina97• 11 months ago

They would definitely be using rubies robots given how uniform hard drives are. The only reason warehouses still have humans is that heterogeneity (different sizes, different textures, different squishiness, etc).

Twirrim3• 11 months ago

I can't talk about it, but I've yet to see an accurate guess at how Glacier was originally designed. I think I'm in safe territory to say Glacier operated out of the same data centers as every other AWS service.

It's been a long time, and features launched since I left make clear some changes have happened, but I'll still tread a little carefully (though no one probably cares there anymore):

One of the most crucial things to do in all walks of engineering and product management is to learn how to manage the customer expectations. If you say customers can only upload 10 images, and then allow them to upload 12, they will come to expect that you will always let them upload 12. Sometimes it's really valuable to manage expectations so that you give yourself space for future changes that you may want to make. It's a lot easier to go from supporting 10 images to 20, than the reverse.

mh-• 11 months ago

..oh. That's clever. Thanks for posting this.

kelnos• 11 months ago

I can't talk about it, but I've yet to see an accurate guess at how Glacier was originally designed.

It feels odd that this is some sort of secret. Why can't you talk about it?

donavanm6• 11 months ago

Im like 90% sure ive seen folks (unofficially) disclose the original storage and API decisions over the years, in roughly accurate terms. Personally I think the multi dimensional striping/erasure code ideas are way more interesting than the “its just a tape library” speculation/arguments. That and the real lessons learned around product differentiation as supporting technologies converge.

jp572• 11 months ago

I think folks have missed what I think would have been clever about the implentation I (apparently) dreamt up. It's not that "it's just a tape library", it's that it would have used the existing FC and picker infrastructure that Amazon had already built, with some racks containing drives for removable media. I was thinking that it would not have been some special facility purely for Glacier, but rather one or more regular FCs would just have had some shelves with Glacier media (not necessarily tapes).

Then the existing pickers would get special instructions on their handhelds: Go get item number NNNN from Row/shelf/bin X/Y/Z and take it to [machine-M] and slot it in, etc.

ElijahLynn• 11 months ago

That. Was a decent investment of my time as a devops engineer. Right to the point. I learned things.

kassner• 11 months ago

I haven’t used AWS in the last 5 years. Is IPv6 still somewhat of an issue? I remember some services not supporting it at all and making it impossible to manage as a IPv6-only network.

skywhopper2• 11 months ago

Yeah, it’s still limited, and a few things still require at least a dual stack setup.

1oooqooq• 11 months ago

gotta milk those ipv4 investment

mdaniel13• 11 months ago

You'll be glad(sic) to know they have specialized hostnames for IPv6 control plane endpoints, because AAAA records scare the olds https://github.com/aws/aws-cli/blob/2.28.14/awscli/botocore/...

conradludgate2• 11 months ago

We enabled ipv6 for our APIs at work. Nothing broke immediately, but we've had a steady stream of unreachable host errors related to ipv6 since then.

Turns out there're many incorrect implementations of Happy Eyeballs that cancel the ipv4 connection attempts after the timeout, and then switch to trying the AAAA records and subsequently throwing a "Cannot reach host" error. For context, in Happy Eyeballs you're supposed to continue trying both network families in parallel.

This only impacts our customers who live far away from the region they're accessing, however, and there's usually a workaround - in Node you can force the network family to be v4 for instance

donavanm6• 11 months ago

because AAAA records scare the olds

No. They break existing customer expectations.

There are heaps of dualstack API endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#dua... if that's what the client wants.

The amazonaws.com domain endpoints did not introduce ipv6/AAAA directly is (mostly) due to access control. For better or worse there are a lot of "v4 centric" IAM statements, like aws:SourceIp, in identity/resource/bucket policies. Introducing a new v6 value is going to break all of those existing policies with either unexpected DENYs or, worse, ALLOWs. Thats a pretty poor customer experience to unexpectedly break your existing infrastructure or compromise your access control intentions.

AWS _could_ have audited every potential IAM policy and run a MASSIVE outreach campaign, but something as simple as increasing (opaque!) instance ID length was a multi year effort. And introducing backwards compatibility on a _per policy_ basis is its own infinite security & UX yak shaving exercise as well.

So thats why you have opt-in usage of v6/dualstack in the client/SDK/endpoint name.

lysace3• 11 months ago

Paid AWS support got a lot less capable on average during these two decades . :/

My recent interactions with them would probably have been better if they were an LLM.

oblio2• 11 months ago

They probably are an LLM and if they aren't, their higher management is pushing for them to be LLMs by 2027 at the latest.

lysace3• 11 months ago

I still get strong Hyderabad vibes from the copy/pasting plus the occasional original sentence.

Perhaps they trained the LLM using that data though.

(Small customer though: yearly AWS spend around 80k. Support is 10% of that.)

raffraffraff2• 11 months ago

Availability Zones used to be randomized between accounts (my us-east-1a was your us-east-1c)

WTH?

slashdev• 11 months ago

Yeah this one drove me crazy

skywhopper2• 11 months ago

They did this to stop people from overloading us-east-1a.

It was fine, until there started to be ways of wiring up networks between accounts (eg PrivateLink endpoint services) and you had to figure out which AZ was which so you could be sure you were mapping to the the same AZs in each account.

I built a whole methodology for mapping this out across dozens of AWS accounts, and built lookup tables for our internal infrastructure… and then AWS added the zone ID to AZ metadata so that we could just look it up directly instead.

mpyne• 11 months ago

Presumably it would help ensure that everyone selecting us-east-1a in their base configs didn't actually all land in the same AZ.

zbentley• 11 months ago

It was for spreading load out. If someone was managing resources in a bunch of accounts and always defaulted to, say, 1b, AWS randomized what AZs corresponded to what datacenter segments to avoid hot spots.

The canonical AZ naming was provided because, I bet, they realized that the users who needed canonical AZ identifiers were rarely the same users that were causing hot spots via always picking the same AZ.

Twirrim3• 11 months ago

Almost everyone went with 1a, every time. It causes significant issues for all sorts of reasons, especially considering the latency target for network connections between data centres in an AD

the8472• 11 months ago

As of very recently, you can also force EC2 instances to stop or terminate without waiting for a clean shutdown or a ridiculous timeout

Not true for GPU instances, they're stuck 5 minutes in a stopping state because they run some GPU health checks.

latchkey• 11 months ago

For our VM solution, we get around this by hot staging VM's. As soon as one customer stops theirs, we reset everything and start it up again. To the end user, our compute seems to be instantly available. Unless of course, we run out.

biimugan• 11 months ago

You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.

From my understanding, I don't think this is completely accurate. But, to be fair, AWS doesn't really document this very well.

From my (informal) conversations with AWS engineers a few months ago, it works approximately like this (modulo some details I'm sure the engineers didn't really want to share):

S3 requests scale based on something called a 'partition'. Partitions form automatically based on the smallest common prefixes among objects in your bucket, and how many requests objects with that prefix receive. And the bucket starts out with a single partition.

So as an example, if you have a bucket with objects "2025-08-20/foo.txt" and "2025-08-19/foo.txt", the smallest common prefix is "2" (or maybe it considers the root as the generator partition, I don't actually know). (As a reminder, a / in an object key has no special significance in S3 -- it's just another character. There are no "sub-directories"). Therefore a partition forms based on that prefix. You start with a single partition.

Now if the object "2025-08-20/foo.txt" suddenly receives a ton of requests, what you'll see happen is S3 throttle those requests for approximately 30-60 minutes. That's the amount of time it takes for a new partition to form. In this case, the smallest common prefix for "2025-08-20/foo.txt" is "2025-08-2". So a 2nd partition forms for that prefix. (Again, the details here may not be fully accurate, but this is the example conveyed to me). Once the partition forms, you're good to go.

But the key issue here with the above situation is you have to wait for that warm up time. So if you have some workload generating or reading a ton of small objects, that workload may get throttled for a non-trivial amount of time until partitions can form. If the workload is sensitive to multi-minute latency, then that's basically an outage condition.

The way around this is that you can submit an AWS support ticket and have them pre-generate partitions for you before your workload actually goes live. Or you could simulate load to generate the partitions. But obviously, neither of these is ideal. Ideally, you should just really not try and store billions of tiny objects and expect unlimited scalability and no latency. For example, you could use some kind of caching layer in front of S3.

cmcarthur• 11 months ago

This is my understanding too, and this is particularly problematic for workloads that are read/write heavy on very recent data. When partitioning by a date or by an auto-incrementing id, you still run into the same issue.

Ex: your prefix is /id=12345. S3, under the hood, generates partitions named `/id=` and `/id=1`. Now, your id rolls over to `/id=20000`. All read/write activity on `/id=2xxxx` falls back to the original partition. Now, on rollover, you end up with read contention.

For any high-throughput workloads with unevenly distributed reads, you are best off using some element of randomness, or some evenly distributed partition key, at the root of your path.

kentm• 11 months ago

Yep, this is still a thing. In the past year I’ve been throttled due to hot partitions. They’ve improved the partitioning so you hit it less, but if you scale too fast you will get limited.

Hit it when building an iceberg Lakehouse using pre existing data. Using object prefixes fixed the issue.

danpalmer• 11 months ago

Lots of this seems to boil down to: AWS shipped something that was barely usable, but then iterated.

That's a reasonable approach, but the fact this post exists shows that this practice is a reputational risk. By all means do this if you think it's the right thing to do, but be aware that first impressions matter and will stick for a long time.

abullinan• 11 months ago

This article was a relief. I’m always a tiny bit worried Amazon will change some thing drastically and I’ll have to migrate. I’ve had an ec2 instance running since 2013. It requires effectively zero maintenance. So I am glad there were no surprises in this article. Thanks OP.

romperstomper• 11 months ago

DynamoDB You can now have empty fields

When this was changed? I think this is still an issue, I've had some such errors quite recently.

bluecassette• 11 months ago

18th May 2020

DynamoDB now supports empty values for non-key String and Binary attributes in DynamoDB tables. Empty value support gives you greater flexibility to use attributes for a broader set of use cases without having to transform such attributes before sending them to DynamoDB. List, Map, and Set data types also support empty String and Binary values.

https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

bambax• 11 months ago

Self-hosting is the future. If you have small-to-modetate traffic it's extremely easy to do and very cheap.

Then you can self-cloud. Several startips are in this space. It gets you the best of both worlds: scaling, freedom, cost-control.

And no marketing jargon that you need to learn, and then unlearn!

albert_e• 11 months ago

A "Catch me up" on AWS (and for that matter other large platforms) would be very useful for many folks.

Ideally it should be a stream of important updates that can be interactively filtered by time-range. For example, if I have not been actively consuming AWS updates firehose for last 18 months, I should be able to "summarize" that length of updates.

Why this is not already a feature of "What's New" section of AWS and other platforms -- I dont know. Waiting to be built -- either by OEM or by the Community.

ash_091• 11 months ago

I played a lot of DOTA2 in the past and I've often thought that big tech could learn something from Valve's patch notes. Especially in the context of process changes, stuff you should know, etc. Expecting folk to read a series of lengthy emails/blog posts to stay up to date is unrealistic.

selinkocalar• 11 months ago

Every AWS update can potentially affect your SOC 2 or HIPAA compliance posture. I've seen companies fail audits because they assumed their security configurations were still current.

The cloud moves fast. Compliance processes need to keep up. Manual annual reviews aren't enough when your infrastructure is changing constantly.

This is also why we built automated compliance monitoring - because what worked last quarter might not work today.

AWS in 2025: Stuff you think you know that's now wrong

Related Stories