Modern Applications at AWS

https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html

75 42 7 years ago

DVassallo7• 7 years ago

To succeed in using application development to increase agility and innovation speed, organizations must adopt five elements, in any order: microservices; purpose-built databases; automated software release pipelines; a serverless operational model; and automated, continuous security.

When I worked at AWS, it used to take us months to change the position of a button in the UI. Things that our competitors did in hours and days, we did in months and years.

I'd say that as a general heuristic, if you want "to increase agility and innovation speed" do the opposite of what Amazon does.

jayd162• 7 years ago

This is because feature verticals build their own UI, right? Seems like there should be a solution to this.

DVassallo7• 7 years ago

Changing a button was just one example. How about this? https://twitter.com/dvassallo/status/1166700686467166208

Ididntdothis• 7 years ago

I think from a certain size and number of users any innocent looking change can cause a lot of unintended consequences. I always wonder how Microsoft or Apple make any changes to their OS considering the large number of legacy code that should keep running.

solarkraft• 7 years ago

The considerations part of developing an operating system can be viewed pretty well in the mailing lists of Linux Distributions (and other OSes, but less so because of the fewer stake holders).

The decision making processes are influenced by different cultures: Debian aims for a very stable system with no breakages at least within a major version, Windows aims for close to endless backwards compatibility (hence the bloat), while Apple designs its systems with more of a "move fast" attitude (hence the quick proliferation of new features).

influx• 7 years ago

When I worked there we had continuous deployment with pipelines straight into Lambda in native AWS. There’s so many different teams at Amazon I wouldn’t take anyone’s advice that claim they can speak for all of development there.

DVassallo7• 7 years ago

Agility is not about deployment speed. It's about being able to react quickly and easily to new information (customer complaints, bugs, new opportunities, etc). You don't need a perspective from the inside to make the claim that AWS's evolution rate of existing products is weak. Innovation speed might be high horizontally (launching a ton of new services), but not vertically (improving existing ones).

tnolet• 7 years ago

Ok, so how do you explain the blistering speed at which AWS (not Amazon) releases new services, updates and features.

Not being smarty pants here; It truly boggles my mind how fast they move.

DVassallo7• 7 years ago

AWS is great at horizontal innovation (launching a lot of brand new stuff). It's vertical innovation where they struggle (improving existing products). Software architecture helps the latter much more than the former.

paulddraper2• 7 years ago

Do they move fast? I'm still waiting for Aurora to offer PostgreSQL 11 from last October.

Hell, I'm still waiting for AWS to offer an upgrade path to PostgreSQL 10 from October 2017.

segmondy• 7 years ago

An interface even if it's just a user interface is a contract. You can't just go changing it because you want to. Many immature companies do this and think it's innovation. It's not, all your users will have to relearn the new UI change.

DVassallo7• 7 years ago

That attitude is what leads to the AWS console.

tptacek• 7 years ago

I'm guessing the person you're replying to understands the implication of changing a UI. Maybe, instead of telling them why they're wrong, you could ask them if resistance to UI changes was the problem, or something else was? They were there. Were you?

cle2• 7 years ago

I'd say that as a general heuristic, if you want "to increase agility and innovation speed" do the opposite of what Amazon does.

That’s a pretty vague and useless heuristic. Since you worked at AWS, maybe you could answer why AWS moved slowly on releasing features for existing services? An Amazon-y “five whys” analysis would probably yield a lot more specific and useful heuristics.

DVassallo7• 7 years ago

Don’t take it too literally. My point was that AWS doesn’t show agility and rapid innovation on existing products, so when it tries to tell you how to do that, maybe take it with a grain of salt.

ffflush• 7 years ago

When I worked at AWS, it used to take us months to change the position of a button in the UI. Things that our competitors did in hours and days, we did in months and years.

From what I understand, the culture in a large company varies from team to team. Was it your experience that all the teams were that slow?

Are you claiming the whole of AWS is that way or that since your team was that way we should disregard any advice about AWS?

If I invert all the advice from the post, would that be better as a development practice? A large monolith, backed by an off the shelf database where there is a central security team looking after every line of code?

DVassallo7• 7 years ago

It's a company-wide issue, and it can be observed from the outside too. Existing products improve at a rate that is significantly slower than competitors (with a few exceptions). And I'm not referring to feature creep, but fixing obvious gaps and issues that the company would have been aware of for years (or more than a decade in some cases).

bg24• 7 years ago

Your last statement is being blunt on Amazon.

Customers come first in AWS culture. With a massive user base, they need to care about not breaking existing customer applications/integrations.

If AWS did not evolve its software processes, they would have already fallen apart by now. The teams move reasonably well for the size of a big company.

twic• 7 years ago

Customers come first in AWS culture

As a customer of AWS, it never seemed that way.

SkyPuncher• 7 years ago

Are you comparing to competitors of your size?

I ran an enterprise-backed startup. We built way faster than our enterprise backers when it was our own name. If something carried an association or backing of the enterprise, it took at least 10 times longer. They had a reputation to uphold and needed to ensure everything reflected that.

laluser2• 7 years ago

For large corporations like Amazon, there are huge opportunities to move to Lambda. Back when I worked there, teams would over-provision their services like crazy. Most of it to scale for the holiday season. Most teams would not give that extra capacity back.

vinay_ys• 7 years ago

how long ago was this?

laluser2• 7 years ago

This was < 2 years ago

twblalock3• 7 years ago

The other way of dealing with this is to put normal services in an autoscaling environment like Kubernetes. You don't have to go as far as Lambda to get the benefits.

Moving to autoscaling is really more of a cultural problem than a technical one. If teams are used to being able to reserve excess capacity they will be wary of autoscaling -- what if they try to scale but there is not capacity to do so because other teams used it up? People need to trust that the system will give them the capacity they need or they will continue to overprovision.

redact207• 7 years ago

It's sad to see this approach being pushed so hard these days. I believe AWS recommends these things so your app becomes so deeply entrenched in all of their services that you've locked yourself into them forever.

Unfortunately I see a lot of mid and senior devs try to propose these architectures during interviews and when asked questions on integration testing, local environment testing, enforcing contracts, breaking changes, logical refactoring a lot of it falls apart.

There's a lot of pressure for those guys to deliver these sort of highly complex systems and all the DevOps that surround it when they enter a new company or project. Rarely is the project scope or team size considered and huge amounts of time are wasted implementing the bones since they skip the whole monolith stage.

Precious few companies are at the size where they can keep an entire Dev team working on a shopping cart component in perpetuity. For most it's just something that gets worked on for a sprint or two.

AWS made a fundamental assumption in this that monoliths and big balls of mud are the same thing. Monoliths can and should be architected with internal logical separation of concerns and loose coupling. Domain driven design helps achieve that. The other assumption that microservices is a fix to this isn't, because the same poor design can be written but now with a network in between everything.

I think this advice is worth a read, but it's not law. The most pragmatic approach to microservices is to start with a monolith and shave off the intensive parts to separate services as you need to.

mattbillenstein• 7 years ago

Yes! Been preaching this for years!

I've untangled more than a couple small eng teams (~ < 20 devs) that were trying to do microservices using this recipe on aws and just horrendously failing at it.

A well-architeched monolith will really go a long way - generally perhaps easily up to 100 or more devs before thinking about microservices.

shantly• 7 years ago

- If it doesn't have very different scaling needs from the rest of the application, probably don't make it a microservice.

- If it isn't something you could plausibly imagine using as a 3rd party service (emailer/sms/push messages, authentication/authorization, payments, image processing, et c.) probably don't make it a microservice.

- If you don't have at least two applications at least in development that need the same service/functionality, probably don't make a microservice.

- If the rest of your app will completely fall over if this service fails, probably don't make it a microservice.

- Do write anything that resembles the above, but that you don't actually need to make a microservice yet, as a library with totally decoupled deps from the rest of your program.

twblalock3• 7 years ago

The most pragmatic approach to microservices is to start with a monolith and shave off the intensive parts to separate services as you need to.

I've seen that attempted several times but I've never seen it succeed even after several years of shaving things off here and there.

The problem with this approach is the same problem that causes all technical debt -- cleaning up technical debt takes time away from developing new features, which is the company's highest priority. Adding more engineers to the team to help tackle debt won't help because management will see that as an opportunity to get more new features.

Now that the dust has settled, it's become clear that the real advantage of microservices is to decouple teams.

Shared codebases become harder to work with as the number of engineers increases, and that hurts developer productivity.

Don't build a monolithic service for the whole company -- have every team manage their own services, on their own development and release schedule, and their own production scale. Their dependencies on other teams should be based on APIs, not code sharing. Whether an individual team chooses a monolith or microservices doesn't really matter.

cle2• 7 years ago

It isn’t as black and white as you are portraying it. You can go too far in the microservices direction, and you can go too far towards a monolith. There’s a sweet spot. I currently think that this sweet spot is around one service per “team” (around no more than 10 people). This lets teams operate and deploy their services independently, while still retaining some of the cohesiveness of a monolith. It also lets the company “refactor” engineering teams by moving service ownership around. There are definitely legitimate reasons for a team to own multiple services though (organizational changes, differences in scale, security constraints, etc.).

Monoliths shared across dozens of people are operational nightmares because there’s not a strong sense of operational ownership, so you often end up with a bunch of process and bureaucratic overhead to compensate.

Your advice to start with a monolith is too simplistic. If you know how your service will scale and your engineering teams will grow, then you have enough information a priori to avoid the waste of attempting to split apart a monolith later.

ris• 7 years ago

Yes, it's awfully fun that they're using the term "Modern" applications to describe applications which I wouldn't touch with a bargepole. Guess I'm just not "Modern".

Then again, I was chuckling in the a similar way a few years ago when not using mongodb et al made me not "web scale".

treggle• 7 years ago

Cloud applications have become much much more complex than OS based application development.

Don’t take the path of cloud based development if you are aiming to simplify.

ilovecaching• 7 years ago

I have a request for opinions!

I’ve noticed that software quality never seems to enter into conversations about scaling. Do code hygenie, writing clean code, and good documentation not matter when you’re moving fast and growing like crazy? I’ve noticed the approach seems to be to focus on operations rather than the way the code is written. Use CI, monitoring, organize teams around services, etc. From my friends in the bay however, I hear the code itself is mostly written and band aided over and over again with refactoring and rewriting seen as “wasting time” when you could be writing more band aids or improving operations.

Does the band aid and if it’s not broke don’t fix it and do exactly what you need to fix it approaches really yield the same or better results as approaching software as a craftsmen? How do you convince an AWS engineer that these things are important when what’s in prod is working and a rewrite just seems like pot stirring? Or a refactor isn’t critical work to add more features?

Modern Applications at AWS

Related Stories