SRE Doesn’t Scale

https://bravenewgeek.com/sre-doesnt-scale/

by kiyanwang • 5 years ago

162 111 5 years ago

karmakaze• 5 years ago

To expand on the brief title

Google [...] says the SRE model [...] does not scale with microservices. Instead, they go on to describe a more tractable, framework-oriented model to address this through things like codified best practices, reusable solutions, standardization of tools and patterns

Basically anyone planning on microservices should define and monitor bounds on which frameworks, tools and diversity of design patterns in use. Good advice at any scale.

mbesto• 5 years ago

Isn't this more of a comment about microservices than it is about SRE? It reads to me like "once you hit a number of microservices it ends up looking like a monolith":

http://highscalability.com/blog/2020/4/8/one-team-at-uber-is...

iamstupidsimple• 5 years ago

Forgive me but aren't 'macroservices' just... services? I don't see the difference.

steveBK123• 5 years ago

unless its a gigaservice

wara23arish4• 5 years ago

dumb question time but what exactly makes something a micro service.

Is the separation of a specific functionality from a wider array of functions to its own vm make it a microservice?

When does something stop being a microservice i guess?

forty• 5 years ago

I remember asking a candidate whether they were doing microservices at her current job.

She answered "I don't know if we have microservices, but we do have services that don't do much"

It's since then that's my definition of a microservice :)

thecleaner• 5 years ago

My definition is separation of infrastructure and deployment cycles. Everything that always in one deployment is one service or stuff thats part of your code-base is definitely not a different service.

notyourday• 5 years ago

dumb question time but what exactly makes something a micro service.

This leftpad as a service, over HTTPS

igetspam2• 5 years ago

It stops being a microservice when a developer starts saying, "oh! We can do X in service Y too! It already does ${similar work} and reads/writes from/to ${data source}, so why not?"

The intended model is to do one thing, thus enabling surgical changes to functionality without having to rebuild everything. As long as you stick to your API contracts, you can muck around with the internals without effecting anything else.

chippiewill• 5 years ago

I think it's the same line of thinking as RISC and CISC (reduced/complex instruction set computer). There's no such thing as a RISC machine or a CISC machine, all computer ISAs live on a spectrum that can be described as RISC-like or CISC-like.

There's no such thing as a micro service or a macro service (in terms of there being a perfect delineation). They're simply terms to describe certain design goals. Any given service might have several qualities that can be described as befitting a macro or a micro service.

It's all too easy to fall into the no true scotsman trap.

jayd162• 5 years ago

I would say its best to only think about any deployable thing as a service. "Microservices" is just the idea that those things do not have a minimum size.

Essentially, "is this enough functionality to make a new service?" is the wrong question.

rmbyrro2• 5 years ago

A microservice is deployed independently, has its own hardware and can scale independently from the rest of the system.

If there are multiple services and they all share hardware, they're not really microservices.

snug• 5 years ago

I like to think of it more of a separation of components needed to do the job.

Need a database, use a database as it's own service, don't couple the database to the app, so if you need to scale the app horizontally you can add more replicas. If you need to scale the database vertically, you can scale the database vertically. If these are coupled, you scale only whats needed.

On top of that use the language or tool that is meant for the job. If you need something written in go, build your service with go, if you need nodejs, do it with node, etc.

Microservices don't need to be micro or small

willcipriano• 5 years ago

It's always microservices, whatever you build. You just decide what kind of IPC you use between some of the pieces.

pram• 5 years ago

“Hiring experienced, qualified SREs is difficult and costly. Despite enormous effort from the recruiting organization, there are never enough SREs to support all the services that need their expertise.”

Uh huh. Maybe what isn't scaling is their onerous recruiting filters.

anonporridge• 5 years ago

A different element of this is that software professionals are often paid so well that a lot of people can take early retirement relatively easily if they're good about saving and investing. And the FIRE movement is only growing.

I sometimes wonder if the software world has relatively greater 'leakage' via early retirement than other fields, creating a constant problem of not enough highly experienced people who remain stuck as wage slaves throughout a 40 year career.

bostonsre• 5 years ago

It's good to be an sre. From my experience, there is high demand, low supply and schools don't really teach for sre roles so the pipeline stays small.

outworlder• 5 years ago

Yes.

And then there's the unrealistic demands from companies that didn't even understand what DevOps was supposed to be, and are misunderstanding SRE even more. Many companies still treats this the same way they did "ops".

zaptheimpaler4• 5 years ago

Listen, if you can't write a working text justification algorithm in 45 minutes a decade after college, you have no business doing completely unrelated SRE stuff like monitoring services or scaling databases. Everybody knows leetcode is the key to software. If you can't do that, maybe you just don't have the right IQ to build the next chat app @ Google.

GauntletWizard3• 5 years ago

This but unironically. If you can't do a simple programming task, (we can argue over what text justification is, because "left-pad" could qualify, as could a full TrueType renderer) you don't belong in SRE - Much of the point is to have cross-functional people, who can interface with developers on their level, and do the math and grunt work around operating a database cluster.

goodpoint• 5 years ago

This is absurd. A lot of companies do interviews that make sense and are not focused on some obscure bitshift puzzles.

You need to be a strong developer to work efficiently as an SRE.

klodolph3• 5 years ago

It sounds like the lesson here is that tacking on SRE to an out-of-control development process, churning out new services by the boatload, doesn't scale.

This is caused by the typical attitude software companies have towards development. The most common model for software development is simple... rush to push out new features to the market, and pay the costs later--and then everyone is balking at the costs.

The solution is kind of brilliant, IMO. You don't have to pay technical debt on projects if you shut them down and delete the code from your repository. Migrate to industry-standard solutions. Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions.

The SREs can help you with that, but they can't help with out-of-control development. As your code base gets larger, the cost of supporting that code base gets larger too. The difficulty of scaling your SRE to match development reflects your out-of-control development process, not a problem with SRE. Keep the costs under control by keeping your code base under control.

lykr0n3• 5 years ago

Yep. SRE is not a substitute for high level, overarching architects and designers.

One pattern I see is that, as the company grows the development gets split into different product groups which will organically diverge unless there is rigid enforcement of design patterns. In some places, SRE does this implicitly because they will only support X, Y, or Z but in others each product group will have their own group of SREs.

There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. If you don't have this, you'll wind up with long migrations and legacy stuff that never seems to go away.

colechristensen• 5 years ago

When you get very large you need both enterprise wide SREs that are responsible for consulting and approving architecture for reliability purposes AND localized per service or other small breakdown SREs to support small sets of services. How you break this down is tricky and there is definitely overlap.

Crappy attitudes towards support and reliability don’t scale, what you can get away with a few people keeping things barely held together stops working as you grow.

lmm2• 5 years ago

I don't find having high-level architects to be a good pattern. They can make mistakes like anyone else; indeed having people who are no longer day-to-day coding make decisions that they don't feel the effects of makes wrong decisions more likely.

SRE exists to support product functions and like everything else should be attached to and understood in terms of those functions. Yes, every product group probably should have their own SREs, so that product group can own its whole lifecycle. Yes, different groups will do their own things and there will be mismatches and duplicated effort. That's less bad than the alternative.

mavelikara• 5 years ago

There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct.

How many times have you been in the "everyone else" camp and was course corrected? How did those efforts work out for the firm in the long run? Any experiences to share that could be useful for the community?

rektide2• 5 years ago

My read on the article was that much more was related to each team being on their own to set up & drive their pipelines, operate their own services, and there being a lack of commonality/shared experience.

A vast number of the software engineers don't get the ops (running software) stuff hardly at all & half of them can sort of play along, hack stuff into place. The engineers on product teams who do know how to do things meanwhile don't get all the constraints, best practices, ideas that other various DevOps folk have done & have their own wants/desires/expected ways of doing things, so they end up creating their own very unique sub-ways of doing things within the org. None of these practices converge on regularity or consistency with what DevOps machinery ends up being built.

What we do have often is just a random pile of containers and scripts that a couple people sort of know decently & everyone else suffers through & survives within. Almost never does it look like any other company's devops kitchen.

SRE doesn't scale because it's an every now and then thing, and few people notice or care about the difference between a well-built corporate citizen that runs well & is monitored & operated according to whatever the in-power SRE cabal wants. People start to care only if things are going bad, either via services not building/integrating/deploying/running as well as they should, or from too much confoundedness/general head scratching by either the SRE or regular engineers. SRE is not a priority, it's not practiced regular, it's only an every-now-and-then thing, so we don't have the chance to get good, to institutionalize the right ways of doing things. That's what the articles is discussing. Not the rest of the everyday normal software development rushing-bedlam you describe.

klodolph3• 5 years ago

SRE doesn't scale because it's an every now and then thing, ...

That's the part that doesn't scale... tacking on SRE at the end, or doing it every now and then. The reason people don't care about the software being a "well-built corporate citizen" is because they care more about shipping features. If you have an SRE team that will say "no" to you when you try to ship new stuff, you'll eventually figure out a way to build new things in a way that the SRE team will say "yes". When I say "no", that could be a hard pushback like "no, that's not getting shipped" or it could be an answer like, "no, the SRE team will not support that, yet."

These kind of decisions need to be made at a high level, because everyone in the institution is typically operating with the wrong incentives. That's why you end up with a random pile of containers and scripts. It doesn't have to end up that way, even when you have microservices.

That's what the articles is discussing. Not the rest of the everyday normal software development rushing-bedlam you describe.

I disagree with the article, so necessarily there are going to be differences between what I'm saying and what the article is saying.

rhizome3• 5 years ago

Everything in your second paragraph points to companies that either through hiring, firing, or attrition have successfully excised all real systems expertise from the entire company. This isn't the fault of any ops methodology, so let's maybe call it "shrug-ops."

few people notice or care about the difference between a well-built corporate citizen that runs well & is monitored & operated according to whatever the in-power SRE cabal wants

QED

bcrosby95• 5 years ago

Funny, I read it differently. They talk about frameworks, libraries, and best practices.

Effectively, they're talking about standardization across your teams/services so they don't fuck things up. Essentially, you're taking away some of the purported freedoms of microservices (complete independence - eg I can write this service in brainfuck if I want!) and reigning it in a bit so you don't build a pile of trash.

klodolph3• 5 years ago

I think of that kind of standardization kind of like deleting code. Stuff like, "We are deprecating support for Python in SRE, no new projects may be shipped in Python."

jayd162• 5 years ago

Now all your trash sorta looks the same.

emmelaich• 5 years ago

It must be possible for SREs to refuse a service that doesn't meet certain criteria.

AFAIK this doesn't happen anywhere. Although I believe G has something like this.

pkhuong• 5 years ago

Google SREs definitely have that stick.

dilyevsky• 5 years ago

In order to make this possible you need an entirely separate chain of command for SRE like what google has or very influential SREs either of which is exceedingly rare so not surprised op thinks sre doesn’t scale

jedmeyers• 5 years ago

I believe G has something like this

It does. And until the service meets the criteria, SWEs on the project are actually partially playing the SRE roles under the guidance of the SRE org.

eikenberry• 5 years ago

Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions.

Be very careful following this advice. Only use an off-the-shelf if it is pretty close to what you need or if you can change the need to fit the tool. Trying to shoehorn processes (or whatever) that don't fit a tool to a tool often end up being a lot worse than a custom coded solution. I've seen this multiple times at enterprises that preferred vendor solutions as you suggest and they were worse train wrecks than places that used more custom solutions.

geodel2• 5 years ago

Manager: We need API gateway and Service Mesh in Kubernetes for our service

Engineer: I have come up with solution that wouldn't need all these frameworks.

Manager: Good. See how soon you can integrate with API Gateway and Service Mesh.

Heard at friend's workplace.

pc86• 5 years ago

The key in using off-the-shelf software, especially if you're not a capital-E Enterprise that can dictate new features, is to be willing to change your business processes so that they match the tools you're buying. Part of the purchasing process for SMBs should be (but almost never is), evaluating current processes, and which parts of those processes are arbitrary and/or can be changed, then finding COTS software that fits all the must-have/can't-change processes. Then you adjust the last few remaining things to fit the software.

CobrastanJorji• 5 years ago

The best way I've heard this put to me was this: "Your business has some special selling point where it stands out, the reason it's unique and people should buy your thing. Your business has hundreds of bog standard things it needs to do. You only have so much development time; spend it on the stuff that makes your company important and not on the stuff that already has perfectly good solutions sitting on the shelf."

Aeolun• 5 years ago

See, basically any SAP integration.

davedx• 5 years ago

I worked at one place that had the worst of both worlds: SAP Hybris, customized to the nth degree with rushed features.

It was pretty awful and had significant, material negative impacts on revenue due to all the operational issues.

pjc502• 5 years ago

This is making me realise how strong the analogy between Conway's Law and Coase's Theory of the Firm is.

(recap: https://en.wikipedia.org/wiki/Theory_of_the_firm especially "Why is the boundary between firms and the market located exactly there with relation to size and output variety? Which transactions are performed internally and which are negotiated on the market?")

The answer involves looking at informational and coordination costs. Microservices are extremely Coasian: rather than have a part of the software/company under direct, traditional command-and-control management, it's kept at arms length and interacted with via a software contract.

Using microservices theoretically reduces the coordination costs between parts of the team, but in practice risks just shovels them around. After all, the easiest way to reduce a cost is to dump the externality on to someone else. If you were deploying a monolith, it would obviously have only one set of SRE practices; with microservices, you end up duplicating those and introducing costly variation.

igetspam2• 5 years ago

Google enforces standards and opinions around things like programming languages, instrumentation and metrics, logging, and control systems surrounding traffic and load management.

I think the author read this as more of a problem than a solution. This concept is supported by the DevOps model too. Your infrastructure is just as much a part of your product and the teams providing the infrastructure just as responsible for the service levels and API contracts as any customer facing product team.

gautamdivgi• 5 years ago

And that move to microservices—in combination with cloud—unleashes a whole new level of autonomy and empowerment for developers who, often coming from a more restrictive ops-controlled environment on prem, introduce all sorts of new programming languages, compute platforms, databases, and other technologies.

You need standards, without that SRE is pointless. Everything needs a standard method of monitoring. As an e.g. - stick to Java/Spring Boot, MariaDB and K8S. That will generally cover 85% of your use cases.

The automation and advantage of SRE is derived through standards and familiarity with the tool chain.

slaymaker1907• 5 years ago

I kind of disagree on the Cambrian explosion of languages being a bad thing. While it has downsides, it has a massive upside of discouraging stuff like crazy custom build pipelines. Big companies love to burn money and time by reinventing everything. Bigco isn't going to try and implement their own build system if they need to support a dozen languages.

high_5• 5 years ago

Microservices have a marvelous long-term potential to trash the service with a myriad of legacy and obsolete languages instead of just one. So you just have to run faster to stay in the same place.

kodah2• 5 years ago

The thing that made Google SRE scale was in the fine print and covered in this article. SRE is a premium service, and in order to get the premium service you had to meet some criteria to show you weren't about to turn highly paid, highly knowledgeable folks into professional firefighters.

During my time as a SRE at multiple organizations what I witnessed was executives that owned Operations teams renaming the team, paying for a Python course, and calling it "change". Operations Engineers have a culture unto themselves that will not be broken in a years time. A culture of firefighting and heroism as opposed to proactivity and dull standards. They're rarely, if ever, programmers which was a prerequisite for a SRE-SE and SRE-SWE. You don't need to be a master in logic to know that if you dress your old pig up with lipstick and a new title that it's still a pig.

I agree with the article: read the SRE book and understand why they did what they did, and if you hire SRE-SE or SRE-SWE types, don't colocate them with Operations types or make them a 1:1 map with your dev teams that haven't earned them.

Cthulhu_2• 5 years ago

I heard about that; amongst other things, if the amount of times SRE gets pinged because of a software fault exceeds X amount, they basically give the pager back to the team that built the software. They aren't taking responsibility for reliability if the software is not reliable.

I really appreciated working with a team of SRE types, it gave me a newfound appreciation for quality in software development. I remember one instance where a colleague in my team went up to the 'ops' team and wanted SSH access to a production server to check on some settings (environment variables, I believe). He thought it was a trivial thing, you know, "just lemme have a peek" kinda thing, but the ops team flat-out told him no, if he needs to print env vars, he can do it in his own code - we did continuous releases, a patch could be deployed within minutes if passing all the checks. I loved that the ops team had the mandate from higher up to say no to requests like this, and I found them a lot more professional than the software developers, including myself.

bytelines• 5 years ago

the ops team flat-out told him no, if he needs to print env vars, he can do it in his own code

This is still the wrong attitude from a "productization" of infrastructure perspective. Configuring the environment a program runs in is a core responsibility of infrastructure. So is being able to query it. Build that infrastructure product feature.

Or select a platform that makes this trivial, e.g kubernetes.

kodah2• 5 years ago

Ideally what that developer was wanting should be a function of the platform. That's likely not in an operations teams scope because they're mostly just fire fighters. Also ideally in a situation like SOX where devs can't have access to production their preproduction environments share the same interface that production has and if their values differ that much is documented.

ilrwbwrkhv• 5 years ago

Microservices should be used as an optimization. Not as a starting point.

nateberkopec• 5 years ago

At some point, it feels like every company is basically re implementing a kind of internal Heroku. I feel like there’s a billion dollar business in putting that on rails for medium to large enterprises.

jacques_chester• 5 years ago

There are many products that aim to fill this niche. I worked on one for years (Cloud Foundry). There is a lot of money to be made, but it's surprisingly stony ground.

Oblig. self-promoted thread: https://twitter.com/jacques_chester/status/14357388253476454...

cletus• 5 years ago

It actually sounds more like what I'd expect: microservices don't scale.

I've never been a fan of microservices for this simple reason: every boundary, every interface, every connection is friction and a potential source of bugs, performance issues and problems.

It's now been several years since I worked at Google so I can't speak to the current state of things but when I was there, monoliths ruled the roost.

It seems possible there was some experimentation with microservices given the rise of Docker and Kubernetes in the last few years. I'd be surprised if this made serious inroads into high-traffic core services.

It's also worth mentioning that generally Google has (had?) SRE teams rather than SREs embedded on SWE teams. You'd never have one SRE supporting a service as there'd have to be enough to maintain a healthy oncall rotation (typically 8-12).

I can't help but feel like the author is projecting his own experiences onto what Google has said here.

jatins• 5 years ago

I previously worked in a BigCo that did microservices and currently work at BigCo that leans towards monoliths.

It's completely true that microservices makes development _very_ hard. Just being able to run things locally was very hard since instead of understanding how to build/run one service, you had to figure our how to build/run N services. Testing was another major headache.

Monoliths help with technical aspects but organizationally they make your life hell and can severely limit your ability to move fast.

Cthulhu_2• 5 years ago

Microservices are a cargo cult thing. It tries to solve a problem that 99% of companies that apply it wish they had.

75dvtwin2• 5 years ago

Hiring SREs is costly...

Hiring narrowly focused people with significant pre-existing experience -- Is always costly.

... and I also think, it is practically always a wrong strategy...

I am fundamentally against micro-managing labor pools by federal government.

However, there are need to be economic incentives (including immigration policies) -- that make it more difficult for employers to hire for 'tool-centric' positions, unless those tools are very expensive physical devices (eg telescopes, quantum computers, etc).

The incentives need to direct employers at training on the job (not at after-work online classes).

When I see on HN's hire threads -- 'if you have Azure experience -- you will get on top of the pile' -- I cringe.

This is absolutely ridiculous. Same pretty much with Ruby/Php Rust/C++ Haskell/Scala modalities of the same problem.

Yes hire people who understand process/idioms/patters, but invest in the f..ing training -- if you need somebody 'yesterday' to help, get consultants -- and have them help and at the same time hire for full time roles -- and train your internal stuff (possibly, even, have them learn from the consultants).

Afraid of people leaving after acquiring the highly thought-out skills ?

-- implement meaningful compensation/retention policies that reflect effort that you spend on training, and risks that you are taking if a person who was just trained -- leaves.

winrid3• 5 years ago

What you are saying is generalists are cheaper to hire and that you can train them. That's true.

That's why, as a seller (worker) you should try to specialize in at least one area to sell yourself better.

75dvtwin2• 5 years ago

That's part of it.

I am also saying that there needs to be a system of incentives, that then, produces, organizational/hiring practices that favor non-tool-specific employment process.

pjmlp• 5 years ago

Speaking from experience, it sells better to bring along special soft skills in the package, than yet another technology stack.

Too many coders, generalists or not, cannot engange properly talking to something that isn't a computer.

Those that manage to merge coding skills, with UI/UX, marketing, understanding the customer point of view, already have an upper hand.

stefanve• 5 years ago

To me SRE + Cloud means SRE is responsible for the cloud foundation/ platform but the software teams should be DevOps teams and they are responsible for the (micro)services they create. SRE helps out by providing the tools, libs, dashboard , general principles etc.

So SRE in responsible for the health of Kubernetes, networking etc and providing libs, support and infra but the DevOps teams are responsible for the health of the services/ software that is running on the cloud platform.

A DevOPs teams should not consist of developers and one ops person but the team as a whole is responsible for the stuff they create. They are on call just like SRE is, if something go's down SRE + DevOps work together to fix it

KaiserPro• 5 years ago

this person typically just becomes the “whipping boy” for the team.

That sounds like a culture problem to me.

SRE is just a fancy name for a class of people that make stuff more reliable. However it appears the reader of this book forgot about the other part: making stuff easy to use.

Everywhere I have worked in the last 10 years has had an embedded sysadmin/devop/SRE/PE of some sort. They are there to take the raw, difficult to use tools and adapt them so that developers are able to deploy without hand holding.

This means evangelising metrics, showing how to manage alerting, coordinating standards and leading incident responses.

The symptoms the writer describes are of a dysfunctional dev environment. Ideally your SRE should be able to rotate teams quickly. That means standards. Once a team is setup and running, they shouldn't really need ongoing SRE functions, unless they are making massive changes.

This can only happen if developers are re-using tools, rather than making new wheels out of space age materials and getting bored and moving onto new things.

SRE Doesn’t Scale

Related Stories