How 'DevOps' is Killing the Developer (2014)

https://jeffknupp.com/blog/2014/04/15/how-devops-is-killing-the-developer/

by zippy786 • 11 years ago

52 66 11 years ago

mattkrea4• 11 years ago

One coworker posted this a while back and I just dismissed it as someone afraid of something that they can't handle.

While I understand the point.. I'm also a dev and I love "DevOps". Aren't we all doing this because we love problem solving? How is this not just another problem to tackle? I love having to figure how to automate mundane tasks (again.. this is just code.. not much different than we write elsewhere) to make sure I don't have to manage any systems manually.

/rant

zippy786• 11 years ago

Don't you see tight coupling between two components/roles which may be bad ? Of course, for management it's less money they have to pay so they will always prefer a one man doing everything scenario.

mattkrea4• 11 years ago

I see that I can't trust what is the traditional IT department to understand how my application works so I strongly prefer shipping Docker containers or using something like Elastic Beanstalk if that is not available.

nstart2• 11 years ago

It's actually a complex debate. We as developers naturally gravitate to a role like this because it's in our nature to believe that we can solve all the tech related problems. And all other problems, we believe can be solved with tech and therefore we believe we can solve everything. Ok. That's an exaggeration but there's quite a large shred of reality in that statement. Being fullstack/devops/<insert fancy term> satisfies that craving to jump on to new problems that we developers have.

And that CAN be a problem. It isn't necessarily always a problem. But think about it. How many of us prefer jumping up to try working on a brand new project. Shiny stuff everywhere. And then everything gets abandoned at the grind and polish stage. The enthusiasm is gone. We crave the next new problem that we can use as an excuse for jumping away from the grind we are in. I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything. Ask a developer (myself included) to evaluate a set of JS frameworks for a particular project which has somewhat specific requirements. Behind most answers (this is based on observation, myself included again) the primary argument will be the developer's preference; not an objective breakdown of what is most suited for the task. We might not see too much of this on HN (I think. I see a lot of intelligent debates here which I learn from all the time) but on a larger scale developers are becoming half baked.

And that's basically the other side of what you pointed out. Problem solving, learning new things is a lot of fun (discovering and configuring vagrant and docker for the first time was incredibly fun). But it possible that the full stack role is churning out people with broad but just somewhat in depth knowledge. Tough call. Question is, is somewhat deep deep enough for most everything?

eropple• 11 years ago

I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything.

I dunno about that. My experience has been more that a team may need one person who's both wide and deep (T-shaped, or maybe just "rectangular"), but the companies I run across generally seem to hire younger developers who do have one or two core competencies--I-shaped, if not just dot-shaped and hoping to train them up.

mattkrea4• 11 years ago

I can agree with that.

Though on the other end of things.. I have a friend that is very into how some languages are optimized for various tasks and he'll mention to me that X language does something with their loops for speed, etc.

I don't have that same low level knowledge of the languages (not as low-level at least; I have some) but at the end of the day X language is slower than those that I prefer for the type of work we do (APIs mostly) so I don't care that X language does all these things internally.. and I don't need to know because it doesn't matter to us (nor our employer) in the end.

So my--hate to use a buzz phrase here but it's appropriate--'get shit done' style tends to work for me and our company because evaluating tools comes down to "does a framework or tool accomplish what we need without too much overhead".

Is this the type of deep knowledge you were talking about?

serge2k3• 11 years ago

The problem I have with DevOps boils down to 2 things.

1. It's really just a dumb buzzword. 2. I really don't like companies dumping more work on devs for the same pay and using a buzzword to justify it. Oh here, you get a pager now because we're DEVOPS! Hire a god damn support staff.

That said, letting people form little kingdoms and toss up walls so that people just end up getting bound up in bureaucracy is also terrible.

slowmovintarget• 11 years ago

Yarg!

Teams do DevOps. It isn't supposed to be an excuse to make developers do everything. It's supposed to combine operations and development on the same team.

This is just like the perversion of agility in development by "Agile."

This, too, shall pass.

phamilton• 11 years ago

I recently was part of a team that properly practiced DevOps and as soon as it started working we had a reorg and it was broken apart.

If the premise of the article is "Developer time is precious, don't make them do grunt work" the I would counter with "Developer momentum is precious, don't let developers get blocked by operations/QA/release."

On our team, we had Ops and QA. But I needed something done and they were already working on something then I could have a quick 15 minute chat after standup about the Puppet change I needed to make and then go do it myself. An hour later I'm done, the change is deployed and I'm moving on to the next task. Now, without an embedded Ops resource on the team, I have to file a ticket and pray someone picks it up within a few days.

Turbots• 11 years ago

I couldn't agree more!

If you've read the Phoenix Project, it clearly states DevOps solves the problem of bottlenecks, which is the single most important aspect in the Product Development lifecycle. You just can't have developers waiting anymore for Ops to do something, or vice versa... Shit needs to go fast and blocking factors need to disappear, or else you sacrifice agility

lowmagnet3• 11 years ago

The way to not get "blocked by QA" is to have automated tests and QA people who can write them as well as dream up scenarios to break your toys. They are also able to do code reviews in your stack as well.

bro-stick• 11 years ago

"Full-stack" is a skillset for founders/first employees and hack-of-all-trades SREs. Later on in a now enterprise shop, deployments and development gets formalized, so it's crucial to have people/service providers whom know the nitty-gritty of the stack components really well... but it would be a distraction to expect a senior Postgres DBA to know client-side JavaScript. The overlap on the interface between sysadmining and development is deployment, where TwelveFactors-like shops can excel and reduce dev-ops turf wars. DevOps is about working together to make the overall service better; not shifting responsibilities around, getting people to take on more hats or renaming sysadmins.

Disclaimer: I've been a (in no particular order): industrial embedded developer, sysadmin, engineering manager, founder, security researcher, consultant and SRE.

myblake• 11 years ago

Yeah, the dentist analogy was really interesting because there's no way in hell you'd have the receptionist and hygienist and dentist just throwing things over a wall to each other and not coordinating carefully on patient care, yet without DevOps that's all too often what happens.

bradhe• 11 years ago

If you're a developer working on a service your primary objective is service delivery. It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom.

atarian• 11 years ago

Understanding the entire lifecycle of a service (from development to production) is a must for a developer. But it should not be the primary objective of the developer to deliver the service; developers should be working on development.

When developers are in charge of delivering a service, they will end up spending most, if not all, of their time debugging operational issues instead of improving the service. This problem gets worse over time as the service will need to be deployed to more places to address scale.

An organization with good technical leadership should know this.

onion2k2• 11 years ago

It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom.

I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand.

There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great.

The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.

MichaelGG• 11 years ago

This works great when it works great. Black boxes everywhere. But then you run into some issue, like, I dunno, your monitoring service is telling clients the site is going down, even though you can't find any evidence of this.

If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks."

Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors."

Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way.

Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.

keyanp• 11 years ago

Interesting read, but I wish it wasn't so inflammatory.

"...release coordinators and the like are at the bottom of the totem pole. Why is it arranged like this? Because each role can do the job of all roles below it if necessary."

I find this to be belittling those who choose to work on a different part of the stack than the author. It seems to suggest there is an innate incompetence that keeps release engineers from functioning as "developers". That couldn't be further from the truth.

Release Engineering @ Mozilla: https://www.youtube.com/watch?v=7j0NDGJVROI

falsedan2• 11 years ago

Release Engineer here too.

FUNNY STORY: we make the developers act as release coordinators, because it distracts us from the work of improving the release process by replacing it with automated software systems.

The top of his hierarchy should be PMs/executive team; betrayed by his own developer ego…

serge2k3• 11 years ago

It's a hierarchy of skills. PM and exec skills are almost entirely orthogonal to tech skills. Putting them in this hierarchy makes zero sense as a result.

serge2k3• 11 years ago

not gonna watch, but developers who work on release tooling are just a category of developer.

The article is inflammatory to a degree, but it does make sense applied to groups rather to individuals. Devs do have to learn a lot of the skills the other groups do, but it doesn't go the other way. There are still obviously valuable skills that a DBA or sysadmin will have and your average dev won't. It doesn't change the fact that, as groups, I'd rather have a group of devs be forced into DBA and Sysadmin tasks than the other way around.

timv• 11 years ago

As well as being inflammatory and belittling, his reasoning in this area also fails to question the status quo, and realise that maybe a form DevOps is a path to a better outcome.

It is the case, that in many organisations, the roles down the bottom of his totem pole are less skilled than the top.

There are point-and-click DBAs and Server Admins and release managers who don't have the base skills to write software. But they're rarely very good at the roles they're doing.

I have a bunch of issues with what a lot of organisations do under the banner of "DevOps", but part of what it is trying to do is bring engineering solutions to those layers "down the totem pole".

I want DBAs that invest in reliable, repeatable, automated processes. And that means I want them to be able to build working software. And similarly with the other roles he mentions.

Now, generally speaking, you'll get more useful work from a system admin who just follows the procedure they were given, than you will for a developer who does likewise. But, if you can do it, you will get plenty of value from hiring people with some engineering skills/training into all those roles.

The counter examples are the system admin who takes 2 weeks to setup a new server because it's all done by hand every time. Or the release team who buys fancy "automation software" to automate a broken process because they don't understand the process well enough to build a working solution. Or the operations team that just reboots the server when it's broken because they don't have access to people with skills to diagnose the problem. Or the QA team that builds automated testing tools that are completely unmaintainable because no one in the team has a background in software engineering.

The totem pole hurts because

- it (often) treats the people at the bottom quite poorly.

- as a consequence it discourages good people from taking roles "at the bottom".

- which leads those teams to produce far-from-optimal solutions. Not (necessarily) for lack of effort, just because they don't have the skills + training to build the optimal solutions.

(Obviously, as your Mozilla example shows, some organisations manage to escape that trap)

If you need to hire less qualified/skilled people, then you're probably better off putting them into the roles that are "at the bottom" of this supposed totem pole, but if you can fill all the roles with solid engineers then you'll reap rewards from it.

hliyan2• 11 years ago

   There are two recent trends I really hate: 
   DevOps and the notion of the "full-stack" developer

I have to question the wisdom of bunching these two together. One is about freeing the developer as much as possible from peripheral duties so he can concentrate on his primary responsibility - design and development. The other, at least to me, is more about avoiding systems composed of modules that don't fit together properly, rather than getting more work done for less money.

krzyk2• 11 years ago

Not sure about the DevOps, you are talking like if operations guy would start doing some dev stuff.

But our company goes the other way, they make Devs do Ops stuff and QA stuff (and bussiness analytic stuff also). I really hate this.

hliyan2• 11 years ago

Perhaps my comment wasn't clear. I'm against making developers perform operations and QA tasks too. It's the author putting devops and full-stack engineers in the same basket that I question.

That said, developer/unit testing is one aspect of QA you just can't offload to a separate team.

lowmagnet3• 11 years ago

Why hate that? The work is part of what is necessary to deliver software. It's like saying you want to be a general contractor on you own house but don't want to have architected plans, framers, or a final inspection.

orthoganol• 11 years ago

The trend OP doesn't like is primarily because 'full stack' responsibilities, DBAs, release coordinators, etc. is probably the easiest it's ever been, so you might as well have developers throw in some extra hours to do it. I've worked at a fortune 500 that used a Ruby gem called "lol_dba"... you can spin up environments and deploys trivially with PAASs, or at the least use various automation tools, scripts, and cookbooks for provisioning machines, running deploys, CI, etc.

OP seems to think "full stack" developers no longer devote time to writing code... "This is why we see so many developers that can't pass FizzBuzz"... but really it just doesn't take us long to do the non developer stuff because it's just easy these days, maybe he's not aware of the tools people use.

scurvy4• 11 years ago

I've never met a developer that could setup a MPLS circuit or OSPF over an IPSEC tunnel. The author says that developers can do these things, but I highly doubt it. Let alone be proficient enough to do it in JunOS and IOS. Network operations should not be lumped in with devops.

uniclaude• 11 years ago

The author says that developers can do these things.

I couldn't find this in the article.

You sound even more outraged and polarized than OP is. I've met more than one developer able to do this (hint, I can also do it). Some people are actually paid to write networking software, and they often have a clear understanding of how JunOS and IOS work.

This said, for different reasons, I agree that network operations should not generally be done by your development team, but no one is arguing this anywhere.

scurvy4• 11 years ago

No I wasn't really outraged at all. My stance is that at some point you should stop trying to hire "full stack" developers. Chances are, they're not very good at everything and will miss some tiny detail that blows up big or creates a mountain of technical debt. Outside the founders and first three engineers, you shouldn't be hiring "full stack." You should start to specialize and start fixing the first five employee's problems.

As the business grows, you're going to need some specialization skills for some project or deal that comes your way. You should hire the right person for that job. Chances are, you needed them all along and this situation is a good catalyst to get them in.

amelius• 11 years ago

I think the point of the article is that if developers were required to do this, they could do it.

bryanrasmussen• 11 years ago

yeah I think I could figure out how to do it by reading a bunch of documentation, set up some scripts to do it, whenever I have to do it take longer than necessary until I had done it enough times that I had it as one of my remembered functions, and basically perform subpar on doing this thing. Which I guess is the author's point.

zamalek2• 11 years ago

The author says that developers can do these things, but I highly doubt it.

Being able to do something from memory is something we only care about in high school. Take me, for example: did my router support L2TP termination? No. Did DD-WRT support it out of the box? No. What a developer can do is cobble together information in order to find a working solution. So would I, as a non-Dev-Ops be able to set up MPLS over IPSEC, having never done it before? Yes: I have Google and I have a brain between my ears.

levosmetalo• 11 years ago

And then you happily charge 10x more than a networking guy.

devnonymous• 11 years ago

As a sysadmin in a prior life, I see this as the equivalent of a non-developer (for instance, a sysadmin) asking can I create a site scraper using google, curl/sed/awk/..bash, copy/paste and a little bit of of brain between my ears ? Yes. As a developer you know there is a lot my subtlety to programming than that.

I think what @scurvy was driving at is there is a lot more subtlety to network ops than developers would be willing to acknowledge.

friendzis• 11 years ago

What Google will give you are usually some snippets for a [very] particular and usually constrained setup that Just Works™. But once you start adapting configs for your own setup - from software versions/brands, auth/app/whatever backends to security features, network setup (just otoh: Googled snippet might assume local sqlite db and you have remote MSSQL with LDAP auth, or the given setup might intend to run own unicorn and you want to proxy to supervisorded uwsgi, whatever) - things start to break in weird ways (if work at all).

It is a much smaller issue if you are building a system from scratch and only the launch date depends on your speed, but if you are trying to modify a part of mission-critical running deployment that already generates a revenue stream... well you have to first negotiate reasonable maintenance windows and then must squeeze yourself in. Googling around will give you zero clue how much time fixing broken things will take, because, well, deboogling is done by error messages as keywords.

Furthermore, only an experienced (I'm refraining from a term "trained" here) professional can evaluate multiple possible setups, give proposals to decision makers with more or less quantitatively expressed installation/maintenance/growth costs, evaluated downtime/security risks and business impact (solution A might forget the order while still charging the customer with 0.001% chance, while solution B might forget to charge the customer at all with 0.002% chance when the load reaches 47 requests/s, so which one?). Sometimes you just want to deploy something small quick and then at some time transition to something bigger. And knowing when that is reasonable and when the small system will stop handling the load allows to actually forecast and take weighted decisions.

To the defence of DevOps: cobbled together solution is often the best (weighing in deployment time/costs) for small internal services, first dev environments or early startup prototypes. And for large deployments, someone having a clue about both sides (code and operations) is necessary to moderate the decision taking process. Because what an operator thinks to be a quick change to code might actually take several man-weeks (who was the idiot behind this architecture?) and what a developer thinks to be a small change in configs, might actually require changing core hardware (what a moron deploys like that?). In my own short experience (including, but not limited to IT/CS/EE), I have failed to witness a project where everything is decided upfront and does not change until at least first (alpha/beta/customer preview) release. The more startupy the project (the less domain knowledge), the more first release target deviates from initial goal.

Just my 0.02$

jldugger• 11 years ago

The reverse is true as well. I have a dev-ops job description, but I've been tasked at one point with creating and maintaining a web app. Arguably, it's a lot simpler to write a new webapp with Google than try to determine how to configure the particular version OpenLDAP you're running or debug the last guy's custom schema.

But really, the author here is just misguided. Just because you can 'figure it out' eventually, doesn't mean you'll be cost effective at it. Employee productivity comes from specialization. It's not about how many jobs you can do, it's how quickly you can do them well. When developers are paid well, it's because they have specialized knowledge, just like DBAs and network engineers. There's not even really a need to put developers on some fucking pedestal make the argument against 'full stack developers.' It functions purely as bait for angry comments.

glenjamin2• 11 years ago

Having recently tried to do this & similar in a "DevOps" capacity, before handing the task over to a network engineer, I have to concurr here.

However, the problem seemed to stem from cryptic tooling and almost non-existant documentation (especially on OSS tools).

sargun2• 11 years ago

Two things:

1) The developer would just say - no, instead I'm not going to use IPSec, and I'm going to bootstrap this myself with TLS. I'm going to push complexity UP the stack, not requiring esoteric network hardware to do this.

2) I think this is a bug. The fact that the underlying devices don't have easy to understand semantics is problematic. I think that if an interested developer had a program that requires MPLSoIPSEC, they would consider it, read the relevant RFCs, and then read Junos's documentation and give up - because it's locked down.

scurvy4• 11 years ago

1) What if your deal partner (who intro'd the requirement for IPSEC) has a corporate security policy of IPSEC? You need the business, you'll do what they ask. Also, you can do IPSEC on every major OS, no hardware required.

2) Again, you could use Linux for this. You can ask the whitebox switch guys, but there's a lot of stuff Linux can't do. It's a great cause and I've got a few whitebox switches in my lab, but there's definitely still a place for Cisco and Juniper in this world.

mattkrea4• 11 years ago

You are right but I don't think a lot of this is what is actually being done when it comes to whatever 'DevOps' means.

Where I work I'll push into production a few times a day after testing in staging and the only changes really are what container is running what code.

The firewalls and routing tables I don't manage and with good reason (unless they are on AWS since security groups are pretty cut and dry but even those basically don't change much after initial configuration)

jlawer• 11 years ago

I am an "Infrastructure Engineer" by job title, but really I am a jack of all trades. Many people working in small business need to wear many hats.

You can be deep on a single area (specialist) or be spread out (generalist). Larger businesses want specialists, small business want generalists.

Personally I am a generalist. I deal with security, servers, routers, switches, applications, laptops, Operating Systems, Backup, Databases, etc. I write code in python, java and starting to get better with go. Its a handful and I am always juggling things, but I also think that broad knowledge helps things. I understand TCP/IP because I wrote a TCP/IP stack. I feel I have a better understanding of the complex systems as I have a non trivial understanding of each part.

Its a very different working situation, but it has its advantages and I dread ever going back to a large corporate because of it. It isn't as efficient compared to having a team of specialists, where everyone can spend 100% on their speciality, but on the other hand, you can often do things with surprisingly small teams.

einrealist2• 11 years ago

I share your opinion. In agile teams, it is important to do things that work. If a team owns a stack sucessfully, there is no need to change that. But if the environment changes (e.g. growth), the team may decide to split / specialise in order to ensure that the business goal is not in danger. There is no silver bullet.

endymi0n2• 11 years ago

I always like the band analogy: The guitarists / developers always think they can play the bass guitar, because it's basically the same, just with less strings.

Yet, they don't get that it's all about groove, rhythm and dependability and are just way too unrelaxed and attention seeking for doing it right.

I'll take a grumpy BOFH that wires together my AWS regions in his sleep and doesn't make costly mistakes - because he measures twice and cuts once - over some kid trying DevOps any day :)

harryf• 11 years ago

I remember once haranguing a group of Java developers to use the log4j's RollingFileAppender. No matter what arguments I used and despite many nods and promises, on live systems they just kept on using the default FileAppender like there's no tomorrow, despite it being a relatively trivial config change to switch to the RollingFileAppender.

Only the sysadmins "got it" because they were the ones that got to feel the pain of cleaning up a NAS filled up by log files.

glenjamin2• 11 years ago

Out of interest, why wouldn't you configure logrotate on the machine itself, rather than making this an application concern?

Presumably logrotate was already set up for the variety of other logs on the system?

einrealist2• 11 years ago

Did they own the systems their applications were deployed to? Because that is the essence of full stack development. Otherwise, config management is required as boundary context between developers and operations.

mhaymo• 11 years ago

To my mind the whole point of DevOps is to make the developers "get it" by making them feel the pain when they get things wrong. Developers make decisions about the system every day, so it pays for them to understand how it is maintained and operated.

HelloNurse• 11 years ago

This only proves your Java developers were stupid and arrogant; using RollingFileAppender or something even smarter would be the norm in any sufficiently functional organization, regardless of whether developers anticipate the need or sysadmins need to ask.

collyw• 11 years ago

He does say in the article that good devs can make passable DBA's, deployment engineers, etc.

Of course they are not going to be experts at it, because their field is development.

endymi0n2• 11 years ago

the same way, guitarists usually hit the right tones on a bass :D

6t6t6• 11 years ago

I love this analogy. SysAdmins need to have an special mindset that is diametrically opposite than these of the developers.

Developers want features and shiny things and following new trends and unicorns.

SysAdmins want 99.999%, and dependability and knowing that all the backups are running on time and writings contingency plans detailing how many minutes it will take to put a new infrastructure online in the event a fire occurring in the main data-center while the backup data-center is flooded because of a typhoon.

tomohawk• 11 years ago

Devops may be a lame buzzword, but most interesting problems cross traditional boundaries. As a dev becomes more senior and wants to become more valuable, they'll take it upon themselves to learn more about the other domains they interact with. The next step is they'll start trying to take on these cross domain problems that become apparent at this point. What I've seen is that when devs stick to straight coding, the products they work on tend to be less successful and their careers tend to be less than they should be.

kriro• 11 years ago

The article seems all over the place. For example I think the conclusion

"""This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code."""

Doesn't really follow from the previous parts. DevOps is not the reason for this nor is the full stack developer trend.

The author also seems to start from an implied base premise that developers wouldn't do devops/fullstack if they had a choice. I disagree with that. Many developers are very interested in understanding the implications of their code better (more DBA skills help a ton for example) and quite a few gasp actually enjoy devops or understanding the entire product from start to finish at least on some level (full stack).

The benefits of DevOps also seem to be completely ignored. It's precisely the fact that common development best practices are integrated with sysadmin, dba etc. knowhow which makes it useful. It's not just "let the dev be a sysadmin in a pinch". It's systematically improve things by automating them (following development best practices instead of the hacked together bash script approach).

DevOps and fullstack dev are conflated throughout the entire article in a strange way which makes it kind of hard to argue.

sargun2• 11 years ago

Personally, I love DevOps. I've played on both sides of the fence. More ops than dev, I'll admit.

The way I view DevOps is that you have end-to-end guarantees about software delivery. A lot of this draws from personal experience, so YMMV.

Story:

I was working in an organization where engineering had built a custom solution with their own database to a problem. It was using their own DB system, with the JVM, etc.. I was oncall for said service, because their DB kept getting into a poor state. My only real recourse to a page was (a) escalate to the devs (b) restart the service. I ended up furiously rewriting the service in Erlang over a weekend, and ended up getting told that my replacement was too complicated, and the previous one worked okay, because we could just restart it.

Take-aways: Without feeling the pressure, and side-effects from being woken up, and having to respond to on-call incidents for a service, it makes people put more value on the development time than operational time, although a service is very likely to be in operational time much longer than development time.

I have two other stories, but it'll take some time to write, and it's getting late, so I'll type it up only if someone is interested. Just so I remember - the tale of LD_PRELOAD and HDFS, as well as the networks and root access.

1. I think that ops staff should provide shared services to services organizations (hosting, deployment, etc..) when N services need the shared service (when N>2, depending on how many resources are available in your org).

2. I also think that ops staff should be able to "consult" on projects, as members of those teams during development. This is going to mean that those engineers are probably going to need to do some more development! Or, alternatively, the project team needs to be able to backpedal on some decisions if those ops consultants come in later on the project.

3. I think dev folks need to have end-to-end responsibility, including oncall, AARs, and on-call remediation. They need to take feedback from ops folks and act upon it, just as they would if product told them to do something.

myVocatio• 11 years ago

To spoof J L parker, I was "once a developer". Have played many roles since (PM,strategy consultant, VP, p&l owner,...).

DevOp makes sense to me for building complex solutions..but, for different reasons than discussed here.

Consider this: a small team (usually 3-7 members) of "jack of all trades" is proven across industries where the team has to deliver a complex solutions under a very aggressive timeline ,requiring multiple skills. Two examples from my personal experiences: Strategy consulting - eg @ McKinsey (work includes interviewing executives, presentation, excel, data entry, hypotheses development and verification....) investment banking eg @ Goldman Sachs (work includes industry analysis, financial analysis, scenarios, presentation, excel, data entry, ...)

"Jack of all trade" model in terms of DevOps works because:

1) goal for the team w/ "full-stack" skills is to have a reliably working software solution 2) communication and management overhead increase significantly as the team size grows 3) complex solutions need different "amount" of various skills at different point in solution cycle 4) adding/ removing team members comes w/ significant cost to alignment

How 'DevOps' is Killing the Developer (2014)

Related Stories