The Continuous Delivery Test

https://sourceless.org/posts/the-continuous-delivery-test.html

by sourceless • 4 years ago

87 67 4 years ago

dijit3• 4 years ago

This feels like yet more cargo-culting.

Even the first item on this list is hotly contested, with Facebook/Microsoft/Google and Co. using centralised mono-repos, google itself using a server model similar to Perforce called Piper.

To be clear here, I’m not saying they’re right, but with it feels un-scientific to make a blanket statement that they’re wrong.

theoldlove• 4 years ago

You can easily have a monorepo with a distributed control system.

dijit3• 4 years ago

Yes. But I’m referring to the article:

1. Do you use a distributed version control system?

Edit: the parent originally said “distributed build system”

UweSchmidt• 4 years ago

9. Does your Infrastructure as Code live alongside the service it hosts?

That means minor changes to some test infrastructure have to go through the strict review process, because the gitlab-ci.yaml is in the part of the main code. Last time this happened to me I found it annoying, and I don't think the code quality guardians care about some CI config anyway.

davidmurdoch• 4 years ago

Changes to CI config for release automation are probably vetted and manually tested more than anything else in the projects I work on.

xorcist2• 4 years ago

I will be first in line to suggest to not split repos unless required. It is my experience that deployment, admin and node provisioning (and everything that is generally put under the not always useful IaaC umbrella term) is one of those exceptions. Often documentation is, too.

It all depends on what the release flow looks like. A useful rule of thumb when components can share a repository is when the branch model is identical. If that's the case for codified release and provisioning processes and the code that is shipped, then it is likely that you are not taking full advantage of what that deployment and provisioning code can do. If not anything else, then at least test versions must deploy production code, and test code deploy production versions.

Again, in my experience across multiple organizations, it is a good idea to keep that code both forwards and backwards compatible. It should be natural that is sometimes says things like "if build number greater than x then set parameter y". While it may feel a bit dirty for some, that type of logic is what deployment code does, and it is much more maintainable than keeping branches around for the same result.

If I get to say one thing to new devops/release engineer type people, that's probably it. That, and the value of clear and concise commit messages. Which goes double for this type of code.

alkonaut• 4 years ago

Extremely web centric list. Any other development (mobile, desktop, games, embedded, …) will find a lot of these steps very weird.

kqr7• 4 years ago

Really? I challenge you to find any items other than 7 and perhaps 8 that don't apply to the types of development you mention.

Essentially the only thing that's different with the types of development you mention is that "final deployment to production" looks different, as it usually involves more or less physically transporting artifacts to your customer.

But the rest is just the same. You should be doing trunk-based development, practise something like code review, gate on integration tests, feature flag, spin up virtual production clones, etc.

In other words, the fact that getting software to your end users is a clumsy process does not preclude you from doing every other step of the process with short feedback loops.

maccard4• 4 years ago

Using distributed version control isn't applicable to most people working in games.

9) is also debatable - requiring someone to clone the entire application to make an infra change to a testing environment.

Adding tickets to commit messages isn't necessarily a requirement - some work (at least in my area) is prototypey and maybe ill defined (the task might be to define it).

Being able to deploy from your own machine is a double edged sword; the situation you need this is an absolute last resort. Enabling deployments from dev machines means credentials to environments, write access to infra, and likely skirting around normal processes.

marcosdumay• 4 years ago

What?

#2 and #5 exist only to support #7.

#4 usefulness is completely dependent on the context.

#9 is a joke, right? It's an anti-pattern for most of the web based SaaS applications, and poisonous to anything that isn't SaaS.

And #12... Try that in a large regulated company and people may quite as well end up in jail.

hoskdoug• 4 years ago

Indeed. Does your deploy process self-heal? Err, no, I need to wait two days for App Store review.

ChrisMarshallNY3• 4 years ago

The whole idea of "Continuous Delivery" is not always one that meshes well with some shrink-wrap workflows.

I worked for hardware manufacturers, for most of my career, and software was always just a "sidekick" to The Main Show. We just got the "Supporting Actor" nods.

I'd say that 90% of HN seems to be Web/SaaS (and, these days, crypto), which is an excellent workflow; just not the one I do. Nowadays, I have some integrated stuff, but it's mostly native iOS/TVOS/WatchOS/MacOS deliverables.

CD is nice, but I feel that CI is better, for a team. Even that, is overkill for the way I work. I'm spoiled, I tend to work alone, or very loosely-coupled. That gives me a different workflow from what many experience. I had to spend a couple of years "un-learning" a lot of stuff from my Big Team days.

The way I work results in astonishingly good stuff, very quickly, but the scope is much narrower than what a lot of folks here, do.

As such, I find little utility in telling others that the way they do things is wrong, and that they should be doing it my way. I do talk about how I do things, but I'm not judgmental about it.

I do feel that software quality, in general, is fairly problematic, but don't claim to have the silver bullet. I know what works for me, and I try to stay in my lane.

8n4vidtmkvmk3• 4 years ago

i work on a very large team and solo for my side hustle. love working solo, no red tape to fight. no waiting 24 to 48 hours for a teammate to review my code per iteration and up to 24 more for it to deploy. just signed 2 contractors on for my solo project though and its really messing up my process. all the overhead is so high I'm not even sure it's worth it even if they were free.

jiggawatts2• 4 years ago

I worked for hardware manufacturers, for most of my career, and software was always just a "sidekick" to The Main Show.

Which is an enormous, catastrophic, fantastic mistake that should be leaving everyone breathless with shock.

Realising that the software matters is why Tesla is worth more than most of the other car manufacturers combined.

This is why Apple is the #1 biggest company in the world.

This is why every time some "hardware" has to be deployed, every enterprise admin rolls their eyes and groans.

This is why IoT, medical, and factory automation security is a trash fire.

Smart televisions aren't, and you can waste $5000 on one just to have a substantially better experience by plugging in a $150 box from Apple.

And on, and on, and on...

I literally told the local Toyota rep that I wouldn't buy a 2019 model-year vehicle specifically because it didn't have Apple CarPlay. The built-in system is simply garbage. Maps that are 4+ years out of date!

Apple has nearly monthly updates for iOS, which means if I plug my phone into my car, effectively my car gets monthly updates. With Toyota? Maybe once in a decade they'll release an update, and then never again, slowly but surely deprecating its software capabilities down to "worthless".

Similarly, Nikon releases updates for their existing cameras once in a blue moon. Recently they announced "Firmware 2.0" for their flagship Z9, and I was shocked. This is likely a one-time aberration, probably caused by their software division not being ready in time for the initial shipments. I guarantee that there will never be a Firmware 3.0. Never! Where I live, this camera with one lens and typical accessories will set you back $10K and is deprecating at an exponential rate because Nikon does not give a s%*t about software. Meanwhile my iPhone and its camera will keep getting updates.

So yes, CI/CD is a web-centric notion.

It ought not to be.

choeger• 4 years ago

Pretty much everyone with more than one production environment will. Imagine developing Firefox. How do you do a "self-healing deployment" automatically whenever trunk changes?

zaat2• 4 years ago

You take the standard course of action, sending the PR team to state how Firefox is there as a beacon of hope and aspirations to privacy.

bhawks• 4 years ago

11. Do you include ticket IDs in your commits or branches?

This is one of the things that sound 'good' on the surface and are totally worthless in practice (especially when practicing minimal commits).

In the wild this devolves into a meaningless umbrella issue (Deliver Feature Foo) or 1 liner issues that sometimes did not even match the pr contents.

Just use the commit message and in pr review enforce commit message norms.

kqr7• 4 years ago

My previous team used to have ticket IDs on every commit but just decided to drop that requirement for commits that are truly self-contained and documented properly in the commit message.

That approach makes a lot of sense to me. Link the commit to the ticket when it enhances future reading, but not out of blind application of principle.

whakim• 4 years ago

It's incredibly easy to simply add the ticket number to branch names (or commits if you're rebasing your branches when you merge). Many times the ticket isn't helpful, but "just make the commit/branch self-explanatory" ignores the fact that non-developers - product folks, designers, etc. - are far more likely to leave useful context in the ticket itself.

lelandfe• 4 years ago

Yes, exactly. I treat my commit history like I'm preparing for a detective to investigate years down the line :)

Great commit messages are just one part of the timeline. Ensuring there's a link between the commits/PR and the ticket lets the future detective get more information (frequently, that's "why did we do this").

kqr7• 4 years ago

When there is a ticket, yes, adding the number is trivial and probably worth it. Where the rule breaks down, in my opinion, is when you have to create a ticket and replicate the commit message in it any time you stumble over a code maintainability issue and fix it in stride.

wodenokoto• 4 years ago

Have one branch for every story in Jira has been incredibly helpful in teams I’ve been in.

You can check in git if work has actually begun, and you can get from and old commit to Jira by virtue of the merge commit.

I don’t know how you’d do it in a non-mono repo setting

kqr7• 4 years ago

There's a difference between "every story has a branch" and "you need to create a story for every commit" which is what's implied by TFA.

crdrost2• 4 years ago

I like this, but it violates one of my norms, haha.

Stories can last for a while, but long-lived feature branches have to be kept in sync with the main branch, otherwise they develop nasty merge conflicts and you are not deploying continuously.

I think the better solution is for your versioning and your issue tracker to be integrated: we still expect every developer to merge to mainline one or twice a day[1], keeping merge conflicts shallow... But they should mention what issue they are working on in the commit, and the integration dumps that information into the tracker automatically.

[1] This is not hard to make safe, though it makes a lot of devs skittish at first. See Three-Flow for a non-CD version, https://www.nomachetejuggling.com/2017/04/09/a-different-bra... as a stepping stone. One key thing is that code review needs to become much lighter than it is at many organizations, “I don't care how you did it as long as it can be turned off with a feature toggle and does not break prod or introduce security vulnerabilities when I merge it: those key things are what I am looking for.”

theptip• 4 years ago

I think it can be useful to encourage (not require) this practice since auto-linking between Jira and GitHub/Gitlab is a nice feature. Jira can now show you what releases a given feature is deployed in, etc.

If you are in a regulated environment or selling to large enterprises you may find that having a SDLC policy of “no changes without ticket to track” is a more easily defensible control against unauthorized code changes (you’ll get asked about this in SOC2 and security questionnaires). The policy “anyone can make a change if they find a reviewer to approve” might not float.

Mandating ticket IDs can be really annoying though, for example if I want to make a few-line no-op refactor to clean up some code, I don’t want to have to create a ticket. So it’s definitely a tradeoff.

KronisLV5• 4 years ago

This is one of the things that sound 'good' on the surface and are totally worthless in practice (especially when practicing minimal commits).

I feel like I disagree here.

Turning on Git Annotations in any JetBrains IDE and seeing who made changes, when and which Jira issue necessitated these changes right there is pretty useful. All of the sudden, you can easily say: "This method had been changed in 5 different commits in the last 3 years, what I probably need to read before making my own changes are issues ABC-123, ABC-236, ABC-455, ABC-855 and ABC-1233, to understand the business context and historical limitations behind all of this."

Just use the commit message and in pr review enforce commit message norms.

I fully agree with this, but unless you squash your commits or rebase, finding out which MR/PR a particular set of changes belongs to isn't always immediately doable, especially when looking at things locally instead of the GitLab/GitHub UI.

Furthermore, the barrier of entry is higher: I always describe my changes and add a bit of context in the merge request/pull request description, sometimes even with GIFs or MP4 video demonstrations of what happens as a consequence of them. And yet, the majority of other developers have no desire to do that - I've regularly seen people leave those empty, and even commit messages are sometimes like: "try fix" instead of "make Ansible ensure that file permissions are set to have the newly created directory be readable by the group that will run the app".

In contrast, everyone is capable of adding a simple identifier to the issue management system and there are very few arguments that they can make against adding a few characters to the beginning, as opposed to: "But adding descriptions would take a lot of time and slow down development, I don't think we really need those because you should be able to read the code and understand everything."

Sure, you can enforce it in a top down manner, but it can be like pulling teeth at times, so you might as well ensure that at least the lowest common denominator is in place, before trying to find better approaches.

Exceptions to this might be small changes that don't really correspond to an issue, then you have two choices:

  - create an issue yourself so the work doesn't become untraceable in the issue management/billing system you use
  - just leave a short text description without issue identifier

xorcist2• 4 years ago

ABC-855 and ABC-1233, to understand the business context and historical limitations

This should be what a commit message is. Commit messages should describe the rationale behind the change.

Ticket numbers are great for linking commits together. But they should not have to be relied on to find out why a change was made. Ticketing systems may come and go but commit messages persist.

lawl• 4 years ago

I disagree.

I have dealt with bugs that were thought to be fixed a long time ago, only for them to mysteriously pop up again later. Ticket is created and the old ticket is linked.

It's really helpful to see what was done 2 years ago, including the attempted fix that is now still live in the code base but apparently doesn't work properly.

That said, as always, it's not black or white. There are definitely cases where it doesn't make sense, but I don't think you should call it 'totally worthless'.

Merad• 4 years ago

I've yet to work at a company that has managed to figure out a strategy to get the majority of devs to write clear and descriptive commit messages the majority of the time. At least if they include the ticket number I can look at the ticket to understand what they intended to do. Of course whether they actually did it or not is another matter.

brundolf• 4 years ago

Not remotely true in my experience. I can't tell you how many times I've dug into the commit history to find where a change happened, then wanted to know the full context, and been glad I could just pull up the years-old ticket directly and see a detailed description of intent and requirements, discussions in the comments, mock-ups, etc etc

satyrnein• 4 years ago

I'd like to see a workflow that meets all the criteria. I was surprised to see the item about deploying directly from your own machine; that seems to contradict the other goals which point to automated pipelines that deploy on merge.

TheP1000• 4 years ago

This is a very common pattern especially with the cdk. Having a locally deployed stack makes development in the cloud so much easier.

kqr7• 4 years ago

I feel like a lot of people misread that point. It's not about deploying to production (that sounds nuts!) It's about spinning up and deploying to a full dev environment. This is in part to not have to run everything locally, but mainly to exercise deploy automation regularly.

etamponi2• 4 years ago

7. 7! This is by far the one that resonates with me the most. But for some reasons SREs think the opposite is true: the longest it takes to get to prod, the better. What a pain.

ryanklee2• 4 years ago

I'm an SRE and currently on a mission to get code into prod as fast and safely as possible. Instead of painting such a broad, generic negative picture, maybe articulate an actual position to argue against.

etamponi2• 4 years ago

You are right -- I was referring specifically to SREs in FAANG, which enforce a mandate of maximum 1 deploy to prod per day -- in some case even 1 per week, or less.

To be honest, I think there is a threshold, perhaps expressed in terms of risk of losing money, at which it is more beneficial to delay deploys.

But applying the same deployment rules to a multi-billion-dollar revenue machine and to a smaller project with perhaps a few thousand weekly users is ridiculous.

zaat2• 4 years ago

9. Does your Infrastructure as Code live alongside the service it hosts?

No, the IaC is configuration, and configuration should be separated from the application.

in organizations that implemented GitOps, the IaC files in git are not only a copy of the configuration, it is the source of truth from which the configuration is copied to the systems. Unless all the developers who can commit code should have permission to change production configuration, the IaC repo should be separated from the code repo.

The Continuous Delivery Test

Related Stories