The Difference Between CI and CD

https://fire.ci/blog/the-difference-between-ci-and-cd/

by jpdel • 7 years ago

194 58 7 years ago

coderinsg• 7 years ago

Well written. The line between CI and CD has been blurred esp when they’re commonly mentioned together. Many cant tell the difference

hinkley2• 7 years ago

Few things are as aggravating to me as people who say they understand CD but then manage avoid practicing any of the tenets of CI.

Automated builds are the smallest part of CI. Necessary, but drastically insufficient. If that's all you're doing you've missed the forest for the trees.

tomxor• 7 years ago

Few things are as aggravating to me as people who say they understand CD but then manage avoid practicing any of the tenets of CI.

It is completely reasonable to utilise one without the other, not all projects are giant multi author efforts trying to wrangle commits.

For instance if you have lots of small projects being worked on independently in parallel with no more than one or two authors on a repo at a time, CI is not going to be worth the investment... but CD still has it's uses.

hinkley2• 7 years ago

The scenario we want to avoid is that a faulty commit makes it to the main branch.

Close. The scenario we want to minimize is faulty code on the main branch. As your team grows, as the number of commits go up, it becomes a game of chance. Sooner or later something will get through. The more new teammates you have, the more often that will happens.

This is an inescapable cost of growth. The cost of promoting people to management. The cost of starting new projects. Occasionally you can avoid it as a cost of turnover, but you will have turnover at some point.

What matters most is how long the code is "broken" (including false positives) before it is identified, mitigated, and fully corrected. The amount of work you can do to keep these number relatively stable in the face of change is profound.

If you insist on no errors on master ever you will kill throughput. You will create situations where the only failures are big, which is neck deep in the philosophy that CI rejects: that problems are to be avoided instead of embraced and conquered.

jdlshore3• 7 years ago

It's possible (and not that hard) to define an integration process that prevents faulty commits from being integrated to the main branch.

If you insist on no errors on master ever you will kill throughout.

Not sure why you believe this. It hasn't been my experience; just the opposite, in fact. By using CI in conjunction with a process that prevents errors on master, everything goes more smoothly, because people don't get stalled by the broken master.

richardlblair• 7 years ago

You're both right.

The healthy mentality is to realize mistakes will happen. This creates a healthier culture when things do break.

However, you should take every step to ensure it doesn't happen. You should act as though you want to prevent all faults from hitting you master branch.

Ididntdothis• 7 years ago

"It's possible (and not that hard) to define an integration process that prevents faulty commits from being integrated to the main branch. "

You should strive to do that but you shouldn't be surprised that despite all effort mistakes still happen from time to time.

brown9-22• 7 years ago

It depends on what you means by “errors on master”. Tests won’t catch all possible bugs in production.

dantiberian• 7 years ago

If you insist on no errors on master ever you will kill throughput.

There are a large number of automated tools which will help you prevent merging code that could break master: https://github.com/chdsbd/kodiak#prior-art--alternatives. The basic approach is to make a new branch from master, apply one or more commits on top of that branch, run the tests, and if tests pass, merge those commits (with fast-forward) back onto master. This makes it very difficult to get broken commits on master, as they have to pass the tests before. It is possible if you have a flaky test suite, but in my experience it happens very rarely, and is usually very easy to fix if something creeps in. In my experience, they speed up throughput, not slow it down, especially when you account for the disruption that merging broken code to master can be.

https://graydon2.dreamwidth.org/1597.html has a good discussion of this:

The Not Rocket Science Rule Of Software Engineering:

automatically maintain a repository of code that always passes all the tests

sytse2• 7 years ago

The problem with a popular repository can be that running the tests can take longer than the time you have between merges.

In GitLab we made merge trains https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipeli... to solve this problem automatically.

With merge trains the merge requests with a passing feature branch is placed in a queue and tests are run against the combination of that branch and all the branches before it merged in. Since tests will pass 95%+ of the time the feature branches passes this can speed up the amount of merges you can get into master by 10x or more.

dionian• 7 years ago

automated tests don't necessarily catch all errors

jpdel8• 7 years ago

Agree. Some faulty commits may go through. But then you strengthen your test suite to prevent similar issues to happen again, and so on.

gregdoesit2• 7 years ago

If you insist on no errors on master ever you will kill throughput.

Unless you solve this engineering problem with tooling. At Uber, the full-blown CI mobile test suite takes over 30 minutes to run on a development machine (linting, unit test, UI tests - most of this time being the long-running UI tests, specific to native mobile). So we only do incremental runs locally, and have a submit queue, which parallelises this work and merges only changes that don’t break, into master. And we have one repository that hundreds of engineers work on.

It’s not an easy problem and the solution is also rather complex, but it keeps master at green - with the trade-off of having needed to build and maintain this system. See it discussed on HN a while ago: https://news.ycombinator.com/item?id=19692820

pojzon2• 7 years ago

How do you handle situations like that: multiple dvelopers added merge requests to queue, the changes they made are mutually exclusive (automatic rebase wont work). What happens when the first branch gets merged to master and next 10 are still in the queue ? How do you mitigate that to decrease development cycle ?

Lets just say in my company it also takes 30m to run tests and 4h to run them on merge pipeline with FATs and CORE tests.. Its way too long and highly cripples productivity.

UK-Al05• 7 years ago

This may surprise you, but tests only help you find breaking changes. They don't guarantee it. To guarantee anything costs a lot.

peterwwillis2• 7 years ago

An alternative system includes optimizations: 1) don't use a monorepo, 2) don't run tests that have nothing to do with the code changed. Both require redesign of code structure, testing, execution, but both remove the inherent limits of integration.

Nobody seems to talk about this and I don't know why. It would remove integration complexity and speed up testing. We do the same thing for CD and nobody seems to have a problem with it...

devonkim• 7 years ago

In SRE and DevOps land we’ve mostly had arguments over continuous deployment vs continuous delivery and have mostly let feature engineers decide how they want to use the possible approaches and options available

cottonseed2• 7 years ago

Keep it short. 3-7 minutes should be max.

Who has a 3-7m CI build here?

smcleod2• 7 years ago

I would say for me that's a pretty reasonable estimate for microservice architecture applications / services, of course large legacy monoliths take longer but not more than say 15-20 minutes at most.

cottonseed2• 7 years ago

3m seems aggressive to do builds and spin up infrastructure for anything non-trivial.

Reading a bit closer, I see the author describes CI as a sanity check, "ensur[ing] the bare minimum" and doesn't consider deploying on every commit. Maybe 3-7m is more realistic then.

However, I'm slightly surprised by this definition of CI. According to Fowler [0], "Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. ... The key test is that a business sponsor could request that the current development version of the software can be deployed into production at a moment's notice." So having CI gates on the development version that are weaker than the release tests would not seem to be continuous delivery according to his definition.

We're currently releasing on every commit and our CI build (which implements continuous delivery) takes about 15m.

[0] https://martinfowler.com/bliki/ContinuousDelivery.html

adrianN2• 7 years ago

Building a decently sized C++ codebase takes 20 minutes. Then you want to run tests on it.

jpdel8• 7 years ago

We have a large Java monolith application. Builds ran for 30 minutes. Then we said let's only run the unit tests and critical smoke tests. The build time went down to 7 minutes ... on 12 CPU and 32 GB of RAM build slaves :) There's always a way.

solatic• 7 years ago

Maybe for software it's reasonable (tests should be parallelized more as you go above a 5m build), but for infrastructure it's ridiculous. No IaaS API responds remotely quick enough to bring up whole environments from scratch that quickly.

jpdel8• 7 years ago

Agree. I would definitely put any infrastructure thing on the CD side though, as it is longer indeed.

You can test your software faster in CI using Docker compose. Something along these lines: https://fire.ci/blog/api-end-to-end-testing-with-docker/

petepete• 7 years ago

Our entire test suite takes about six minutes to run, we run all our tests in CI with Capybara. Our CD pipeline runs the same tests but against Chrome, Firefox, Safari and Edge. It takes more than an hour.

glenjamin• 7 years ago

/me raises hand

On our newer services there is a CircleCI pipeline which parallelises work and takes ~1-2 minutes on a branch, and maybe an extra minute at most on master - where it automatically deploys to production if the build is green.

If you make the choice to prioritise this from the start, it isn’t all that difficult.

dewey• 7 years ago

Our biggest Go project has a sub 3 minute build time including running a bunch of tests, staticcheck, pushing to registry etc.

It doesn't have to spin up any DB or things like that though.

baq2• 7 years ago

it takes 3 min to run linters here...

s_Hogg• 7 years ago

The title reminds me a lot of all those fatuous articles about the difference between statistics and machine learning. This one is alright though - I wonder how we got to lumping CI and CD together as is commonplace now.

jpdel8• 7 years ago

I think this is a business trend unfortunately. Tools tackling the CI space wanted a piece of CD and then boom. Things became the same :)

danpalmer• 7 years ago

I wish more CI/CD services understood this.

We recently moved from Jenkins to CircleCI, and while the PR experience has improved dramatically (no queueing, faster builds), the _release_ process is far worse.

The reason seems to be that CircleCI just treats CD as CI. In reality doing CD requires high correctness, care, and nuance.

For example... with CircleCI there's no way to ensure that you release your code in the correct order other than to manually wait to merge your code until the previous code has gone out. That's not _continuous_. This is a very basic requirement.

So perhaps they are not a CD service as they pitch themselves as? That means deploys are manually triggered then? Nope, there is no way to manually trigger a build.

I wish this was an isolated example, but I've yet to see a CI/CD service that is easy to build fast, correct, deployments with. Jenkins is correct but not fast or easy, Circle is fast but not correct, and most others I've used are none of these at all.

a_imho• 7 years ago

The process of Continuous Integration is independent of any tool.

This is one of my pet peeves, people using the term CI referring to the tooling. For me this alone invalidates anything they have to say about the subject.

bump64• 7 years ago

I am using Azure Devops for the past couple of years and they have kind of nailed it. You have builds that do most of the CI and Releases which can be fine tuned to do complex deployments and complete the CD story. Then you can set them as a requirement for the pull request approval to the main and branch it helps to guarantee healthy trunk.

I don't agree that CI is a team problem and CD is an engineering problem. If you are following infrastructure as code principles it is everyone's problem because if you don't add how your new feature should be deployed it will break the CI and CD pipelines and you won't be able to merge it.

jpdel8• 7 years ago

Also using Azure DevOps and it is indeed very well structured.

As for CI/CD differences: how many commits can actually affect code and infrastructure? I think this is part of the engineering problem at the end of the day.

agustif• 7 years ago

I've been using [branchci](https://branchci.com) (free tier) recently to build, lint, and deploy a wp site and been very happy with it!

The Difference Between CI and CD

Related Stories