How to fuck up software releases

https://drewdevault.com/2019/10/12/how-to-fuck-up-releases.html

by zdw • 7 years ago

318 88 7 years ago

sroussey• 7 years ago

Nice heads up! Working on a similar script to help with updating versions and packaging across five platforms. At least I have a (mostly) monorepo. Which is great until it’s not.

vortico6• 7 years ago

I keep a release TODO list and about 50% of every release, I fuck up and need to add something to the list. It's about 40 items long now (containing various terminal commands and activities), which includes making sure manual entries, documentation, the website, social media, dependencies, etc are handled correctly.

Swizec• 7 years ago

Sounds like a great potential SaaS offering

z3ugma• 7 years ago

Manifest.ly is designed for this workflow - https://www.manifest.ly/

PacifyFish• 7 years ago

What kinds of things do you need to check for that aren’t automated? Maybe I’m spoiled by our CI pipeline.

I’d love to take a look at your list

vortico6• 7 years ago

Here's a somewhat redacted version. `m` is my alias for `make`.

    - Finalize source
     - Legal check `LICENSE-dist.txt`
     - Update `CHANGELOG.md`
     - Bump version in `Makefile` and `Core.json`
     - Commit "Bump version".
    - Build (three OS's can be done in parallel)
     - `m clean`
     - `git pull`
     - Make sure you have the latest `Fundamental.zip` package in source root.
     - `m dist`
     - Manually test installer and fragile features (audio drivers, patch loading) for ~10 minutes.
     - `m notarize` (on Mac)
     - `m upload`
     - `git tag vX.Y.Z`
     - `git push --tags`
    - Release
     - Update version title and URLs in `Rack.pug`
      - `m upload`
      - At this point, normal users have access to new version.
     - Update server version in `config.coffee`
      - `m restart`
      - At this point, normal users will swarm to download new version. Keep an eye on server bandwidth.
    - Publicize
     - Twitter https://twitter.com/home
     - Facebook https://www.facebook.com/vcvrack/
      - Share on group
     - Forum https://community.vcvrack.com/c/announcements

gourlaysama• 7 years ago

The Scala release checklist [1] is a good example of a release checklist with both automated and manual steps.

[1] https://github.com/scala/scala-dev/issues/645

chubot• 7 years ago

Every time I "fuck up", I add a few commands to a shell script, so I don't mess up the next time.

Over time that has grown into over 1000 lines of shell, e.g.

https://github.com/oilshell/oil/blob/master/devtools/release...

And this is why I'm writing a shell -- because it's good for automating things that you are otherwise too lazy to automate!

I find that "semi-automation" is a good middle ground. This is where shell is better than "real" languages. You don't have to commit to writing a polished program -- you just incrementally improve things and it makes problems go away.

The time you put in is saved immediately, as opposed to other languages, where you might spent 30 minutes writing something that will save you 2 minutes in the next year, which isn't necessarily worth it.

If you keep a text file with a bunch of terminal commands, then you might as well chmod +x it and put #!/bin/sh on the front. That's what a shell script is!

----

Here is the output of the release process, which has tarballs, change logs, documentation, test results, benchmarks, etc.

https://www.oilshell.org/release/0.7.pre5/

It's way to much to do by hand.

ssully• 7 years ago

It's so relieving to read this thread. Last year I took over management of some legacy scripts and was tasked with updating them (move from Python2 to 3, interfacing with some new APIs, etc). One of the biggest stresses I had was little fuck ups I kept doing with the releases. It was never anything huge, but every few releases I would end up taking 15-30 minutes longer then I planned, and would involve me updating the release process slightly.

peterwwillis• 7 years ago

A lot of these can be avoided by making all the steps repeatable. To do that, you should run everything in a Docker container that's built solely from version controlled files, and run it on a remote system that gets blown away regularly. You can use Jenkinsfiles to specify exactly what to do with different branches, and to really simplify things, you can have a different job for every kind of release. Build in a dry-run option so you can run a whole release without actually publishing anything. Each of your software projects can have its own specific release steps in a script run by your release container, while still following the same basic release process for all your projects.

greggyb• 7 years ago

For some prior art in this space, it may be useful looking at ESR's shipper:

http://www.catb.org/esr/shipper/

And some blog posts he's written on it:

http://esr.ibiblio.org/?s=shipper&submit=Search

kick• 7 years ago

Can dang or sctb fix the title of this to the correct one?

"How to fuck up software releases"

josteink2• 7 years ago

I am doomed to creatively outsmart my tools in releases.

That’s a very honest admission, and one I think lots of “why don’t you automate everything”-crowd seems to fail to recognise as a real-world factor.

praptak• 7 years ago

I would say the article actually supports the "automate everything" position. The author prevents future mistakes by adding more automation rather than less.

rabidrat• 7 years ago

and yet, future mistakes are still happening. A lot of work has gone into automation but the outcome is often the same. So is it worth it?

cjblomqvist• 7 years ago

I agree. While I like to automate away things both for saving time and consistent results, we have a tendency to throw away good old thinking or the window. Have I covered everything? Did I check all the outputs? What am I missing? What step that I did last time have I missed this time?

Sometimes it feels like we're throwing out good old proper thinking and proper work in order to blame automation, that the results is a natural outcome of all the complexity or that we're pressed for time. Sometimes it's just a matter of taking responsibility and doing a good job.

perlgeek• 7 years ago

Another "fun" one: typo in git tag names.

The rakudo project has monthly releases with tags like 2019.07. Once I mixed up the year and month in e release, so I created a tag 2018.06 instead of 2016.08 (don't remember the exact numbers), and pushed it.

I deleted the wrong tag as soon as I noticed, and pushed the correct one, but of course when the date of the typo'ed release month came around some years later, things blew up for everybody who still had the wrong tag in their local repo (which turned out to be quite some developers).

m0xte2• 7 years ago

This is one thing I really loved about subversion. I just used the VCS revision as the tag and version therefore avoiding this issue. Didn’t even need to create tags.

That was until the version overflowed the 16 bit field it was stored in the assembly manifest. Kaboom. :(

People spend way too much time arguing with semantic versioning when 99% of the time the only important thing is the next number is bigger than the previous number.

jakobegger3• 7 years ago

SVN revision numbers are the only thing I miss after switching to git.

thrower1232• 7 years ago

I agree, it is such a cluster to get human-usable version numbers out of git. I use a convoluted Rube Goldberg contraption based on tags and git-version[1]. It's so much less convenient than old SVN revision numbers.

[1] https://gitversion.readthedocs.io/en/latest/

nickjj• 7 years ago

Typos are the worst when it comes to things like that.

Especially when you're the type of person who always seems to have typos after you proof read something a few times. But then when you proof it again AFTER you post it, suddenly they all jump out. I can't count the number of times I've gone and edited a HN comment even after proof reading it a few times.

Needless to say dealing with tags or writing a tweet often includes a silly amount of triple checking.

potta_coffee• 7 years ago

Also typos in container names.

dbrgn• 7 years ago

For bumping the version there's also bumpversion: https://github.com/peritus/bumpversion

merlincorey• 7 years ago

Two things should be noted:

1. `bumpversion` is more general and has more functionality than `semver` in the article 2. `bump2version` is the currently maintained version[0]

I currently use `bumpversion` to manage versioning in applications with many different versioning schemes as well as managing the versioning of their deployment environments (such as updating Terraform files, etc).

[0] https://github.com/c4urself/bump2version

lifeisstillgood• 7 years ago

The big lessons from this article (which I love and resonate with)

- You discover more bugs with your shins than your foresight

- As long as you keep fixing it in code, you will eventually win - it's like a video game, as long as you can respawn, you will always defeat the game. Just keep buggering on.

NKosmatos• 7 years ago

I came here expecting to read about how Apple is (mis)treating their macOS lately or how Microsoft has managed to fuck up their windows update process, but instead I find a few good tips while building and numbering software.

tlocke• 7 years ago

For Python projects I use the excellent Versioneer:

https://github.com/warner/python-versioneer

It means you only specify the version as a git tag, and all the other things that need a version number get it from that.

kissgyorgy2• 7 years ago

A thousand ways. One really good idea I read was that you write an empty script, which prints the instructions for each step and you press enter, it prints the next step. It can only print instructions at first, then later, you can write code to automate some (maybe even validate things) steps. Something like this:

    def git_tag():
        print("1. git tag the repository")
        input("Press enter to continue")

    def git_push():
        print("2. git push the repository")
        input("Press enter to continue")

    def pip_deploy():
        print("3. Deploy, run: python setup.py sdist upload")
        input("Press enter to continue")

    def main():
        git_tag()
        git_push()
        pip_deploy()

As simple as that! Then later, you could write code for every step as you have time.

x1798DE• 7 years ago

print("3. Deploy, run: python setup.py sdist upload")

The setup.py upload command is actually deprecated and has been replaced with "twine upload".

https://setuptools.readthedocs.io/en/latest/setuptools.html#... https://pypi.org/project/twine/

silvester23• 7 years ago

I think this might be the article you read:

https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...

kissgyorgy2• 7 years ago

Yes! Thanks! I couldn't find it.

blesi• 7 years ago

At Braintree, we developed a framework for this:

https://github.com/braintree/runbook

One of our biggest use cases is for including ample sanity checks as part of our releases to make sure we don't mess it up.

yitchelle• 7 years ago

Don't do it on Friday.

On a serious note, I have found that the only way to not fuck up is to remove as much human interaction from the release process as possible, which means scripting. Even with scripting, if the inputs are bad, the output will also be bad.

mattacular• 7 years ago

That's the point of this post. Automation is hard and probably won't ever be perfect. You can't remove human interaction from software development.

thrower1232• 7 years ago

Seriously, though, don't release on Friday. No matter how good your CI/CD pipeline is, and how confident you are in your testing, you'll let some bad releases through, and if you release on Friday, some poor bastard, whether you or your coworker, is going to be left holding the bag, and have their weekend ruined scrambling around to fix what is broken.

It's just inconsiderate. Things can wait until Monday.

Intermernet• 7 years ago

I'm waiting for the next "disruptor" to make it a sales pitch: "Move fast, break things, and only release on Friday!"

pnako• 7 years ago

In fact it's standard practice in the financial industry to deploy everything on the week-end, and indeed starting from Friday night, because that's when most markets are closed.

Obviously don't do that if you don't have to, but sometimes it's actually the best time to deploy stuff.

caseymarquis• 7 years ago

Automating builds is fun! On the one hand I'd be worried if I joined a team or contributed to a project which still had manual processes for release more complex than 'run this script' or 'push to the x branch'. On the other hand, that's a fun initial project.

When developers aren't (or are prevented from) improving their own environment, that's a bad sign...

_57jb• 7 years ago

I love the humble nature of this post and how fallible we are.

I have spent many days playing whack-a-mole with issues and hacking out solutions. What is equally great is the community sharing their gotchas and how they solved them.

A wholesome and organic thread, thanks for taking the time to write it!

mugsie• 7 years ago

I really like how the OpenStack project does releases - it has a ton of tooling, but for people who are deciding what to release, it is super simple.

e.g. - the latest RC of a project I maintain is L10-14 on https://opendev.org/openstack/releases/src/branch/master/del...

by committing that file, the tooling chooses the right commit, does a tag, and builds the python tarballs, and puts them on releases.openstack.org, and for the client libraries, it pushes them to pypi.

it avoids a lot of the footguns of special scripts in repos, while allowing an easy release process.

thedancollins• 7 years ago

Phft, amateur. I've forgotten more ways to screw stuff up than this person ever knew.

RexM• 7 years ago

This reads like the "Deploying on a Friday Afternoon Greatest Hits".

aepiepaey• 7 years ago

Update the signing script to save the tarball to disk (previously, it lived in a pipe) and upload these alongside the releases…

This should have been done from the start.

GitHub does not guarantee checksums for the generated source archives to be stable, so they can change when GitHub updates their software (and yes, this has happened).

jillesvangurp• 7 years ago

A couple of practical solutions here.

1) Automate your releases and run that on a CI server if you can. "It works on my laptop" is not ideal for releasing stuff.

2) Have a release branch, and only allow stuff to be merged that passes your CI tests and trigger the automation under step 1 after a merge to your release branch. This works for continuous deployment but can also work for releases of other stuff. If you use semantic versioning, the CI server should tag the release and publish the artifacts. That's one less thing that can go wrong. If you forgot to bump the version number on your master branch, the build should fail.

3) Have a release checklist. People forget stuff and having a small list of "Are the release notes and readme updated? Is our CI not complaining? Have all PRs been merged? Etc. Even with the above, premature releases through an early merge to your release branch could happen. A checklist can prevent that. And of course, if you can integrate those checklists into your CI. If your release notes are unchanged since the last version

4) Release often. Small deltas are less work to test and less risky to impose on users and you get feedback on stuff you did earlier.

How to fuck up software releases

Related Stories