Don't deploy on Friday and 3 other unwritten rules of software engineering

https://clubhouse.io/blog/dont-deploy-on-frida-3-other-unwritten-rules-of-software-engineering

by ben-clubhouse • 7 years ago

34 21 7 years ago

aliswe• 7 years ago

"Slack is famous for its role in increasing connectivity between team members, but, at what cost (aside from the monthly fee)?"

And what's bikeshedding?

g-erson• 7 years ago

The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed.

-- https://en.wiktionary.org/wiki/bikeshedding

Sarki• 7 years ago

Point "2. Maintain regular backups." should rather be "2. Maintain regular backups and validate your restoration process."

How painful it can be when you realize that your backup process was wrong from the beginning and you're left with a useless gigabytes sized blob when in need to restore your DB...

curtis3• 7 years ago

A corollary to "Don't deploy on Friday": "Don't deploy on Monday."

Unless you really want to work over the weekend.

lojack3• 7 years ago

Why would deploying on a Monday result in working over the weekend?

blacksmith_tb• 7 years ago

I assume preparing for the deploy? But of course that could also have happened in the previous week, and just be ready to deploy on Monday...

curtis3• 7 years ago

I guess it depends on why you're doing the deploy. If the whole point of doing the deploy is to pick up new code changes, then the people responsible for doing those code changes run the risk of having to work over the weekend to make sure they're ready. This can be a big problem at (for example) startups where the same engineers are responsible for writing new code and doing deploys. But it can happen at organizations of all sizes.

If all you're doing is rolling out a new version of Postgres and it's supposed to just work, then a Monday deploy makes perfect sense. However, as a software engineer, the primary reason I've been involved in deployments is because we've written new code and we need to release it. In an ideal world, the code would be completed and rigorously tested by no later than Friday at 5pm. In the not-so-ideal world we actually live in, this isn't always true, which is why I prefer to target deploys for Wednesdays.

esrauch• 7 years ago

"Don't have a deployment process that takes a day ahead"

koneles• 7 years ago

It's funny because I'm reading this from my work, instead of doing somethink productive

clintonb2• 7 years ago

If reading this increases/reaffirms your knowledge of good practices, that is productive.

cphoover• 7 years ago

hmm I dunno... If you have CD working correctly you should be shipping multiple times a day depending on the product. If there is a problem with the deploy it should be as simple as rolling back to the container image of the last version...with K8s or ECS this takes seconds.

I really hate the whole "let's plan for a midnight deploy" paradigm... and find it very unconstructive.

sbergot2• 7 years ago

How do you rollback a DB schema change?

clintonb2• 7 years ago

You deploy in phases. You only remove data in the final phase. I wrote about this at https://engineering.edx.org/django-migration-donts-f4588fd11....

tracker12• 7 years ago

You can absolutely push to development, qa, uat, demo even... just don't deploy to production.

karmakaze• 7 years ago

This point seems targeted more toward those who deploy infrequently, e.g. weekly deploy on Tuesdays.

When working on a few microservices with several deploys per service per day per programmer-pair, not deploying on a Friday means a ~20% productivity reduction. We had one of thosr days as there was an important demo day and our work got blocked in staging. When it finally does deploy to prod, you're exposing your users to all the changes at once and if there's an issue it's much harder to isolate the cause.

Our basic rule is don't deploy and leave any day of the week. If something does happen on the weekend, we're unofficially on-call via slack. I can only remember this happening maybe twice in two years.

ogn3rd• 7 years ago

It's called Read-Only Friday for a reason. Unless of course you enjoy spending your weekends debugging and rolling back. No thanks.

Don't deploy on Friday and 3 other unwritten rules of software engineering

Related Stories