Deployment should be boring

There is a failure loop most teams recognise. Deploying is risky, so you deploy less often. Deploying less often means each release contains more changes. More changes means more risk. Round it goes.

The way out is not being more careful. It is making deployment so routine that frequency stops being a decision.

Automate the deploy itself

Manual deployment means someone remembering a sequence of steps under pressure. Every manual step is a step that can be skipped at 11pm on a Friday. A pipeline that runs tests, builds and deploys on a merge removes that entire class of failure.

Make rollback a non-event

The question is not whether a bad release will ship — it is how long it takes to undo. If rollback takes one command and under a minute, a bad release is an inconvenience. If it means restoring a database backup, it is an outage.

Practise it. A rollback procedure nobody has run is a document, not a capability.

Alert on things that matter

Alert fatigue is real. If the team receives twenty notifications a day, they will miss the one that counts. Alert on user-visible symptoms — error rates, response times, failed jobs, disk filling up — and route everything else to a dashboard.

Test your backups by restoring them

An untested backup is a hypothesis. We have seen backups that ran nightly for two years and produced unrestorable files, discovered at the worst possible moment. Restore into a staging environment on a schedule and confirm the data is actually there.

Write the runbook

One document: how to deploy, how to roll back, how to restore, who to call. It matters most when the person who set everything up is unavailable — which is precisely when incidents feel hardest.

None of this is advanced. It is just rarely prioritised until after the outage that makes the case for it.