Will these action items prevent the next one?

Model review

Request changes. The timeline is good; the root cause is wrong, so none of the action items would have prevented this.

The root cause. The migration was not slow. Adding a nullable column without a default is a catalog change that takes about a millisecond; the timeline itself shows it completing “instantly” at 14:16. The outage was a lock queue: the migration waited, with no time limit, for the dashboard’s 25-minute query; every checkout query then queued behind the migration. The real causes are an unbounded lock wait, and a long analytical query running on the primary with no statement limit.

Item 1, more connections: nothing was short of connections. Every one was held by a request waiting in the lock queue. With 1,000 slots, 1,000 requests would have queued instead of 200, each with its own server process, and the outage would have been the same length. Connections ran out as a symptom.

Item 2, more CPU: the database was idle. Every waiting session was waiting on a lock, not computing. Twice the CPU does not make a lock arrive faster.

Item 3, night-time migrations: smaller blast radius, same mechanism. Nightly reports, backups and batch jobs run at exactly those hours, and fewer people are awake to notice. It also makes deploying slower and riskier for everyone, permanently.

Item 4, training: the engineer ran a correct, safe migration. Nothing in the process would have stopped anyone else doing the same. Postmortems are meant to fix systems, not people; an action item naming one person also makes everyone else less willing to write honest postmortems.

What I would ask for instead:

1. Bound the wait. Every migration runs with lock_timeout of a few seconds, and the migration tool retries with backoff. This alone turns the incident into a failed deploy step.
2. Bound the dashboard. Give the reporting role a statement_timeout, and move long analytical queries to a replica.
3. See it sooner. Alert when any session has waited on a lock for more than 10 seconds, and when the oldest open transaction is older than a few minutes. Both are single queries on pg_stat_activity.
4. Keep a way in. Give on-call and monitoring their own role with reserved connection slots, and put the “who is blocking whom” query (pg_blocking_pids) in the runbook. Six of the 14 minutes were spent getting in.
5. Rewrite the root cause as the mechanism, without a name in it, so the next reader learns the actual lesson.