Database Migrations That Can Roll Back at 2 AM

At 2 AM, I was woken up by an alert and discovered that a database migration without a rollback step had dropped a column—and the backup hadn't been made yet. The author spent three hours reconstructing data from logs. This experience gave rise to a set of migration practices that make rollbacks boring. This article is based on the original RSS source and distills actionable operating principles.

Every change comes with up and down steps

Migrations have two directions: up advances the schema, down reverts it. Most ORMs can automatically generate both, but many people delete or ignore the down step, considering it extra burden. The author's rule is hard: if you can't write a down step, don't ship an up step. The reason is that at 2 AM, half-asleep with errors spiking, no one wants to write SQL on the fly—they just want to run one command to restore the schema to five minutes ago.

The down step has another hidden value: it forces you to judge whether the change is truly reversible. Adding a column is easy to revert—just drop it. Renaming a column is actually two operations disguised as one, and the naive approach will lose data on rollback. Dropping a column has no honest down, because the data is already gone. When you write "down: add the column back" and then realize there's no data to fill it with, that realization is the whole point—the down step is a test of change safety. Better to die at your desk than die in production.

The author tests before deployment: up, then immediately down, then up again. If the schema doesn't return to a usable state after one round trip, the migration is not acceptable. Common triggers include dropping a foreign key on rollback without recreating it, or an index lingering after rollback and conflicting with the next up.

Database Migration is a versioned script describing database schema changes, and a concrete practice of Code as Infrastructure. Every schema change (adding tables, adding columns, changing indexes, changing types, etc.) corresponds to a migration record, and tools execute them in order to ensure environment consistency. Mainstream tools include Rails' ActiveRecord Migration, Django's migrations, Flyway, Liquibase, Alembic, golang-migrate, and others.

The core contract of a migration is that it comes in pairs: up defines how to advance the schema to a new version, and down defines how to return to the previous version. Many ORMs can automatically derive down from model differences, but they are often helpless or generate incorrect down for data-type changes (such as dropping columns or changing types), and this is exactly where human review is most needed. Putting migrations under version control, CI, and code review is the prerequisite for making rollbacks predictable.

A reliable migration standard usually includes: timestamps in names to guarantee order, prohibiting calls to application model code inside migrations (models deform over time), explicitly declaring lock strategies for operations on large production tables, and running a full up-down-up in a staging environment.

Expand and contract: split one change into four steps

What really saves you is the expand and contract pattern. Don't modify a column in place; split the change into individually safe stages. Take renaming user_name to username as an example. The dangerous approach is a single migration that renames it directly: if something goes wrong mid-deployment, old code looks for user_name and new code looks for username, so one side is always wrong.

The expand-contract method completes the change across multiple deployments in four steps.

Step one, expand. Add a username column next to the old column, with no one reading it yet. This is purely additive, and the down step is simply dropping it cleanly, with zero data loss.

Step two, backfill. Copy the data from user_name into username, and make the application write both columns on every update to keep them in sync. Old code reads the old column, and new code can start reading the new column.

Step three, migrate reads. Deploy code that reads username. The old column still exists and is still being written, so this deployment won't break even if rolled back.

Step four, contract. After confirming the new column is correct and no one reads the old column anymore, wait a few days and then use a separate migration to drop user_name.

Except for the last step, every step is reversible. The only irreversible deletion happens when you are calm and certain, not during a deployment storm. For large tables, backfill in batches to avoid locking fifty thousand rows in a single transaction. The author's original words are blunt: it feels slower, and it actually is slower—four deployments instead of one three-step change—but four small steps take two minutes each, while one big step going wrong costs three hours.

Order of deployment and migration

A single rule that avoids most 2 AM incidents: additive migrations run before deployment, destructive migrations run after deployment.

During deployment there is a window when old code and new code coexist, instances restart one by one, and the database may serve both versions for thirty seconds to two minutes. If a new column is added before the code that uses it, both old and new code are safe: old code ignores the column, and new code finds it already in place. Schema always leads code.

But if you drop a column during the window, old code still expecting that column will crash. So deletion waits until all instances run the new code, and destructive migrations execute last. The order flips with direction: for additions, migrate first then deploy, schema leads; for deletions, deploy first then migrate, code leads.

Rollbacks follow the same logic in reverse. When rolling back a deployment, old code comes back and needs the old schema to still support it—this is exactly why destructive steps wait. The column is still there, and the rolled-back code finds everything it needs. The author puts a line in deployment notes: rollback must work on the currently live schema and must not run any down migrations at all. If it can't, the migration order is wrong.

Failures that need retry, not rollback

Not every migration problem should be rolled back. Long backfills on large tables can fail midway due to dropped connections, lock timeouts, or lagging replicas. If you panic and roll back, you lose the part already completed. A better approach is to make migrations safe to rerun.

The author writes backfills to be idempotent: not "copy all rows," but "copy all rows where the target is still empty." Run once and fill twenty thousand rows, the connection drops and you run it again, fill the remaining thirty thousand and skip the completed ones—no duplicates, no corruption, no need to start over. For transient failures of the migration executor, use retries with backoff and jitter—when you hit a temporary lock, wait and retry instead of failing the whole deployment.

The distinction matters: roll back changes that are structurally wrong, retry changes whose structure is fine but that hit network jitter. Confusing the two magnifies a small problem. The author also sets a statement timeout for every migration: without a timeout, a migration fighting for a lock blocks all queries on the table and drags down the whole application; with a timeout, the migration fails fast, you see the lock contention, and you rerun it in a quieter window. A migration that fails in two seconds is a small matter; a migration that holds a lock for ten minutes is an incident.

Minimal actionable checklist

If you change only one thing: write the down step before the up step, and test the full up-down-up cycle at your desk. This habit catches irreversible changes before they reach production.

Then layer on the ordering rule: additive changes migrate first and deploy after, destructive changes deploy first and migrate after, and deletions wait for several days with their own separate migrations. Rollback should work on the currently live schema without any down migrations. If it can't, the order is wrong.

Expand-contract looks like overkill until the first time it saves you from a 2 AM data loss. After that, it's just the normal way to rename a column. Idempotent, timed out, and rerunnable—these three things keep a shaky migration from becoming a difficult 2 AM decision.

---

The original article mentions that the author integrates logs, retries, and migration rules into the project blueprint, and includes promotional links and disclaimers. This article only distills transferable engineering practices and does not constitute a recommendation for any specific tool or service.

One-line Summary

Split database migrations into small, rollback-safe, rerunnable steps so 2 AM incidents become routine operations.

Paragraph Summary

The article stems from the author's real incident of being woken by an alert at 2 AM and spending three hours rebuilding data from logs. The core claim is to make rollbacks boring: every migration must have a writable and tested down step, and if you can't write a down, don't ship an up. Dangerous changes should use the expand-contract pattern, split into four steps: expand, backfill, migrate reads, and contract, with the only irreversible deletion happening when calm and certain. For deployment order, additive migrations come before deployment and destructive migrations after, and rollback should work on the currently live schema without running down migrations. Only structural errors should be rolled back; transient network or lock issues are solved with idempotent backfills and retries with backoff, and every migration should have a statement timeout to avoid lock table problems.