Most infrastructure work ships without a written answer to one question: if this goes wrong, what do we do? Usually the answer lives in someone’s head. That holds until the person is on leave, or the failure is one nobody modelled, or the pressure is high enough that “in someone’s head” quietly becomes improvisation.

We keep a small habit. Before any meaningful change to a client’s estate, we write the rollback plan. First. As a deliverable. Not buried at the end of the runbook, not implied by the change procedure, a document with its own filename, handed over with the design. The people who have to live with the change read it before they read the change.

01 / SectionThe change succeeds. The system fails.

Most changes that go badly are not failures of the change. They are failures of recovery. The change succeeds in a narrow sense, the new CA issues its first certificate, the storage migration completes, the cluster comes up, and then something downstream gives way. An application pinned to the old root. A backup job that no longer runs. A monitoring integration that stops alerting, quietly, which is the worst way for monitoring to fail.

The change succeeded. The system did not.

At that point the conversation is no longer about the change. It is about whether the people in the room can put the old state back: in what order, with what data, at what cost. If that conversation starts at the moment of failure, most of the time you had is already gone.

02 / SectionFix forward, roll back, and the question under both

There is a respectable argument that you should not lead with rollback at all. In the world of frequent, small, stateless deployments, the prevailing wisdom is that rollback is the normal, boring, correct response: revert first, investigate second, keep recovery time low. There is an equally respectable counter-argument that rolling back is sometimes the riskier move: a hasty revert lands you in a state you have not exercised under current load, and once data or a schema have changed, the safer path is often to fix forward and never look back.

Both are right, in their place. Neither settles an infrastructure change.

The reason is that most of what we touch, a certificate authority, a storage platform, a directory, a network core, is stateful, slow to propagate, and trusted by things we do not control. You cannot canary a root CA across every device that trusts it. You cannot un-issue a certificate that clients have already cached. “Revert first” assumes a revert exists and is cheap. “Fix forward” assumes forward is reachable from where you are standing. The rollback plan is how you find out which of those is true before you need the answer.

So the debate is useful, but the honest version of it is not “rollback or fix forward.” It is: at each step of this change, is going back actually an option, and what does it cost? That question has to be answered in writing, in advance. Everything else is a preference.

03 / SectionWhat a rollback plan actually is

A rollback plan is not the inverse of the change procedure. The inverse of “install the new system” is not “uninstall the new system.” It is “return the previous system to a state where it can carry the workload again, with the data that has accumulated since cutover, in a way the downstream will tolerate.” That is a different document, and it has to answer harder questions:

  • What state was the previous system left in at cutover, and is it still recoverable?
  • For how long? What ages out, a snapshot, a lease, a cached trust decision?
  • What data has changed since, and does any of it need reconciling by hand?
  • Which downstream systems need re-pointing, and by whom?
  • At what point is rollback no longer cheaper than fixing forward?

That last line is the one people skip, and it is the one that matters. There is almost always a crossover, a time, or a volume of accumulated data, past which going back costs more than going on. A plan that names the crossover turns a panicked judgement call into a decision you already made calmly, in daylight, with the right people in the room.

04 / SectionThe plan’s real output is a map of doors

Here is the part that makes people uncomfortable, and the reason the exercise is worth more than the document.

Written honestly, most rollback plans reveal that rollback is not actually possible past a certain point, or that it is possible only at a price no one has said out loud. Some steps are two-way doors: you can walk through, look around, and come back. Others open one way only. Once you have distributed a new trust anchor, revoked the old intermediate, or let clients cache the new state, there is no clean way back. Going “back” becomes its own migration.

The temptation is to stop writing when you reach that discovery. We think the discovery is the point. Knowing where rollback ends tells you which steps are reversible and which are not, which means it tells you where to slow down, what to test hardest, and what to keep warm.

And it changes the design. The reason we cross-sign during a CA migration, rather than cutting straight over, is precisely to keep a door open: both chains validate while trust propagates, so a problem becomes a configuration change rather than a recovery operation. That is not extra caution. It is converting a one-way door into a two-way one, on purpose, because the plan showed us we would want it.

Whether a change can be undone is settled at design time, not discovered at 2am.

05 / SectionAnatomy of a plan we’d keep

A rollback plan we like has five parts. None should be longer than it needs to be. Brevity is a feature; a plan no one can read at speed is not a plan.

  1. Trigger conditions. The specific, observable conditions under which we invoke rollback. Not “something seems off”, conditions you could write a monitoring check against.
  2. Decision authority. Who calls it, who is consulted, who is told. By name and by role, with a backup for each. The worst moment to find an ambiguity here is during the incident it describes.
  3. Procedure. The steps, with the actual commands, in order. Tested where the test is cheap. Read by a second pair of eyes who did not write it.
  4. Data implications. What is lost, what is reconciled, and how, including the honest case where some data cannot be recovered, and what we do then.
  5. Communication. What is said, to whom, through which channel. Pre-written if the change is visible to end users.
// rollback-plan.md - required headings, in order

01. Trigger conditions
02. Decision authority
03. Procedure
04. Data implications
05. Communication

// optional, if relevant:
06. Time budget
07. Known gotchas

06 / SectionA template you can take

We keep a stripped-down version of this as a single markdown file: the five headings, a few prompts under each, nothing more. It is deliberately short. The rollback plan template is published as a separate page so you can take it, fork it, or rename it.

If you take one thing from here, take the habit, not the template. Write the answer to what do we do if this goes wrong before you write the change. Some of the time it will tell you the change is safer than you feared. Some of the time it will tell you to stop. Both are worth knowing on a Tuesday afternoon, rather than a Saturday night.