Ask most organisations where their resilience lives and they will point to a contract: a support agreement, a backup service, an insurance policy. Those things matter. But they only pay out after something has already gone wrong, and they can only restore what the system was designed to be restored.
Resilience is not a product you add. It is a property of the system — of its architecture, its habits and its people — and it is decided long before the first bad day. The only question is whether it is decided on purpose or by accident.
Start by describing a bad day
Before any tooling, write down what “a bad day handled well” looks like. Which services must keep running, even in a reduced form? Which can pause for an hour, and which for a day? Who decides, and how do they find out?
This sounds obvious, and it is surprisingly rare. Without it, every component is treated as equally critical, which in practice means nothing is. With it, architecture choices suddenly have a yardstick: does this design make our bad day better or worse?
Degrade, don’t collapse
Resilient systems have more than two states. Between “working” and “down” there should be deliberate middle modes: read-only instead of offline, cached instead of live, manual approval instead of automation. Each one buys time for people to respond calmly.
Designing those modes is cheap when the system is on the whiteboard and expensive when it is in production. It is the single most effective resilience investment we know of, and it rarely appears in a budget line.
Rehearse the boring version
Incident plans are written for the dramatic version of events. Real incidents are usually mundane: an expired certificate, a full disk, a change that looked harmless. Teams that rehearse the boring version — restoring a backup, failing over a database, rotating a credential — discover the gaps while the stakes are low.
A restore that has never been tested is a hope, not a backup. The same is true of every runbook that has never been followed by someone who didn’t write it.
See it before your customers do
You cannot respond to what you cannot see. Good observability is not a wall of graphs; it is a short list of signals that tell you whether the promises you made — to customers, to regulators, to your own staff — are being kept right now.
When those signals are wired to clear decision paths, detection turns into action in minutes instead of hours. When they are not, the first alert is often a phone call from someone outside the building.
Where to start this quarter
Pick one service that matters. Write down its bad day. Design one degraded mode for it. Test one restore end to end, with someone new following the runbook. Then decide which signal would have told you first — and make sure it reaches a person who can act on it.
None of this requires a new platform. It requires deciding, in advance and in writing, how your systems should behave when things go wrong. That decision is what resilience is made of.
