Introduction
Everything works until the morning it does not. A calendar goes down, a form changes, a connection expires, a supplier moves an address, and something that has run quietly for months stops.
The question worth asking before you build anything is what happens next. A good answer is dull. It pauses, somebody is told, the team keeps working, and nothing reaches a customer that has to be taken back.
What does a bad day actually look like?
It is rarely a crash. Most of the time it is a silence.
The dangerous failures are the quiet ones. A step that stops running and does not complain. A message that reports success but was never delivered. A record that saved half its fields. A workflow still going on stale information because the source it reads has not updated since Thursday.
An outage announces itself. A silent failure gets discovered a week later by a customer who never got called back, which is a worse conversation than any outage.
How do you find out that something failed?
By watching for the thing that should have happened, not only for errors.
Error alerts catch loud problems. They do not catch absence. So the system also checks for expected events that never arrived: the nightly job that did not run, the confirmation that was never sent, the enquiry sitting for hours with no owner, the record still waiting on an approval from yesterday.
What should pause, and what can safely keep going?
This is the failure path, and it is the same shape every time we build one.
- The system notices that something expected did not happen, or that a step returned something it cannot trust.
- That step pauses. It does not retry blindly and it does not push on with a guess.
- The affected record stays visible and is marked clearly, so nobody assumes it was handled.
- A named person is told, through a route they will actually see rather than an inbox nobody reads.
- The rest of the system keeps running, because one broken connection should not stop the parts that are fine.
- Nothing resumes until a person has checked what was missed during the pause and repaired it.
The failures worth designing for are ordinary ones: a source that will not respond, an input the system cannot make sense of, the same event arriving twice, an action the system is not allowed to take, an action that arrives too late to be useful, and an output that is simply wrong. Each one needs a decision made in advance about what pauses and who hears about it.
What does the customer see while the normal path is unavailable?
Ideally nothing, because the pause happened before anything went out.
The order matters. Anything customer facing pauses first and the internal work pauses second, so the failure lands on your team rather than on the customer. Where a customer is already mid conversation, they get a plain message that a person is picking it up, and a person actually does.
What we never do is fill the gap with something plausible. A confident message built on information the system could not confirm has to be corrected later, which costs more than the silence would have.
How does the team keep working by hand?
Because the manual path was never removed.
Every workflow keeps one, written in plain language. Where the waiting work is, what the customer was last told, what has to happen next, and who does it. It lives with the runbook rather than in somebody’s head.
The test is simple and worth doing once a year. Turn the workflow off on a normal working day and see whether the business can still take an enquiry, quote a job and book it in. If the answer is no, the automation has become a dependency.
What has to be true before it starts again?
Restoring the connection is the easy half. Somebody has to check the gap it left.
A workflow can come back the moment a connection is restored, but that is not the same as the business being caught up. Before it resumes, somebody confirms what did not happen during the pause, catches those customers up by hand, and makes sure the restart will not repeat actions that already went out. Duplicates after a recovery are the most common second failure.
Then the part most providers skip: working out what would have caught this sooner, and changing something so the next one is noticed in minutes.
What should you ask any provider to show you?
Three things, before anything goes live.
The off switch, demonstrated by you on the actual system rather than described. The runbook, written so somebody who did not build it can follow it: what each workflow does, what it touches, who owns it, what to do when it stops. And the alerting, including the check for the thing that should have happened and did not.
Australian government guidance for businesses adopting AI puts it in similar terms. Keep a way to override or stop the tool, and keep an alternative path so critical work can continue if the system is unavailable or retired.
A provider who cannot show you all three is selling you a dependency on them.
FAQs
How would we know if something failed silently last month?
You would not, unless something was watching for the absence. That is why those checks get built at the same time as the workflow. Going back afterwards to work out what quietly did not happen is slow and usually incomplete.
Who should be told when something breaks?
One named person per workflow, with a second who covers when they are away. Alerts sent to everybody get read by nobody. The alert should say which record is affected and what the manual next step is, not only that an error occurred.
Does this mean automation is risky?
It means it is operational, like a van or a phone system. Things stop. What matters is that somebody decided in advance what happens when they do. Bring one workflow to an Operations Review and we will map its failure path with you.
SOURCES
CONTENTS