Introduction
The demo always works. It was built to work, run by the person who built it, on an example they chose.
None of that tells you what happens when a customer sends half a message at 11pm, or sends the same enquiry twice, or asks for something you do not do. Testing is where you find that out, and it should happen before a customer is the one finding it out.
What does a demo actually prove?
That it worked that time.
A demo is the best case. Chosen example, clean details, the person who built it driving, everything it depends on available. Your business is not the best case. It is a Tuesday with jobs running late, a customer sending a voice note, and a form that half submitted.
The gap between those two is where most automation stories go wrong, and all of it is testable in advance.
What has to be agreed before testing starts?
You cannot test anything until you can say what passing looks like.
Before we write a single test, four things get written down and approved. The exact job the workflow does, stated narrowly enough that you could check it. What it is allowed to reach, read and change. Which decisions it may never make. What the business should be able to notice afterwards.
Skip that and testing becomes a demo with more steps. Everybody watches it work and nobody can say whether it passed.
Which cases belong in the owner test pack?
Five families. You can build this list yourself from a week of real messages, and you should, because your bad cases are more useful than any generic list.
- Normal. The everyday enquiry. Complete details, in your area, work you actually do. Run several, not one.
- Incomplete. Half a message. A photo and nothing else. No suburb. A phone number with a typo. A name and the word “quote”.
- Unusual. Out of area. Out of scope. A job you refuse. Somebody angry. Somebody at 2am. Something nobody has asked before.
- Duplicate. The same person twice. A form and a phone call about the same job. A reply to a reply.
- Failed. The calendar will not load. The record will not save. The message does not send. The connection is down.
Two more sit underneath. Permission tests, where you check the system cannot reach records it has no business reaching. Handover tests, where a person takes the job over mid conversation and has everything they need.
How do you test the cases nobody wants to think about?
By causing them. You do not wait for the calendar to be unavailable, you make it unavailable and watch what the workflow does.
The result you want is boring. The step stops, the record is marked, a person is told, nothing goes to the customer that would have to be taken back, and nothing is quietly retried until it has happened several times.
A failed case that ends in a clean stop has passed. One that ends in a guess has failed, however sensible the guess looked.
What is shadow mode and why run it?
Shadow mode is the workflow running on real work with the last step disconnected. Real enquiries come in, the system prepares real responses, and nothing goes out. You read what it would have sent.
This is usually the most useful week of the whole build. Real messages are messier than any test you would think to write, and you find out where your own rules were vague before a customer does.
Whoever reviews it marks every item as send as is, send with changes, or do not send, and notes why. The reasons are the valuable part. They are usually a missing rule rather than a broken system.
What should the go or no go decision look like?
It should be a decision a person makes and somebody writes down.
The pack you sign should hold the job, the access, the decisions reserved for people, every test case with expected and actual results, what happened in shadow mode, what is deliberately out of scope for now, what to watch after launch, and who owns it. Our implementation standards set out that record.
Signing the pack is the moment the business decides it is comfortable, and it gives you something to check against later when somebody asks why the system did what it did.
Australian government guidance for businesses adopting AI asks for the same shape: clear acceptance criteria, documented tests, an accountable person authorising deployment, and monitoring afterwards.
What gets watched once it is live?
Testing finishes at launch, which is when watching starts.
Pick a small number of things you would want to know within a day. How many items the system handled. How many came back to a person, and why. How many customers replied in a way nobody expected. How many times the same correction was made.
That last one is the most useful signal in the set. The same correction appearing again and again means a rule is wrong, and rules are cheap to fix while everybody still remembers the build.
FAQs
How long should testing take?
Long enough to run the five families and a real shadow period, and no longer. Most of the time goes into agreeing what passing means, which is work you would have to do eventually anyway.
Can we test with real customer data?
Only with care. Use real messages with the identifying details removed, use your own records, or test with people inside the business. Test records have to be removable, and cleaned up properly afterwards rather than left sitting in the customer database.
What if it fails the tests?
Then it does not launch, and the process worked. A workflow that fails on incomplete or unusual cases usually has a rule missing rather than a fault. That is quick to fix when testing finds it and expensive when a customer does. Bring one real workflow to an Operations Review and we will show you the test pack we would run for it.
SOURCES
CONTENTS