Automation removes routine human touchpoints. That is useful when the workflow is healthy, but it also removes the person who might otherwise notice that a document was not sent, a record was not created or a customer was never routed to the next step.
A loud failure is inconvenient but honest: something stops, someone sees it and it gets fixed. A silent failure is more damaging because the business keeps operating on the assumption that the work is being done. By the time anyone notices, the problem has usually been accumulating for days or weeks.
How silent failures happen
Silent failures are rarely caused by one dramatic fault. They come from ordinary mechanisms:
- The trigger never fires. A form changes, a field is renamed or a filter condition stops matching, so the automation simply has nothing to run against. No run means no error.
- The run succeeds technically but not operationally. A record is created in the wrong state, an email goes to an old address or a required field is left blank.
- Retries exhaust quietly. Microsoft's Retry pattern guidance recommends logging early failures as informational and only the final failed attempt as an error, to avoid flooding operators. That is sensible, but it means the final failure must be routed somewhere a person will actually see it.
- Partial completion. A multi-step workflow completes three steps out of five. The first steps look healthy, so nobody checks the last ones.
- Duplicates rather than gaps. Retrying an operation that is not idempotent can cause it to run more than once, as the Retry pattern guidance notes. The result is duplicate records or repeated notifications rather than missing ones.
No error message is not proof of success
A technical process can complete while the business outcome is wrong. A trigger may run against incomplete data, a downstream system may accept a request without producing the expected state, or a retry may quietly exhaust itself. Monitoring therefore needs to answer both technical and operational questions.
Google's Site Reliability Engineering book draws a useful distinction between monitoring "what's broken" and "why", between the symptom and its cause. It also distinguishes black-box monitoring, which tests behaviour as a user experiences it, from white-box monitoring of internal metrics and logs. Business automations need both. White-box signals tell you that a step threw an error. Black-box, outcome-level checks tell you that the invoices expected today do not exist, whatever the logs say.
Illustrative example (hypothetical, not a client case): a business automates the creation of onboarding tasks when a deal is marked as won. A sales manager adds a new deal stage and moves won deals into it. The automation still runs and reports no errors, but it now watches a stage that is no longer used. New customers wait without onboarding until one of them complains.
Warning signs
- Nobody can say when a given automation last ran successfully.
- Failures are discovered by customers or by month-end reconciliation.
- Alerts go to a shared inbox or to a person who has left.
- There is no count of how many items should have been processed, only of errors.
- Staff keep manual checks "just in case" because they do not trust the automation.
- Duplicate records appear periodically without explanation.
Observe the business result
- Was the expected transaction created?
- Did the item reach the next workflow state?
- Are failure and retry counts visible?
- Can the team identify work that is stuck?
- Is there an owner and a recovery procedure?
The most reliable signal is often a simple comparison of expected against actual: orders received today against invoices created today, or new signed customers this week against onboarding records created. A gap between the two is a symptom worth investigating, regardless of what the logs say. The SRE book's guidance on alerting is useful here: an alert should detect a condition that is urgent, actionable and otherwise undetected, so that people learn to trust it rather than ignore it.
A practical diagnostic
For each automation the business depends on, answer the following:
- What business outcome is it responsible for, in one sentence?
- How would we know, within a day, if that outcome stopped happening?
- Who receives the alert, and do they know what to do with it?
- What happens to the work when the automation fails: is it queued, lost or duplicated?
- Can we safely re-run it for the items it missed without creating duplicates?
- When was it last tested against a realistic failure?
Any automation where questions two to five have no clear answer is a candidate for attention. Our pieces on who watches the automations and why every automation needs a failure path go further into ownership and design.
Design for recovery before production
A business-critical automation should have an explicit failure path, bounded retries, traceable events and a way to reconcile what should have happened with what actually happened.
- Bounded retries. Retries should be limited and, for persistent faults, should stop and report rather than continue indefinitely. The Retry pattern guidance warns that an aggressive policy can further degrade a busy service.
- Safe re-running. Microsoft's Idempotent Consumer pattern describes designing processing so that handling the same message more than once has the same effect as handling it once, using a stable key to record what has already been processed. The same principle applies to scheduled jobs and webhooks that may run twice.
- A place for failed items. Work that cannot be processed should land in a visible queue for review rather than disappear.
- Traceability. Each item should carry an identifier that can be followed across steps and systems.
Trade-offs
- Too many alerts are as bad as too few. Noisy alerting trains people to ignore it. Alert on business outcomes and final failures, and log the rest.
- Monitoring is ongoing work. Someone has to own dashboards, alerts and recovery procedures after launch. That cost belongs in the automation's business case.
- Not every automation needs the same rigour. A convenience notification can fail without much harm. Automations that touch money, customers or compliance deserve stronger controls. See what service levels make sense for business-critical automations.
What better looks like
Automation becomes dependable when success is observable and failure is actionable. The operating team can see exceptions early, understand their impact and recover without reconstructing the entire workflow from logs and memory. Each important automation has a named owner, an outcome-level check, a route for failed items and a tested way to re-run safely. Where a business relies on many automations, that operating discipline is often best treated as a service in its own right, which is the role of Managed Digital Operations.
Sources and further reading
- Google SRE monitoring ditributed system - sre golden signals (last verified 10 October 2026)
- Retry pattern - Azure Architecture Center (last verified 10 October 2026)
- Idempotent Consumer Pattern - Azure Architecture Center (last verified 10 October 2026)
Related Mellorca reading
- How to design retries without creating duplicate work
- Why automation documentation matters
- How to design automations that people can understand