The answer that sounded right
A bid manager at a Gauteng engineering firm asks an AI assistant to summarise the compliance requirements for a municipal tender. The reply is fluent, structured and confident. She pastes it into the bid document. Two of the stated requirements are wrong, and one required certificate does not appear at all. The tender is submitted, then disqualified on a technicality that took ten minutes to check and eleven weeks of effort to lose.
Nobody was careless in the usual sense. The tool produced an answer that looked finished, and the person using it was busy. This is the pattern behind most costly AI errors in business: not a dramatic failure, but an unchecked fact that moved quietly into a document, a quote, a contract or a customer message.
Why AI answers are easy to trust and hard to verify
Generative AI tools produce text that is probabilistic: they predict plausible wording, not verified truth. The NIST AI Risk Management Framework and its companion generative AI profile name confabulation, where a system produces confident but false output, as one of the risks specific to generative systems. Tone and fluency do not signal accuracy. A wrong answer reads exactly like a right one.
- Confidence is uniform. The tool rarely hedges, so the reader cannot tell which statements are solid and which are invented.
- Specifics are the weak point. Figures, dates, clause numbers, legal references and names are the details most likely to be wrong and most costly when they are.
- Checking feels like undoing the benefit. The reason for using the tool was speed, so a full verification step seems to defeat the purpose.
- Errors are inherited. Once a wrong figure is in a quote or report, others copy it, and it stops looking like an AI answer at all.
Where confident mistakes cost the most
- Tenders, quotes and proposals: wrong specifications, prices or compliance statements.
- Contracts and policies: misquoted clauses, wrong notice periods or misunderstood obligations.
- Tax, payroll and compliance: outdated rates, thresholds and filing rules. Rules published by SARS and other bodies change, and a model can describe a version that has been superseded.
- Customer communication: promises, prices or policy statements sent to customers. The related article on AI-drafted customer messages sent without review covers this in detail.
- Management reporting and analysis: a summary or calculation that looks authoritative but misstates the data. Poor inputs make this worse, as set out in AI on poor-quality data.
- Technical and safety information: specifications and procedures that are plausible and wrong.
How the pain becomes money
- Lost or voided business: disqualified bids, rejected quotes and contracts signed on wrong terms.
- Rework: corrections to documents already circulated, often at the point where reversing the mistake is hardest.
- Penalties and compliance cost: wrong tax, labour or regulatory assumptions. Where personal information is involved, POPIA obligations also apply, and the Information Regulator's POPIA page explains the duties of the responsible party regardless of what tools produced a decision or a message.
- Customer trust: an incorrect statement to a customer or supplier is costly to repair.
- Senior time: managers who find one error begin to re-check everything, which erases the productivity gain.
An illustrative cost model
The figures below are illustrative assumptions for a hypothetical 50-person professional services and engineering business in which about 15 staff use AI tools for drafting and research. They are not measurements from any real business.
| Assumption | Value |
|---|---|
| Staff using AI for work content | 15 |
| Documents per person per month containing AI-supplied facts | 12 |
| Share containing a material factual error that goes unchecked | 2 percent (assumed) |
| Share of those errors that cause a cost | 25 percent (assumed) |
| Average cost of an error that bites: rework, concession or lost work | R18,000 |
| Checking time that would catch most errors | 10 minutes per document at R240 per hour |
- Documents a year: 15 × 12 × 12 = 2,160
- Unchecked errors a year: 2,160 × 2 percent = about 43
- Errors that cost money: 43 × 25 percent = about 11
- Cost of those errors: 11 × R18,000 = about R198,000
- Cost of checking every document: 2,160 × (10 ÷ 60) × R240 = R86,400
- Illustrative net saving from a checking habit: about R111,600 a year
The assumptions can be argued with, and that is the point. Replace them with your own and see whether the arithmetic favours checking. In most businesses it does, particularly if checks are focused on documents that carry money, legal or customer consequences rather than applied to everything.
Why the pain is tolerated
The first successes are visible and the failures are not yet attached to the tool. An error found later is blamed on the person who wrote the document, not on the process. Managers are pleased with the speed and do not want to slow it down. Staff fear looking slow if they say they spent time checking. And nobody has defined what must be checked, so checking becomes a matter of personal judgment.
When it becomes commercially urgent
- a bid, quote or contract is rejected or disputed because of a wrong AI-supplied detail;
- a customer receives incorrect information about price, delivery or policy;
- a tax or labour compliance error is traced back to an AI answer;
- a professional body, client or insurer asks how AI output is reviewed;
- usage grows from a few enthusiasts to the whole business.
What better looks like
The aim is not to forbid the tools. It is to apply human checking where an error would be expensive. The design of those checks is covered in designing human approval checkpoints for AI workflows, and the principle that checking should be proportionate is in how to reduce human error without removing human judgment.
- Classify the content. Decide which outputs are low-risk drafts and which carry money, legal, tax, safety or customer consequences.
- Name the facts that must be checked. Figures, dates, names, legal and regulatory references, and anything quoted to a customer.
- Check against the source. Verify against the original contract, the official SARS or regulator publication, the supplier price list or your own records, not against another AI answer.
- Ask for sources, then open them. A cited source that does not exist or does not say what is claimed is a clear warning.
- Record who checked. A simple sign-off on important documents makes responsibility clear.
- Learn from the catches. Keep a short list of errors found and share it, so staff learn where the tools tend to go wrong.
Practical next actions
- Ask each team where they use AI output in documents that leave the building.
- Pick the three document types where a wrong fact would cost the most and define a mandatory check for each.
- Run a one-week test: have a colleague verify the facts in every AI-assisted document and count the errors.
- Write a half-page rule on what must be checked before AI content is used.
- Review the first month of catches and adjust the rule.
Bottom line
AI tools make drafting faster, but they do not make facts true. The cost of a confident mistake is paid later, in rework, lost work and damaged trust, and usually by someone other than the person who used the tool. A short, targeted checking habit costs far less than the errors it prevents, and it lets the business keep the speed without gambling on the details.
Sources and further reading
- AI Risk Management Framework (last verified 10 October 2026)
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (last verified 10 October 2026)
- https://inforegulator.org.za/popia/ (last verified 10 October 2026)
Related Mellorca reading
- Automating the Easy Things First: The Opportunity Cost of Leaving Bottlenecks Alone
- Gathering Information Before Work Can Start: The Payroll Cost of Preparation
- Predictable Appointments Still Carry a Cost When Staff Arrange Every Slot