Short answer
AI language models sometimes produce fluent, confident answers that are false, called hallucinations, because they predict likely text rather than look facts up. A business stays safe by grounding answers in approved information, letting the AI say “I don't know”, requiring a person's approval for anything costly, controlling what data goes into AI tools, and measuring accuracy on real examples.
01Three real cases worth remembering
- Air Canada, 2024. A customer asked the airline's website chatbot about bereavement fares and was told he could claim the discount after travelling. The real policy said otherwise. The airline argued that the chatbot was responsible for its own words. A Canadian tribunal disagreed: the chatbot is part of the company's website, and the company is responsible for everything on it. The airline had to pay.
- The invented court cases, 2023. Two lawyers in New York filed a brief citing earlier cases that ChatGPT had produced. The cases did not exist. The judge fined the lawyers and their firm $5,000.
- Samsung, 2023. Engineers pasted confidential source code and internal notes into ChatGPT to get help. The company then restricted staff use of generative AI tools. Nothing was hacked; staff simply typed company secrets into a public tool.
None of these were failures of exotic technology. They were failures of process: no grounding, no checking, no rules about data.
02Why AI makes things up
A language model does not look facts up. It writes the most probable next words, based on patterns it learned. Most of the time the probable words are true. When the model lacks a fact, it can still produce text that sounds exactly like a true answer, because sounding right is what it was trained to do.
The common kinds of AI error:
- Invented facts: a policy, an offer or a feature that does not exist.
- Wrong numbers: prices, dates, quantities and calculations.
- Fake references: citations, links or quotes that look real and are not.
- Outdated information: what was true when the model was trained.
- Misreading: taking a real document and drawing the wrong conclusion.
- Hidden instructions: text in an email or web page that tells the AI to do something else, known as prompt injection.
03A traffic-light rule for AI at work
| Risk | Examples | Rule |
|---|---|---|
| Green | Brainstorming, first drafts of internal notes, summarising your own meeting notes, adjusting tone | Use freely; a person reads it before anything is shared |
| Yellow | Customer replies about products and delivery, staff answers from the handbook | Only from approved information, with sources; regular spot checks; easy hand-off to a person |
| Red | Prices and discounts, refunds, payments, legal, medical or financial advice, anything irreversible or sent to many people | A person approves every time, or AI is not used |
- Risk
- Green
- Examples
- Brainstorming, first drafts of internal notes, summarising your own meeting notes, adjusting tone
- Rule
- Use freely; a person reads it before anything is shared
- Risk
- Yellow
- Examples
- Customer replies about products and delivery, staff answers from the handbook
- Rule
- Only from approved information, with sources; regular spot checks; easy hand-off to a person
- Risk
- Red
- Examples
- Prices and discounts, refunds, payments, legal, medical or financial advice, anything irreversible or sent to many people
- Rule
- A person approves every time, or AI is not used
Write this table for your own business, put it where staff can see it, and revisit it every few months.
04What should never go into an AI tool
The second risk is quieter: data leaving the business. Whatever you type or upload into an AI tool is sent to the provider. On free consumer plans it may be used to improve models unless you opt out; business and API plans from the major providers do not use it for training by default. Either way, some things should never be pasted into a general AI tool:
- Passwords, PINs, OTPs and API keys.
- Customers' national ID numbers, card details, bank or mobile wallet numbers.
- Other people's personal information beyond what the task needs.
- Unreleased financials, contracts and pricing strategy, unless you use a business plan with clear data terms.
- Source code and system details, unless the company has approved the tool.
05Guardrails for AI that talks to customers
- Ground every answer in approved information: products, prices, policies. Nothing else.
- Allow, and encourage, “I don't know; let me connect you with our team.”
- Never let the AI promise discounts, refunds or delivery dates unless the data behind them is checked live.
- Tell customers they are talking to an automated assistant, and make a person easy to reach.
- Keep a complete log of conversations so mistakes can be found and fixed.
- Test with real past questions before launch, and keep a set of tricky ones, including attempts to fool it, for every update.
- Review a sample of conversations every week.
06A ten-minute safety check of your AI use
- 01List every AI tool anyone in the business uses, including personal accounts used for work.
- 02For each, note the plan (free or business) and what kind of data goes in.
- 03Mark any AI that talks to customers and ask: what is it allowed to promise?
- 04Take one recent AI-written message that reached a customer and verify every fact in it.
- 05Choose one rule to add to your one-page AI policy this week.
Questions
Can hallucinations be eliminated completely?
Not completely. They can be reduced to a manageable level by grounding answers in your own information, limiting what the AI may answer, checking risky outputs and measuring accuracy. The aim is a system where errors are rare, caught early and cheap to fix.
Is a business responsible for what its chatbot says?
In the Air Canada case, a tribunal held that it was, because the chatbot is part of the company's own channels. It is sensible to assume that customers, and regulators, will treat your AI's words as your words.
Is free ChatGPT safe for business work?
For general, non-confidential tasks it can be. For customer data, financial details or confidential plans, use a business plan with clear data terms, follow your policy, and never paste passwords, ID numbers or card numbers.
How do I know whether my AI assistant is accurate enough?
Measure it. Collect real questions with known correct answers, run them, score the results, and set a minimum accuracy before the AI is allowed to answer customers without review.
Last updated