Your AI chatbot needs a useful ‘I don’t know’
%20(1).avif)
Written by Derrick KityoWhat should an AI feature do when its sources don't contain an answer? A worked support example, response choices and a review pack for testing the gaps.
Your AI chatbot needs a useful way to say it doesn't know. If the help pages don't contain a refund policy, a fluent guess doesn't give the customer the answer they need. The product should explain the gap and offer a next step it can actually carry out.
I think this belongs in the first version of the feature. It's part of the experience somebody gets when their question is reasonable and the available information isn't enough.
An AI hallucination is a plausible-looking answer containing an incorrect or unsupported claim. You can try to reduce those answers, but the interface still needs a plan for missing information.
Write the unanswerable questions first
For a small support feature, I'd write a few questions the product should be able to answer, then add the ones it shouldn't answer yet. That second group makes the scope much clearer.
Take a fictional booking tool. Its help pages explain how to invite a colleague and move a booking. They don't supply a refund policy. They also don't give the chatbot access to an individual account.
“How do I invite a colleague?” has a route to an answer in the supplied help. “Can I get a refund?” doesn't. “Has my cancellation gone through?” needs an account record, even if there's a perfectly good article explaining how cancellation works.
These are authored review examples, not transcripts from a tested model. I've put the fictional source material and expected behaviours into the download so you can inspect the distinction.
Download the AI answer review pack
A useful fallback says what's missing
Here's sample wording for the refund question:
The help pages I can access don't include a refund policy. I can't confirm whether this payment is refundable. You can send the billing team a question using the support form.
That wording only works if the support form exists and reaches the billing team. If the product can prepare a message but can't send it, say that. “I've escalated this” would be another unsupported claim if nothing had actually happened.
Keep the gap specific. Is the policy missing? Is the question unclear? Is a record unavailable? A generic apology leaves the person guessing which part to fix.
The response doesn't need a long explanation of the model. It needs enough detail for the person to decide whether to rephrase, open a source or ask someone else.
Sometimes the next step is a question
“Can I cancel it?” could mean a booking, an invitation or the account itself. Answering the most common interpretation would be a risky shortcut if the actions have different consequences.
I'd ask which thing they mean. Keep the follow-up tied to the decision you need to make. Asking for their company size or phone number wouldn't resolve that ambiguity.
This is different from a missing policy. If the user means the account, the help might still have no account-cancellation guidance. Clarifying the question doesn't create a source that wasn't there.
The interface should be able to move from clarification to a grounded answer or a useful fallback. It shouldn't get trapped asking variations of the same question.
Finding a related document isn't enough
A retrieval system searches a collection of sources and supplies relevant material to the model. You'll often see that described as retrieval-augmented generation, or RAG.
It can help the answer use the business's information. It doesn't establish that a retrieved paragraph supports every claim in the reply. A page about changing a booking may be closely related to a refund question while saying nothing about refunds.
Anthropic's work on contextual retrieval looks at improving the material retrieved from a knowledge base. I'd still review the answer against the actual passage, including any limits or conditions it contains.
A citation gives the person something to inspect. The title of the linked page isn't proof. Open it and check whether the relevant sentence is present, current and applicable to this question.
The source can also be the wrong version, or available to the wrong audience. Access checks belong before restricted material reaches the model or the customer. A polite refusal after private content has already been exposed is too late.
Want to apply this to your own website or app?
Tell me what you are working on and where it is getting stuck. I will help you turn the useful ideas into shipped work.
A prompt is one part of the behaviour
Giving a model permission to admit uncertainty is a sensible starting instruction. Anthropic includes that advice in its hallucination guidance, alongside grounding and checking claims.
I'd also define what the surrounding product does when supporting evidence is absent. Which answer states can it show? Does the reply need a source? Where does an unresolved question go? Who owns the missing help content?
A reported confidence score doesn't settle those product decisions. Before relying on one, I'd establish what it measures and test how it relates to correct answers.
The paper Why Language Models Hallucinate argues that common training and evaluation incentives can reward guessing over acknowledging uncertainty. For a support product, I'd make the review criteria explicit about when declining to answer is the correct result.
Test whether it also refuses the easy questions
There's a failure in the other direction too. A chatbot that declines every question may avoid making unsupported claims while leaving the customer with no useful help.
Keep answerable questions in the same review set. Include different phrasings, an obvious typo and a follow-up that needs the previous message. Check that the system still finds and uses the available help.
For the fictional booking tool, the invite question should lead to the invite instructions. A vague “please contact support” would be a poor result when the source already contains the answer.
Review the content of each reply, not just whether it contains a link or an apology. The case pack separates supported answers, clarifying questions, missing information, account access and conflicting sources. Those are behaviours to test in your implementation; the pack doesn't call a model or report a pass rate.
The missing answer creates work somewhere
If people keep asking a question the help doesn't answer, someone should look at why. The source might need updating. The question might need a human decision. The product might need an account action the chatbot can't perform.
I'd give that work an owner. Record enough of the question and its context to understand the gap, using the data handling rules appropriate to the product. Don't create a second inbox of unanswered questions that nobody reads.
When the source changes, rerun the relevant review cases. Check that a previously unanswered question can now be answered, and that an older answer hasn't survived after its support was removed.
Download the response and handoff review notes
If you're scoping an AI feature, my AI development page covers the work around the interface and systems. For the updates that follow, the website and product backlog gives those questions somewhere to become assigned work.
Source and example notes
I checked the linked research and documentation on 20 October 2026. The booking product, source documents and replies here are fictional review fixtures. They are not measured outputs from a live chatbot, and no accuracy or conversion result is claimed.
Frequently asked questions
What should an AI chatbot do when it doesn't know the answer?
Explain the specific information it lacks and offer a next step the product can actually provide. That may be a clarifying question, a source to inspect or a genuine human handoff. Avoid inventing a policy or claiming an escalation happened when it didn't.
Does adding retrieval prevent AI hallucinations?
Retrieval supplies relevant source material to the model, but a related passage may not support the answer being asked for. Review the actual claim against the passage, its version and its applicability. Test missing and conflicting information too.
Is a prompt telling the model not to guess enough?
Treat that instruction as one part of the feature. Define and test what happens when evidence is missing, a question is ambiguous or an account record isn't available. Keep answerable questions in the test set so the feature doesn't send every question to support.
Does the downloadable review pack measure a chatbot's accuracy?
No. It contains fictional source documents, questions and expected behaviours for you to adapt. It doesn't call a model or produce a pass rate. Run the cases against your own implementation and keep the answers and review evidence.
.jpg)
