Building an AI knowledge base for customer-facing teams

What an AI knowledge base is (and is not)

An earlier version of this page looked at how public AI tools such as ChatGPT and Perplexity gather information about your business. That question still matters, and it now lives in how AI search engines find and recommend businesses. This page is about the other side: building an assistant of your own that answers from your information.

One distinction carries over, and it is the most useful thing to understand. There are two ways a model can "know" something. It can have absorbed it during training, which you cannot control and the model cannot cite. Or it can retrieve it at the moment of the question from a set of documents you provide, then answer from those passages. Customer-facing assistants should work the second way. You are not training a model on your data; you are giving it a well-organized library and telling it to answer only from that library.

That library is the knowledge base. Get it right and a fairly ordinary model gives excellent answers. Get it wrong and the best model available will confidently repeat your outdated pricing page.

Source quality comes first

Start with an inventory of what customers and front-line staff actually rely on: website pages, FAQs, policies, product or service sheets, process guides, help articles, the email templates your best people reuse. Then make three hard decisions.

One source of truth per topic

If the refund policy appears in four places with three wordings, the assistant will find all four. Pick the authoritative version, fix it, and retire or redirect the rest. This is tedious and it is most of the job.

Write for retrieval

Retrieval works on passages, not whole documents. A passage that makes sense on its own gets found and used correctly. In practice that means one topic per document, a clear title, headings that say what follows, the key fact stated near the top, and no "as mentioned above". Long PDFs with information buried in tables are the usual offenders.

Leave things out

Not everything belongs. Draft policies, internal debates, old campaign copy, anything with personal information, and anything you would not want quoted to a customer stay out. A smaller, accurate knowledge base beats a large, contradictory one every time.

For bilingual organizations, check parity. If the French content is thinner or older than the English, French-speaking customers will get worse answers, which is both a service problem and, for some organizations, an official-languages one. Bilingual website planning covers the same discipline for web content.

Permissions: who may see what

Customer-facing teams usually need more than customers should see. Your support staff need the escalation procedure and the internal pricing notes; your public assistant must never quote them. Separate the material into collections before you build:

• Public. Anything already published or safe to publish. A public assistant draws only on this.
• Internal. Procedures, scripts and context for staff. A staff-facing assistant can use it, and staff decide what to pass on.
• Restricted. Contracts, personnel matters, client files, anything with personal information. Usually excluded entirely.

The assistant should only be able to retrieve from the collections its user is entitled to. "We told the model not to mention internal documents" is not a permission system. If the document can be retrieved, assume it can be quoted. Your AI governance checklist should record which collections exist, who owns each and which tools may touch them.

How retrieval works, and where it fails

In simple terms: documents are split into passages, the assistant searches for the passages most relevant to a question, and the model writes an answer using them, ideally with a link to the source. Most bad answers trace back to the search step, not the writing step. The common failures:

• The stale document wins. An old version ranks higher than the current one. Fix: retire old versions, and include dates in documents.
• The answer is split. Half the answer is in one document and the exception is in another. Fix: put exceptions next to the rule.
• Vocabulary mismatch. Customers say "cancel"; your policy says "termination of service". Fix: include the words customers use.
• Nothing relevant exists. The model fills the gap with something plausible. Fix: instruct it to say it does not know and offer a person, then test that it actually does.

Keeping it current

A knowledge base starts decaying the day it launches. Prices change, programs end, staff move, the season turns. Give every document an owner and a review date, and tie updates to the events that trigger them: a price change, a new service, a policy revision, an event calendar refresh. The update has to be part of the change, not a task someone remembers later.

This is the lesson we take most seriously in our own work. On Hey Freddy, the studio's bilingual city guide for the Fredericton region, the Ask Freddy assistant answers against the platform's own guides and listings, and those guides are written and maintained by the studio and events are curated rather than scraped. The assistant is the visible part. The curated content underneath is what decides whether its answers are worth having.

For seasonal organizations (ski hills, tourism operators, festivals), build the calendar into the review schedule. An assistant confidently describing last summer's hours in October is worse than no assistant.

Evaluate before launch and after every change

Before anyone outside the team sees it, build a test set: real questions pulled from your inbox, phone notes and chat logs, each with the answer a good staff member would give. Include awkward ones:

• Questions with a correct answer of "it depends", where the assistant should ask a follow-up.
• Questions it should refuse or hand off: complaints, account-specific requests, anything needing judgment.
• Questions about things you do not offer, where the right answer is "we don't".
• The same questions in French, if you serve French-speaking customers.

Grade each answer for accuracy, completeness, tone and whether it cited the right source. Rerun the set after every significant content change and whenever the underlying model changes. After launch, review a sample of real conversations every week, add the misses to the test set, and fix the source document rather than patching the prompt.

Be careful what you count as success. A conversation that ended without escalation is not necessarily a question answered; the person may simply have given up. Better signals are answers that staff reviewers grade as correct, questions that stop arriving by email or phone once the assistant covers them, and short feedback from users at the end of a conversation. Track all three, and read the transcripts behind the numbers.

Escalation: the handoff to a person

A customer-facing assistant without a good handoff is a wall. Define the triggers before launch:

• The person asks for a human. Always honour it.
• The assistant cannot find a confident answer in the knowledge base.
• The topic is on the stays-human list: complaints, billing disputes, account changes, legal, health or safety matters.
• The conversation shows frustration or repeats itself.

The handoff should carry context, so the person picking it up sees the conversation and does not ask the customer to start over. Outside business hours, say plainly when someone will respond. Then track escalations by topic: a cluster of escalations on one subject usually points to a missing or unclear document, which is the cheapest fix you will find.

One design choice worth copying: Hey Freddy runs one knowledge base and one set of guardrails behind two front doors, the Ask Freddy text assistant and Freddy the Radio Wonder Phone, a number people can actually call. Whatever channels you add later, keep a single knowledge base behind them so answers stay consistent. Whether a customer-facing assistant is worth building for your organization at all is a separate question, covered in AI chatbots for business: when are they worth it? If it is, the knowledge base is where the build effort belongs, and a short workflow audit of your current enquiry handling is the right starting point.