Who should own it?
Australia's National AI Centre advises making a specific person accountable for every AI system an organisation uses. For a knowledge base, that is usually an operations or knowledge manager who understands the documents and the questions, with authority to ask document owners for fixes. The owner does not need to be technical. They need time set aside each month, a list of document owners, and a contact at the supplier or IT team for indexing and access problems.
- System owner: accountable for accuracy, scope and the review routine.
- Document owners: keep their documents current and confirm which version applies.
- Technical maintainer: indexing, connectors, permission syncing and model settings.
- Escalation reviewers: answer questions the documents do not settle.
What should be checked every week?
Read the log of questions that returned no answer or a low-confidence answer. Those entries show missing documents, unfamiliar wording and requests outside the agreed scope. Sort each one into a fix for the document owner, a configuration or glossary change for the technical maintainer, or a deliberate decision to leave it unanswered. Check the indexing run history for failed files as well, because a document that stopped being read produces no error in the answers themselves. It simply stops appearing.
- The question as asked, without unnecessary personal details.
- What the system returned and which sources it offered.
- The decision: a document fix, a configuration change or out of scope.
- Who owns the fix and when the result was checked.
What should be checked every month?
Re-run the fixed set of test questions and compare the results with the previous run. A drop on a group of questions can trace back to a document change, a new source or a configuration change. The National AI Centre recommends monitoring AI systems after deployment to detect changes in performance and behaviour, and notes that AI systems can give different answers over time. Add new test questions when staff raise new topics, and keep the old ones so you can see whether a change broke something that used to work.
How do deletions and updates reach the index?
Indexers such as Azure AI Search pick up new and changed files on a schedule, which can run from every 5 minutes to every 24 hours, or only on demand. Deletions work differently. Microsoft states that its storage indexers detect changes automatically but do not track deletions unless a deletion detection policy is in place, and that the policy has to exist from the first run. Without it, a deleted or superseded policy can stay in the index and keep appearing in answers. Test this by archiving a document and asking about it.
What happens when staff or permissions change?
When someone changes role or leaves, their access changes in Microsoft 365 or Google Workspace, and the knowledge base has to follow. Microsoft notes that in Azure AI Search, permission changes in the source only appear in search results after the permission metadata is synchronised, and that some SharePoint changes inherited from a parent site, library or folder need an explicit refresh. Ask your technical maintainer how often permissions synchronise, and test after any restructure. Microsoft also suggests running SharePoint permission snapshot reports quarterly and activity reports monthly.
What should happen after a model update?
Providers update and retire the models behind AI products, and answers can change in length, style and accuracy as a result. Re-run the test questions after any announced model change, and before switching models in a custom build. Record which model version was in use for each test run, so a change in results can be traced to its cause. If accuracy drops on important questions, the owner decides whether to delay the change, adjust the configuration or accept the new results.
What records should the owner keep?
Keep a simple log that a successor could pick up. Record each test run with its date, model version, number of questions and results, along with the weekly no-answer decisions, documents added or archived, and permission changes tested. The National AI Centre recommends keeping clear records of the actions taken under each practice, because good documentation supports audits, reviews and ongoing improvement. If the owner leaves, the log is what lets someone else continue the routine without starting again from nothing.