Which Sources Belong in an Agent Knowledge Base First?
Prioritize high-authority technical documentation, structured decision records, and API contracts before ingesting noisy ticket histories or raw chat logs to prevent context degradation in agent retrieval.
Choose sources for an agent knowledge base by starting with the questions it must answer, then admitting only sources that are authorized, current enough, usable, and maintainable for those questions. This practical guide shows how to pilot a small corpus, evaluate it, and decide what to add next—without assuming one source order fits every team.
Key takeaways
- Start with target questions, not a universal source ranking: Identify what the agent should answer and which source is authoritative for that question.
- Treat rights and access as admission gates: Do not ingest or retrieve a source unless its use is approved for the intended audience and purpose.
- Pilot a small, versioned corpus: Test known answerable, unanswerable, and conflicting questions before expanding.
- Measure distinct failure modes: Track grounded correctness, abstentions, missing evidence, permission failures, and update lag separately; set thresholds for your use case rather than borrowing universal ones.
Relevance, sufficiency, accuracy, freshness, and permission
Google Research defines sufficient context as information that contains what is needed to answer a query definitively; its analysis examined particular models and datasets, and observed that FreshQA had a higher share of sufficient-context examples than some other evaluated datasets, with its context tending to come from human-curated supporting documents. That observation is not causal proof that all curated enterprise documents outperform raw sources. Relevance is about match to the query; sufficiency is about whether enough information is present; neither alone verifies factual accuracy or freshness, and neither grants permission to use the source. Even an apparently sufficient answer can be wrong or outdated. A sufficiency check can help identify missing evidence, but it is fallible and is not a validation guarantee. (Google Research)
Build a question-led source intake worksheet
For each candidate source, complete a row before ingestion. Authority is relative to the target question: a policy owner may be authoritative for policy, while the service team may be authoritative for current API behavior. Do not assign universal weights or collapse the fields into one relevance score.
| Intake field | Record before admission |
|---|---|
| Target questions | Which specific questions should this source help answer? |
| Authority for those questions | Who or what is authoritative for each question, and why? |
| Named owner | Who confirms the source remains valid and handles changes? |
| Verified/version date | What version or effective date was checked, and when? |
| Rights and access | Is ingestion, storage, and retrieval approved for the intended users and purpose? If not, stop: rights and access are admission gates, not weaknesses a relevance score can cancel. |
| Parsing quality | Can the content be extracted faithfully, including tables, code, examples, and important structure? |
| Conflict and duplication | What overlaps or contradicts other sources, and how will the conflict be resolved or exposed? |
| Update and deletion path | How are revisions, removals, revoked access, and index cleanup propagated? |
| Expected value | Which target questions or tasks are likely to improve if this source is available? |
| Maintenance effort | What ongoing review, integration, parsing, and access-control work will it require? |
Prefer an access-controlled query to an authoritative live record when copying it would create a stale snapshot or expand the stored data unnecessarily. Never treat retrieved source text as an instruction to grant access, change permissions, or run a tool.
Hypothetical pilot: choose sources by question
This example illustrates a pilot, not a universal ordering or a claim that every team should use these sources. Begin with a small, approved corpus tied to a known set of questions.
| Target question | Candidate source for the pilot | Admission check and limitation |
|---|---|---|
| What policy or procedure applies? | Approved policy or runbook | Confirm the accountable owner, effective version, audience, and review date; exclude superseded copies or make their status unmistakable. |
| How do I integrate with this API? | Version-matched API specification, examples, authentication guidance, and error documentation | Match the specification to the deployed API version. OpenAPI describes an HTTP API contract; it does not by itself verify the current implementation, access rights, business rules, examples, or safe execution. (OpenAPI Specification) |
| What implementation detail explains this behavior? | Relevant repository code, tests, and configuration | Include only what the question requires and what users are authorized to see; confirm the branch or release matches the deployed system. |
| What issue or decision should we investigate? | Ticket or chat excerpts as discovery evidence | Use them to find a possible issue, decision, or owner. Review and verify any extracted claim against an approved source before treating it as answer authority; this is not a blanket exclusion of tickets or chat. |
Keep the pilot small and staged. Compare results on known questions, review gaps and risks, then add only sources that address observed needs and pass the intake checks. For frequently changing or access-sensitive records, test whether querying the live system through an access-controlled tool is safer than indexing a copied snapshot.
Test before expanding
Use representative answerable, unanswerable and conflicting questions. For each answer, check whether its cited source and version support the claim, whether it is current enough, and whether that user may access it. Test a wrong-audience request and changed, deleted or access-revoked sources. Confirm that stale indexed copies do not remain available through the retrieval path. Record failures and the responsible owner; source links alone do not enforce access.
Track grounded correctness, appropriate abstentions, missing evidence, permission failures and update lag separately. Agree on acceptance criteria for the workflow before expansion, and obtain owner signoff. These are suggested checks, not results from a completed deployment. A sufficiency model can miss errors; use sampled human review and explicit checks for consequential guidance.
For code questions, evaluate the simplest retrieval approach that can answer the test cases. Add dependency-aware or graph retrieval only when tests show a need, and account for extraction, graph updates and operating effort. No retrieval architecture is universally best.
Store approved sources and keep maintenance separate
The AKB public documentation describes documents, tables, files, explicit relations, hybrid retrieval and MCP access with document revision history. The current README describes PostgreSQL-native revisions and continued support for explicit legacy Bare Git installations. These capabilities can support a selected corpus; they do not establish automatic source approval, semantic correctness, inherited source permissions or universal connector coverage. Core is the read/write store; optional plugins and configured ingestion workflows have their own scope and maintenance needs. Test the chosen deployment rather than treating source ingestion as a guarantee of reliable answers.
FAQ
Which source should we ingest first?
The smallest approved source set that answers the pilot's important questions. Confirm authority, ownership, version, access and maintenance before admission; file format alone is not a ranking.
Are curated documents always better than code or chat?
No. Quality depends on the question and the source. Curated documents can be wrong or stale; code can be necessary for implementation questions. Treat chat and tickets according to their authority and review status, not as automatically approved answers.
Should we copy rapidly changing records?
Consider querying the authoritative system through an access-controlled tool. Compare freshness, availability, permissions and operating cost with a maintained snapshot.
How do we know the corpus is ready to grow?
Review results on known questions, resolve consequential gaps, test access and change propagation, and confirm an owner can maintain the added sources. More documents alone are not evidence of better answers.
Explore AKB for a focused source pilot
Review documented knowledge storage, retrieval and agent access for your chosen workflow.