Handling Duplicate and Outdated Knowledge Without Losing Sources
Learn how to detect exact and near-duplicate documents, identify genuinely stale knowledge using multi-signal telemetry, and preserve source provenance through version-controlled retention.
Key takeaways
- Exact hashes identify bytes, not record authority: Matching hashes do not justify merging records with different owners, permissions, jurisdictions, or business purposes.
- Similarity is a review signal, not equivalence: Similarity is not transitive; calibrate detection thresholds on labeled examples and never treat a score as authority to merge or delete.
- Old does not automatically mean stale: Combine source status, effective dates, owner review, references, retrieval patterns, and conflicts; the newest record is not necessarily correct.
- Preserve claim-level provenance and access boundaries: Track which source revision and passage supports each claim, and never make restricted facts visible through an unrestricted summary.
- Choose the least destructive valid remedy: Link, canonicalize, merge, supersede, archive, or delete only with evidence, an accountable reviewer, and a tested downstream plan.
Duplicate and outdated knowledge create different retrieval problems and require different remedies. A safe cleanup process distinguishes identical bytes from similar meaning, tests whether information is still authoritative, and keeps claims traceable to their sources without exposing restricted material or retaining records beyond policy.
1. Detect duplicates without confusing similarity and authority
Start with exact bytes
Compute a cryptographic hash such as SHA-256 over each original file. Matching hashes identify candidates for byte-level comparison; verify the bytes before consolidation. Even identical bytes do not make two records interchangeable. Identical copies can have different owners, access controls, contractual contexts, jurisdictions, or retention requirements. Keep those records separate unless the responsible authority confirms that consolidating the records is appropriate and permissions remain correct.
Normalize extracted text carefully
Files exported from different systems may contain the same substantive text but differ in encoding, whitespace, headers, or boilerplate. A second hash over normalized extracted text can help find these formatting variants. Document every normalization rule. Removing dates, amounts, tables, footnotes, or jurisdiction labels can erase material distinctions. Keep the original source and record the normalization version used for comparison.
Treat near-duplicate scores as candidates
Textual comparison, locality-sensitive methods, and embeddings can identify overlapping content, but they answer different questions. A high semantic similarity score suggests related subject matter; it does not prove textual identity, factual agreement, or business authority. Similarity pairs are not equivalence classes: A may be similar to B and B to C without A being similar to C. Avoid chaining pairwise matches into a single merge group. Calibrate thresholds using labeled examples from the actual corpus, review false matches, and record the model or comparison method used.
2. Keep vendor review rules in context
Near-duplicate tools use method-specific grouping and labels. Read each vendor's documentation to understand what its group labels mean, what field or text is analyzed, and what the tool recommends for review. A principal or representative label is not, by itself, evidence that a document is authoritative or suitable for deletion. Do not generalize one vendor's selection rule into a universal canonical-source policy.
3. Assess staleness with multiple signals
Age alone is not a reliable freshness test. An older stable specification may remain correct; a recently edited page may repeat an obsolete instruction. Review evidence such as:
- Source status and authority: Is the source system active? Is the named owner still responsible? Has an authoritative replacement been published?
- Effective dates and revision context: Do the effective date, applicable release, product version, or jurisdiction match the question being answered?
- References and dependencies: Do links resolve, and do they point to current systems, policies, and supporting material?
- Use and retrieval signals: Are people or agents still retrieving the item? Do sampled affected queries return obsolete or conflicting answers? Retrieval frequency signals impact, not correctness.
- Conflicting claims: Do other current sources disagree on an amount, date, requirement, or jurisdiction? Preserve each claim and escalate the conflict; do not average values or choose the newest file by default.
For each flagged item, record the signal, the authority consulted, the decision, and queries used to verify the outcome. A changed relationship or notice does not itself update dependent records or retrieval indexes; verify those effects separately.
4. Choose a remediation action
Use this matrix to select a remedy after reviewing the underlying sources. When dates, numbers, legal terms, or jurisdictions conflict, preserve the distinct claims and escalate to the accountable authority before canonicalizing or merging.
| Action | Evidence required | Reviewer | Downstream effects and recovery limits |
|---|---|---|---|
| Keep separate | Distinct owners, permissions, audience, jurisdiction, effective period, or record purpose; or unresolved factual differences. | Each source owner or relevant domain authority. | Retain separate identities and access rules. Do not merge retrieval results in a way that hides the distinction. Revisit only when authority or context changes. |
| Link as related or duplicate | Verified overlap plus a clear description of whether the relation means duplicate, related, derived from, or superseded by. | Source owner or knowledge steward; escalate material conflicts. | Add a typed relationship while keeping both records. Check that search and agents retain the distinction. Removing a link does not undo any copies or summaries already derived from either source. |
| Choose a canonical view | An authoritative owner identifies the source or view for a defined scope and effective period; access classifications are compatible. | Accountable domain owner, with security review where permissions differ. | Point readers to the selected view and preserve links to alternatives. Do not union restricted content into a broadly visible canonical summary. Rollback requires restoring the prior view and checking dependent indexes and citations. |
| Merge compatible claims | Claim-by-claim comparison confirms there are no unresolved contradictions; every retained statement has a source and scope. | Domain owner approves meaning; editor or steward verifies the diff and citations. | Create a new revision or record with claim-level lineage. Preserve source identity and permissions. Reversal requires the reviewed base revision and source records to remain available under policy; deletion may make recovery impossible. |
| Supersede | An authorized replacement explicitly applies to the same scope, and effective dates and exceptions are understood. | Policy, product, or service owner responsible for the subject. | Mark the earlier item as superseded and direct retrieval to the replacement without erasing the old identity. Confirm that dependent answers stop presenting withdrawn guidance as current. |
| Archive from active retrieval | No longer active for ordinary use, with a documented reason and a retention or legal requirement to preserve it. | Records or governance owner, plus security/privacy review as needed. | Remove it from active search and agent retrieval while retaining it only in the approved archive. Test that it is not surfaced through snippets, caches, or derived content. Retrieval restoration is possible only while the archived record remains retained and accessible under policy. |
| Permanently delete | Confirmed deletion eligibility under applicable retention, privacy, legal-hold, and security policy; dependencies and copies identified. | Authorized records, privacy, or security approver. | Remove the record and propagate withdrawal to indexes and derived systems. Do not promise rollback: permanent deletion may be unrecoverable, and separate backups or legally required copies follow their own policy. Verify completion and document any partial failures. |
5. Preserve claim-level provenance and permissions
A consolidated answer should make it possible to determine which source supports each assertion. For every retained claim, capture:
- Stable source identity: A persistent identifier for the original record, not just its title or current URL.
- Revision: The source revision or version used when the claim was reviewed.
- Passage: A section, page, line range, or other usable anchor to the supporting text.
- Relationship type: For example,
supports,derived_from,related_to, orsupersedes, with direction and scope clear. - Claim and decision record: The actual assertion, reviewer, decision, and review date or workflow reference.
Provenance does not mean keeping every source forever. Retention, access, deletion, and legal-hold requirements still govern source material and derived records. When a source must be deleted, remove or revise dependent claims as policy requires and preserve only permitted audit metadata.
Never combine restricted material into a summary visible to a broader audience. Enforce the user's current permissions when retrieving the source and when producing results: test search results, passage snippets, citations, generated summaries, and downstream agent context. A citation or relationship must not become a route around the source's access controls.
6. Execute changes against reviewed revisions
- Prepare an exact diff. Show additions, removals, links, status changes, permissions, and affected records. Name the source revisions on which the proposal is based.
- Review authority and scope. Confirm current source authority, effective period, jurisdiction, owner, and access classification. A previously approved proposal can become stale if its source changes.
- Enforce an atomic revision precondition. At execution, recheck executor permissions, approval validity and reviewer authority under current policy. Enforce target revision and relevant evidence preconditions in the conditional write. Stop for fresh review if they fail; do not rely on a separate check followed by an unguarded write.
- Make batch outcomes explicit. Record which items succeeded, failed, or were skipped. Use idempotent retries so a retry does not create duplicate relations, summaries, or deletion side effects.
- Propagate and verify withdrawals. Update or remove obsolete content from keyword and vector indexes, caches, citations, and derived claims. Run representative affected queries and confirm withdrawn guidance no longer appears where it should not.
- Respect recovery limits. Revision history can help recover an earlier version while it is retained, but it is not a complete backup or approval audit trail. Do not promise an undo for permanent deletion. Maintain separate backups and approval records where policy requires them.
7. Measure the workflow without overstating results
Hypothetical candidate precision
Suppose a process flags 100 candidate pairs and reviewers examine all 100. If 70 are confirmed duplicate pairs, the candidate precision in that reviewed set is 70 / 100 = 70%. This hypothetical result says nothing by itself about recall: it does not reveal how many duplicates were never flagged. It also does not establish that any confirmed pair is eligible for deletion or quantify storage savings. A confirmed duplicate may instead be linked, canonicalized, superseded, or retained separately.
Acceptance worksheet
Fill this in with observed results from a defined corpus, review period, and test set. Set thresholds before the review and explain the basis; do not substitute an unmeasured benchmark or projected return for evidence.
| Check | Record for this review | Evidence to retain |
|---|---|---|
| False merges or incorrectly combined claims | ____ count; ____ examples; disposition: ____ | Reviewed pair IDs, source revisions, and correction log |
| Candidate precision | ____ confirmed pairs / ____ reviewed flagged pairs = ____% | Labeled review set and decision criteria |
| Recall or missed candidates | ____ / not measured; method: ____ | Separate sampling method for unflagged records, if assessed |
| Stale answers in tested affected queries | ____ failing / ____ tested; query set: ____ | Queries, expected answers, observed results, retest evidence |
| Valid source citations | ____ supported claims / ____ checked claims | Claim-to-source IDs, revisions, passages, and permission checks |
| Review effort | ____ reviewer hours for ____ items | Review log and time period |
| Oldest pending item | ____ days; owner: ____; next action: ____ | Queue snapshot and escalation record |
| Index/cache withdrawal verification | ____ affected systems checked; result: ____ | Propagation status and query checks |
8. Decide between ongoing tooling and a finite cleanup service
| Responsibility | Ongoing tooling | Finite cleanup service | Enterprise retains |
|---|---|---|---|
| Detection and connectors | Repeated scans and connectors to changing source systems, if included in the chosen product. | Initial inventory and assessment within the engagement scope. | Decide which systems and records are authoritative and grant appropriate access. |
| Review workflow | Candidate queues, comparison views, reviewer assignment, and change controls, if provided. | Remediation plan, mapping, and agreed cleanup actions during the project. | Provide subject-matter reviewers and approve canonical decisions, merges, and deletion. |
| Ongoing governance | Can support recurring detection and monitoring; it does not decide business authority on its own. | Usually ends at the contracted deliverables and handoff. | Maintain policy, permissions, source ownership, retention, and post-project review. |
| Verification and recovery | May report operations or revisions; verify which indexes, caches, backups, and audit records it actually covers. | Should deliver outcomes and handover evidence defined in the statement of work. | Test retrieval, maintain backups and approval records where required, and handle future changes. |
Choose ongoing tooling when the source estate changes continually and the organization can operate a recurring review process. A finite service can help assess and remediate a defined backlog, but does not replace continuing ownership. In either model, the enterprise remains responsible for authority, security, retention policy, and maintenance after the work is done.
9. Where AKB fits
The AKB public repository is the reference for AKB's current storage and integration details. As checked on September 28, 2026, the public repository describes PostgreSQL-native revisions as the default, with Bare Git supported for legacy deployments. The read/write Core offers Web UI, REST API and MCP surfaces; its design philosophy separates storage from external curation rather than bundling a knowledge gardener. Treat storage and retrieval separately from curation: duplicate detection, staleness decisions, review approvals, retention policy, and verified withdrawal need an explicit operating process. Confirm deployment-specific revision and integration behavior in the current repository documentation before planning technical changes.
FAQ
How do exact duplicates differ from near-duplicates?
An exact-byte hash matches identical file bytes. A normalized-text match can identify formatting variants after documented normalization. Near-duplicate or semantic matching identifies possible overlap, not identity, factual agreement, or permission to merge. Review the source context and each material difference before choosing an action.
How should we choose an authoritative or canonical source?
Ask the accountable owner for the relevant subject, scope, and effective period; check the source system and applicable policy. A tool's similarity score, file age, or representative label does not establish authority. Preserve unresolved differences in dates, amounts, and jurisdictions and escalate them rather than averaging or selecting the newest record.
Does an old document count as stale?
Not by age alone. Check whether its source remains active and authoritative, whether its effective period and dependencies are current, and whether it conflicts with other sources. Retrieval frequency can indicate impact, but does not prove correctness. Test representative queries before withdrawing or replacing guidance.
How do we preserve provenance without exposing restricted information?
Track the stable source ID, reviewed revision, supporting passage, and relationship type for each claim. Apply current permissions to retrieval, snippets, citations, summaries, and derived agent context. Do not put a restricted source's facts into a broadly visible canonical summary. Retention and deletion still follow the applicable policy; provenance is not a promise to retain source content forever.
What role does AKB play in a cleanup workflow?
AKB is a core read/write knowledge store. The public repository separates Core storage from external curation. Teams should plan duplicate review, staleness decisions, permission checks and withdrawal verification as explicit workflow responsibilities. It documents PostgreSQL-native revisions by default and legacy Bare Git support; confirm the deployed version and configuration. Consult the current AKB repository for technical details.
Explore AKB
Consult the repository for current AKB project information.