AI Agent Source Conflict Resolution: When Evidence Disagrees

An AI agent can gather more evidence than a person would usually open by hand. It can search a knowledge base, inspect a ticket history, compare policy pages, read logs, and bring back a tidy answer. That speed is useful, but it creates a quiet problem: the agent may find sources that disagree. A stale policy says one thing. A newer document says another. A customer record contradicts a support note. A code comment describes behavior that the tests no longer prove. A web page looks current but repeats an old claim.

Source conflict resolution is the discipline of deciding what the agent should do when evidence does not line up. The goal is not to force every source into a single confident answer. The goal is to keep disagreement visible long enough for the workflow to route it correctly.

This topic sits between AI Agent Knowledge Bases , AI Agent Source Provenance , and AI Agent Output Verification . A knowledge base helps the agent find trusted material. Provenance keeps evidence attached to the work. Verification checks the finished output. Conflict resolution handles the awkward middle, where the agent has found material but cannot honestly treat it as settled.

Disagreement is a signal, not a failure

Many weak agent workflows treat conflict as an inconvenience. The agent is asked for an answer, so it smooths the evidence into something that sounds reasonable. It may average positions, prefer the most recent-looking page, or write a vague sentence that hides the disagreement. The final result feels calm, but the calm is misleading.

A conflict often tells you something important about the workflow. It may reveal that the knowledge base has stale material. It may show that two teams use different definitions for the same term. It may expose a policy exception that was never folded back into the official source. It may show that a tool returns cached data while another returns live state. In software work, it may show that documentation and implementation have drifted apart.

The agent should be allowed to say that the evidence is not ready for action. That does not make the run useless. A clear conflict report may be the most valuable artifact the agent can produce. It tells the human owner where the system needs repair before more work can be delegated.

Authority should be named before the run

The easiest conflict to resolve is the one whose hierarchy is already known. If a current policy page outranks old training material, the agent should know that before it reads both. If a production record outranks a copied spreadsheet, the route should say so. If a signed approval record outranks a chat message, the tool contract should make that distinction visible.

Instruction hierarchy is not only about prompts. AI Agent Instruction Hierarchies explains how goals, policies, and evidence need different authority. Source conflict resolution applies the same habit to materials. The agent should not decide from vibes which source is official. It should receive source roles, freshness signals, owners, and stop conditions.

This matters because many sources look authoritative on the surface. A polished internal page can be abandoned. A spreadsheet can have the right column names and the wrong data. A vendor page can sound official while describing a product version the team does not use. A policy excerpt inside a ticket can be accurate for that ticket but not general enough for the next one.

Authority is operational, not decorative. It should answer a plain question: if these two sources disagree, which one may govern the result, and when must the agent ask instead of choosing?

Freshness is not the same as truth

Agents often lean on recency when resolving conflict. That instinct is useful but incomplete. A newer source may be a draft, a copied mistake, or a local exception. An older source may be the governing policy because it has not changed. A timestamp helps, but it does not settle authority by itself.

AI Agent Knowledge Freshness covers the broader maintenance problem. In a live run, the agent needs a practical freshness posture. It should notice effective dates, revision histories, source owners, and references to superseded material. It should also notice when those signals are missing.

Imagine a support agent reading a return policy, a regional exception note, and a recent ticket where a supervisor allowed something different. The recent ticket is real evidence, but it may not be policy. The agent can use it to explain uncertainty or ask for review, but it should not silently generalize it into a new rule. The same pattern appears in codebases. A recent workaround in one file does not necessarily define the architecture. It may be a local patch waiting for cleanup.

Good conflict handling keeps freshness and authority separate. It asks what is current, what is official, what is local, and what is merely nearby.

Compare the same claim before choosing

Start by writing the question as a claim that can be checked: which customer, product, region, action, and point in time does it concern? Read the relevant passage or record, not just a search snippet. Two statements may look contradictory because one describes a general rule and the other an exception, or because they apply to different versions or periods. Record missing context as unknown rather than filling it in.

Check the extraction before calling it a source conflict. A summary can omit a condition, a unit conversion can be wrong, or a search result can splice together text from different sections. Return to the original fields or passages and compare like with like. If the underlying sources agree, correct the extraction and keep the correction in the evidence trail. If one source cannot be read, report that gap; agreement among the sources the agent happened to retrieve does not prove that no conflict exists.

Then compare the sources in a consistent order:

1. Scope and status: Does each source apply to this case? Is it approved, draft, archived, or an individual decision?
2. Authority: Which source is designated to govern this type of question? A policy may govern eligibility while a live transaction record establishes what happened; neither answers every question.
3. Time and completeness: Check effective dates, version, retrieval time, exceptions, and whether the result is partial or cached. A failed lookup or permission denial is not evidence that a record does not exist.
4. Independence and support: Trace repeated claims back to their origin where practical. Several pages copying one report are not several independent confirmations. The Cochrane Handbook’s guidance on collecting data makes this distinction for multiple reports of the same study and recommends planning how discrepancies will be handled.
5. Disposition: Apply a pre-agreed rule if the applicable authority and facts are clear. Otherwise ask the source owner or reviewer, and keep any consequential action on hold. Do not turn an unresolved conflict into a majority vote or a numerical confidence score without a validated basis.

An authority rule is specific to the decision. A production record can show the current transaction status without proving that the transaction was authorized; an approved policy can state the rule without proving that a particular customer meets it. If the designated source itself is disputed or an exception’s authority is unclear, the hierarchy has not resolved the case.

The comparison should lead to an explicit outcome, not just a preference for one document:

An approved source can still contain an error. If its contents conflict with reliable evidence of a defect, flag that to its owner instead of treating its status as proof that the claim is correct.

The agent needs a conflict artifact

When disagreement matters, the agent should leave a small artifact that preserves the shape of the conflict. It does not need to be a courtroom brief. It should identify the question, the sources read, the point of disagreement, the source roles, the likely governing source if one exists, and the reason the agent did or did not proceed.

This artifact is different from a final answer. A final answer tries to satisfy the request. A conflict artifact explains why the request cannot be safely satisfied yet, or why it can only be satisfied with a caveat. AI Agent Artifact Design is useful here because conflict artifacts are work products. They should be reviewable without forcing the human to replay the entire run.

The artifact should avoid dramatic language. Most conflicts are ordinary maintenance issues. A stale page, a missing owner, a partial record, or an ambiguous exception is not an incident by itself. The agent should preserve the evidence, name the uncertainty, and route the question to the right owner.

For each source, retain a stable link or record ID, the relevant passage or field, its role and scope, version or effective date where available, and retrieval time. Add the proposed disposition, owner, and next check or decision needed. Keep sensitive excerpts in an access-controlled record and use a reference in the handoff when wider disclosure is unnecessary. The W3C PROV data model provides a way to describe entities, activities, and the people or systems associated with them; recording provenance helps a reviewer trace a claim, but does not by itself establish that the claim is true.

Example: a return policy and an exception

Suppose an agent is asked to issue a refund. The approved policy allows returns within 30 days, while a regional note says 45 days. The customer record shows a purchase 38 days ago in that region, and a recent ticket describes a supervisor granting a similar exception. These are illustrative records, not a rule for real refunds.

The agent checks whether the regional note is approved and effective for this purchase, and whether the purchase date and region in the live record are complete. The prior ticket may explain a past decision, but does not establish a general exception. If the note is approved and the workflow explicitly authorizes refunds under it, the agent can follow that rule after checking the current record. If the note’s status or scope is unclear, it should hold the refund and route the question to the policy owner.

A concise handoff might say: “Refund held. Policy version P1 says 30 days; regional note R2 says 45 days, but its approval status is unknown. Record C3 shows a purchase 38 days ago in the region, checked at the recorded retrieval time. Please confirm whether R2 applies.” The real artifact should link those identifiers to the relevant passages and include the proposed action and reviewer. Once resolved, record the decision and its scope; before acting, recheck any policy or customer fields that may have changed. A one-case approval does not automatically update the general policy.

Some conflicts should stop action

Not every conflict deserves the same response. If two public sources disagree about a historical detail in a low-stakes summary, the agent may cite both or choose the better-supported one. If applicable internal sources leave a customer’s refund eligibility unresolved, the agent should hold the refund for review. If documentation and tests disagree about a code path, the agent may prepare a patch, but it should make the mismatch visible in the handoff.

The stop condition depends on consequence. Conflicts around money, permissions, legal commitments, private data, external messages, production changes, or safety-sensitive claims should usually block autonomous action. Conflicts around phrasing, low-risk classification, or draft organization may only require a note.

This connects directly to AI Agent Approval Scopes . Approval is weaker when the reviewer cannot see that the source base was conflicted. A reviewer may approve a proposed action for one reason and reject it for another. The agent should not bury the disagreement inside a polished recommendation.

Define the handoff as well as the stop: name the responsible owner, the question they must answer, and what the agent may continue doing while it waits. If no owner is available, leave the item unresolved in a review queue rather than treating silence as approval. A reviewer can resolve the case, request more evidence, or return it to the source owner; the agent should record which happened.

Record the resolution separately from the original evidence: who decided, when, what sources and reasoning they used, and whether the decision applies only to this case. Link any approved correction to the source that changed. If the decision or underlying facts may expire before execution, recheck them at the action boundary; if they changed, reopen the decision rather than carrying forward an old approval.

Tool outputs should keep source roles intact

Conflict resolution becomes harder when tools flatten evidence. A search tool that returns snippets without source type, date, owner, or confidence makes every result look similar. A record lookup that blends system data and user notes into one paragraph invites the agent to treat everything as equally authoritative. A browser tool that strips page metadata removes useful context.

AI Agent Tool Contracts should therefore support conflict handling. Tools can label sources as policy, draft, customer-provided, external, generated, archived, or live system state. They can return timestamps and source identifiers. They can distinguish not found from permission denied and stale cache from live response. These details feel boring until a conflict appears. Then they become the difference between a useful stop and a confident guess.

The agent should also be careful with untrusted material. A web page, email, ticket, or document can contain instructions that try to steer the run. AI Agent Prompt Injection explains the authority problem. Conflict resolution depends on the same boundary. Content can be evidence without becoming a command.

Test the resolution path

Use a small set of realistic cases before relying on the workflow. Give the agent an approved policy and a newer draft, a regional exception with unknown approval, a permission-denied lookup, and two documents that agree only because they copy the same source. Check whether it identifies the actual issue, preserves the relevant evidence, and holds any action that depends on an unresolved conflict. Also include a clear, authorized case so that an agent that always escalates does not appear to pass.

Use simulated action tools for these cases and inspect attempted calls as well as the final explanation. Test the real authorization control separately: a good conflict report alone cannot prevent an unauthorized action. The agent evaluations guide explains how to turn those expectations into observable checks.

The mature habit

A strong agent system does not pretend all evidence will agree. It designs for disagreement. It names source roles, preserves provenance, separates freshness from authority, gives the agent a conflict artifact, and defines when conflict blocks action.

That habit makes the agent more useful, not less. Instead of producing a smooth answer that hides weak foundations, the agent can tell the team where the foundations need attention. It can still summarize, draft, compare, and recommend. But when the evidence disagrees, it keeps the disagreement in view long enough for the right person or rule to resolve it.