AI Agent Memory and Context: What to Remember, What to Forget

Memory sounds like an obvious upgrade for AI agents.

The agent remembers your preferences. It remembers the project. It remembers the customers. It remembers how you like summaries written, which files matter, and what happened last time.

Then one day it remembers the wrong thing.

It uses an old policy. It repeats a preference you changed. It treats a temporary exception as a permanent rule. It remembers a private detail in a context where that detail does not belong.

That is the central tension. Memory makes agents more useful, but bad memory makes them confidently stale.

Context is not one thing

People often say “give the agent context” as if context is a bucket. It is better to split it into layers.

Working context is what the agent needs for the current task: the question, recent messages, files, tool results, and decisions so far.

Project memory is durable knowledge about the work: architecture notes, house style, accepted tradeoffs, customer segments, release rules, or process steps.

Personal preference is how a person likes work delivered: concise summaries, no calendar changes without asking, a preferred tone, a usual meeting window, or a formatting habit.

Policy memory is what must be followed: security rules, legal language, approval thresholds, data handling requirements, and escalation paths.

Those layers should not be blended casually. A personal preference is not a company policy. A project habit is not a legal rule. A temporary workaround is not architecture.

The working context is the information actually supplied to the model for a particular step. Durable memory is stored outside that context and must be retrieved or supplied before the agent can use it. A conversation summary or checkpoint can carry work across a context reset, but it is not necessarily a permanent preference or an authoritative record. Model training is a separate mechanism: changing a memory entry does not necessarily change the model’s weights, and the memory controls available to a user depend on the product. A summary can also omit a qualification, so keep links to the original decisions and check consequential details there. Anthropic’s account of context engineering describes compaction and persistent notes as distinct ways to carry information forward. For the practical limits of the current working set, see AI Agent Context Windows and Working Sets .

Good memory is small

The best agent memory is not a transcript of everything.

It is concise enough to review, current, sourced, and useful. The right size depends on the task: a short note that omits an important exception can be worse than a longer one.

A strong memory entry might look like this:

Preference: User prefers implementation summaries with changed files, test results, and any known risk.
Scope: Coding tasks only.
Source: Repeated instruction from user.
Review: Remove if user asks for shorter summaries.

A weak memory entry looks like this:

User likes detailed answers.

That is vague enough to become annoying everywhere.

Memory needs ownership

Every durable memory should have an owner or a source.

If the agent remembers a policy, where did it come from? If it remembers a preference, who stated it? If it remembers a customer constraint, is that still true? If it remembers a project decision, was it accepted or only discussed?

Without ownership, memory becomes rumor.

For teams, a useful pattern is a living source of truth: policy files, project docs, workflow instructions, and user-editable preferences. The agent should retrieve memory from known places rather than hoarding unreviewed impressions forever. A memory note can point to a policy, but it should not replace the policy or grant permission by itself.

A record is easier to maintain when it includes the claim, its scope, source and location, owner, date recorded or last checked, and a review or expiry condition. Distinguish a stated preference from an inference and a proposed decision from an approved one. If there is no owner for a shared rule, assign one before treating the note as authoritative. Review dates prompt a check; they do not prove a fact is still correct.

Choose what enters the next run

A memory store can contain more than any one task needs. Start retrieval with the current user, project, and purpose; filter by access and scope before ranking for relevance. Then include the few entries that help with the task, along with their source and any important qualification. A search match is a candidate, not proof that the entry is current or authoritative. If nothing relevant is found, the agent can proceed without a memory or ask for the missing information.

For example, a request to prepare a release summary might retrieve the user’s preferred summary format and a pointer to the project’s release instructions. It should not pull in a different team’s customer notes merely because they mention a release. Before relying on the release instructions, the agent checks the current version. It keeps an old incident summary as background, not permission to repeat an old exception.

This selection needs controls in the retrieval system, not just a prompt telling the model to respect boundaries. Apply permissions before results reach the model; keep caches and derived summaries within the same boundaries, and update them when access changes. The knowledge bases and grounding guide explains how to choose and verify sources.

Forgetting is a feature

Agents should forget more than most product demos admit.

They should forget temporary instructions after the task ends. They should forget sensitive details unless there is a clear reason to retain them. They should forget stale facts when a source changes. They should forget preferences that were only situational.

Forgetting is not a weakness. It is how the agent avoids turning every moment into permanent baggage.

A useful test: if a memory would surprise the user when reused later, ask whether it should be retained at all, whether the person understands the reuse, and whether a shorter lifespan would serve the task.

Removing a note from the next prompt is not the same as deleting it. It may still exist in a memory store, conversation history, logs, backups, or downstream summaries. A deletion process should identify those copies, remove or expire them according to the system’s retention rules, and explain any copies that remain or cannot be removed immediately. A product’s controls and obligations vary, so check its actual settings and retention policy rather than promising complete erasure from a single button.

The stale-memory failure

Stale memory often looks intelligent at first.

An agent remembers that the team used one testing command last month. The repository changed. The agent keeps running the old command, sees strange failures, and starts debugging the wrong layer. Or it remembers that a customer was on a trial plan, even though the account upgraded yesterday. It drafts the wrong reply with perfect confidence.

The fix is not only “make the model smarter.” The fix is to make freshness part of memory design.

Useful memory should answer:

• When was this recorded?
• Where did it come from?
• Does a current source override it?
• Should the agent verify it before using it?
• Is this safe to reuse in this context?

What agents should remember

Agents should remember things that reduce repeated explanation without increasing risk.

Good candidates:

• Stable formatting preferences.
• Project-specific terminology.
• Approved workflow steps.
• Known source locations.
• Tool usage notes.
• Repeated decision criteria.
• Accessibility or communication preferences.

Riskier candidates:

• Personal relationships.
• Health, finance, employment, or legal details.
• Secrets and credentials.
• Temporary emotional context.
• Sensitive customer facts.
• Anything inferred rather than stated.

The more intimate the memory, the more control the user should have over it. Do not store credentials in ordinary agent memory; use an appropriate secret manager and access controls. A sensitive fact may be necessary for a specific task, but that does not by itself justify retaining it for later tasks. The data boundaries guide covers reducing what a workflow exposes.

From a note to a decision

Suppose a coding agent stored a note that the team runs npm test before release. On the next release, the note is a lead, not proof that the command is still right. The agent can check the current repository instructions and scripts, run the applicable check if permitted, and report the result. If the note conflicts with a current instruction from the responsible owner, it should use the applicable current instruction and correct or flag the note. If authority or scope is unclear, it should surface the conflict rather than guess.

A practical memory lifecycle looks like this:

1. Capture: Save only a useful, permitted fact, with its source and scope. Keep a temporary task decision in the task record unless it is explicitly intended for broader reuse.
2. Retrieve: Select entries relevant to the current user, project, and task. Enforce access boundaries before retrieval, including for shared indexes and summaries.
3. Verify: Check volatile facts and consequential rules against their current source. If the source is unavailable, say so and avoid treating an old note as confirmed.
4. Correct or expire: Update a wrong entry, identify conflicting copies, and set a review or expiry condition where appropriate. Preserve an audit record only as the retention policy permits.
5. Delete: Provide a way for an authorized person to find and remove a memory, and account for copies in logs, summaries, and backups under the system’s retention rules.

Test this lifecycle with a changed preference, an outdated policy, a deletion request, and two users or projects with similar names. Check both that the right memory is retrieved and that an unrelated or deleted memory is not reused. For the deletion case, inspect the stores and derived copies the system controls, then run a fresh retrieval under the same permissions. A clean result from one query alone does not prove every copy is gone. Record the outcome and any stores that could not be inspected. AI Agent Memory Audits goes deeper into review and correction.

Treat retrieved memory as data

A stored note can be wrong or deliberately manipulated. A web page, customer message, or repository file might tell the agent to save an instruction for later. That content is not automatically authorized to change a shared memory or override the user’s task. OWASP’s discussion of memory and context poisoning describes how an injected instruction can persist beyond the original encounter.

Restrict who and what can write shared memory, retain provenance, and review proposed durable instructions before promoting them from untrusted material. Isolate users and projects, and enforce tool permissions outside the model. These controls reduce risk but do not guarantee that an agent will ignore every malicious instruction. See Prompt Injection and Untrusted Content for the wider threat.

A memory review ritual

For a personal agent, review memory like a small settings page:

1. What does the agent believe about me?
2. What does it believe about my work?
3. Which memories are sourced?
4. Which memories are stale?
5. Which memories should be deleted?

For a company agent, review memory like infrastructure:

1. Which source controls this instruction?
2. Who can change it?
3. Which agents use it?
4. How is it versioned?
5. How do we detect outdated guidance?

Memory is not magic personalization. It is a knowledge system, and knowledge systems need maintenance.

The memory standard

An agent should remember enough to stop wasting your time, but not so much that it becomes a shadow archive of your life.

That is the standard.

Small, sourced, reviewable, scoped, and easy to delete.

Memory should make delegation lighter. If it makes the agent presumptuous, it is not memory anymore. It is drift.

Put the guide into a real workflow

Start with one recurring task and a small memory store. Record the owner’s stated preferences, links to the current project instructions, and any task-specific handoff that will be needed on the next run. Label each item by scope and source. Leave credentials and unnecessary personal details out.

On the next run, retrieve only the relevant entries, check any changeable or consequential facts, and show the sources used in the handoff. Ask the owner to correct a wrong preference or resolve an unclear policy conflict. At the end, expire temporary notes and review whether any new entry deserves to persist. The checkpoints guide explains how to preserve the state of an unfinished task without turning every detail into permanent memory.