How AI Agents Work: Models, Tools, Memory, and Guardrails
A practical explanation of the parts inside AI agents: reasoning models, tools, memory, orchestration, sandboxes, approvals, and evaluations.
An AI agent can appear to move through a task by itself. Inside, its parts are understandable: a model interprets the goal, tools let it act, context carries information between steps, and software around the model manages the run and enforces boundaries.
The craft is in how these pieces are assembled. A model can propose an action; the surrounding system determines whether that action is available, executes it, and returns the result. Some agents use a single model and a few tools. Others add persistent memory, isolated workspaces, human review, or several specialized agents.
Follow one task through the loop
Suppose a support agent is asked to investigate a delayed order and draft a reply. Its allowed tools can read the order and carrier status, search the support policy, and save a draft. Sending a message requires a person’s approval.
- Receive the goal and constraints. The agent gets the order identifier, the request, and its permitted tools. If the identifier is missing, it asks rather than guessing.
- Choose and request a tool. The model asks to look up the order. The surrounding application checks the request and runs the lookup using the agent’s authorized access.
- Observe the result. The tool returns a status or an error. The model uses that result to decide whether to check the carrier, look up policy, or stop for help.
- Prepare an output. The agent drafts a reply grounded in the records it found. If the carrier status is unavailable, the draft says so instead of claiming a delivery date.
- Review and finish. A person reviews the draft and approves sending if appropriate. The system records the action and its result; a draft alone is not evidence that a message was sent.
This loop can repeat many times. The application should also set stopping conditions: for example, a deadline, a limit on tool calls or spending, and a way to pause when a source is unavailable or an action needs approval. An agent that keeps retrying without checking the result can turn a small problem into a larger one. Anthropic’s overview of agent patterns describes this cycle of tool use and feedback.
The model
The model interprets instructions and the information supplied to it, then produces an answer or a proposed next step. Depending on the model and application, it may work with text, code, images, or other inputs. It can also choose a tool and provide arguments in a structured form.
The model does not automatically know the current state of an order or repository. It needs that information in its context or through a tool. It can also misunderstand the task, choose the wrong tool, or make a plausible claim unsupported by a result. A stronger model can help, but reliability also depends on the tools, constraints, and checks around it.
The tools
Tools are how an agent reaches the world.
A tool can search a database, read a file, save a draft, or run a test. A browser, shell, or computer-use interface can give broader access to software. That flexibility is useful, but it also makes the permission boundary more important.
A good tool has a clear name, inputs, outputs, and limits. For the support agent, a lookup might require an order ID and return a timestamped status; a separate draft tool saves text without sending it. The application validates arguments and enforces access to the relevant account. A model’s request is not itself authorization. See AI Agent Tool Contracts for how to define and test those boundaries.
The Model Context Protocol (MCP) is one way to connect applications to tools and data sources. A common interface can make integration easier; it does not by itself decide what a particular agent is authorized to access. OpenAI’s tools for building agents and Anthropic’s computer-use tool are examples of different ways to give models access to external capabilities.
Memory and context
Working context is the information supplied to the model for a step: the user’s request, instructions, relevant records, and results so far. It is finite, so a long run may need to retrieve documents again or carry forward a summary. A summary can omit an important detail; check the original record when the distinction matters.
Longer-lived memory is information stored for later use, such as a stated preference or a pointer to a project decision. It is not automatically present in every model call. It must be retrieved or supplied, and it can become stale, wrong, or inappropriate to reuse. Good systems make retained information reviewable and correctable, with access and retention controls suited to the data. The memory and context guide explains the difference in more detail.
Planning and orchestration
Some agents write an explicit task list and revise it as evidence arrives. Others choose the next step without showing a separate plan. A short visible plan can help a person check the intended approach before consequential work begins, but a plan is not proof that the steps succeeded.
The surrounding application, often called a harness or orchestrator, manages the loop: it sends context to the model, routes tool requests, returns results, tracks state, and decides when to stop or pause. In some systems it also coordinates multiple agents. Exactly which component does each job depends on the product; not every agent needs a separate planning service or a team of agents.
Sandboxes
Agents that run code or edit files may need a controlled workspace. A sandbox can restrict available files, network access, credentials, and commands. OpenAI’s Agents SDK, for example, supports sandbox execution and a manifest for describing workspace inputs and outputs.
Isolation reduces exposure only to the extent that it is configured and enforced. A sandbox with broad credentials or unrestricted network access can still expose data. Treat web pages, documents, and tool results as potentially untrusted: a page may contain instructions intended to redirect the agent. Keep permissions outside the model’s control and give the workspace only the access it needs. The prompt injection guide covers this threat and its limits.
Approvals
Approvals are where human judgment enters the loop.
An agent may read documents and draft a recommendation without asking, need approval before emailing a customer, and be forbidden from changing account permissions. The same agent can have different authority in different contexts.
A useful approval identifies the exact action, recipient or target, and content to be approved. The system should enforce the gate at the action, record the decision, and ask again if the material details change. If approval is denied or unavailable, the agent should leave the draft pending and report that the action did not happen. See Agent Approval Scopes for designing that handoff.
Observability and evaluation
A person should be able to inspect what an agent did. Depending on the task, useful records include tool requests and results, timestamps, errors, approval decisions, costs, and the final artifact. Logs also need access and retention limits: they can contain sensitive inputs and outputs.
Evaluation means testing the system on realistic tasks before and after changes. For the support example, include a missing order ID, an unavailable carrier, an out-of-scope account, a misleading instruction in a ticket, and a send attempt without approval. Check both the final draft and the actions taken. Use test accounts or simulated tools so an evaluation cannot send a real customer message.
Model output can vary between runs, so useful tests include observable outcomes and boundaries as well as exact answers. A passing test gives evidence for the cases tested; it does not guarantee safe behavior on every task. The evaluations guide goes deeper into building a representative test set.
Multi-agent systems
Some tasks can be split between specialized agents: one researches, another writes, and another checks citations. Clear assignments and a way to combine and verify results can help. Parallel work can also duplicate effort, increase cost, and introduce conflicting findings or failure paths.
Start with one agent when it can do the job. Add another when a distinct role or independent parallel task has a reason to exist, and define what evidence the receiving agent must check before relying on its output.
The real architecture question
The central question is not how clever the agent sounds. It is whether the system can answer:
- What can the agent see and change, and who enforces those limits?
- What information is current, and where did it come from?
- Which actions require approval, and what exactly was approved?
- What record is left behind, and who can inspect it?
- What happens when a tool fails, a limit is reached, or the agent is uncertain?
Those answers turn an impressive demo into a workflow a person can evaluate and operate.