What Is an Agentic Stack, and Why Does It Matter More Than the Model?
A practical guide to the routing, context, tools, verification, and control layers that turn AI models into reliable agents.
TL;DR: An AI agent is not a model plus tools. It is a governed stack of eight layers: task intake, routing, context, tools, verification, human approval, observability, and an improvement loop. Routing is the connective tissue that decides where each piece of work should run, what data it may touch, what tools it can call, and what needs review before an action lands. This blog is for AI product and engineering leaders, and it argues that the stack around the model is what turns a prototype into a dependable agent.
The Question That Sounds Right and Is Not
“Which model should power our agent?” is the first question in the room when a company sits down to design one. It sounds like the central architectural decision, and it is not.
A capable model can still ship an unreliable agent. The issues rarely start at the model boundary. They start:
At the routing rules that decide where work runs,
At the context boundaries that decide what the model sees,
At the permissions on the tools it can invoke,
At the verification step that never runs,
Or at the approval gate that got skipped for speed.
Erik Schluntz and Barry Zhang at Anthropic argued this directly in December 2024. The production agents that work use simple, composable patterns like prompt chaining, routing, orchestrator-workers, and evaluator-optimizer. And the failures start from missing patterns, not from the wrong model.
The agent is not the model. The agent is the stack around the model.
What an Agentic Stack Actually Contains
An agentic stack is the governed system that turns a model call into a trustworthy action. It has eight layers, and each layer is a decision surface, not a piece of code.
Task Intake: What the agent has been asked to do, and at what risk level. For example, a request to summarize a public support thread is low risk, and a request to issue a customer refund is high risk.
Policy and Routing: Where the work should run, and under what constraints. For example, a low-risk summary can run on a small self-hosted model, and a refund flow can be pinned to a frontier model with retrieval, a judge, and a human approval gate.
Context and Memory: What information the agent is allowed to see and remember. For example, a support agent may retrieve the last thirty days of tickets for the same customer, but never credit card numbers or unrelated account data.
Agent Runtime: How the model plans, acts, retries, and hands off. For example, when a tool call fails, the runtime decides whether to retry with backoff, hand off to a stronger model, or halt and page a human.
Tools and Permissions: What the agent can read, write, or send in the outside world. For example, a drafting tool can produce a ticket in a review queue, and only a scoped, revocable token can actually publish it downstream.
Verification and Evaluation: How results are checked before they are acted on. For example, a schema validator catches malformed tool calls, and a judge model scores whether the draft matches the written rubric.
Human Approval: Where a person confirms before a consequential action lands. For example, a product manager signs off before the pull request is opened, and any spend above a dollar threshold requires a second approver.
Observability and Improvement: How traces feed back into better routing over time. For example, a week of traces shows that mid-tier routing fails on requests spanning more than three roadmap areas, which becomes the new escalation threshold.
The word stack is important here. These layers do not run in a straight line. They wrap the model on every call and share state across the run. The Model Context Protocol specification, introduced by Anthropic in late 2024, is one attempt to give the tool boundary a common shape. It does not tell you how to route or verify. That is the rest of the stack.
Routing Is the Decision Layer at the Center
Routing is where the stack decides what to do with a task, not just which model to call. A well-designed router asks the same questions on every request. How sensitive is the data? How complex is the task? How much reasoning depth is required? Are tools or long context needed? What is the cost and latency budget? What is the consequence of getting it wrong? Does the output need an independent reviewer?
The routing table encodes those answers. Task class one might run on a small self-hosted model with no tool access and a short context window. Task class four might route to a frontier model with retrieval, a coding sandbox, an independent judge model, and a human approval gate.
Isaac Ong and coauthors at Berkeley showed that learned routers trained on preference data can cut costs by more than half while preserving quality. Pierpaolo Baccichet and Richard Henwood at Sierra reported a shipping example. Their multi-model router keeps a task-specific ordered list of models and swaps to the next one when provider quality regresses, guided by a congestion-aware selector.
Route down by default, escalate on evidence. That is the principle worth writing on the wall. Routing is a policy system, and it does not belong in a dropdown menu. Diverse model families reduce correlated failure, so a router that spans providers is more resilient than one that does not. For a wider view of how routing sits alongside the rest of a control plane, see our blog on the multi-agent control plane.
Context Is the First Routing Decision
Before the stack decides how the work runs, it decides what the model is allowed to see. Context selection is where most agent quality problems start.
Prithvi Rajasekaran and coauthors at Anthropic framed the discipline in September 2025. Context is every token visible at inference. That includes the system prompt, tool definitions, memory, retrieved data, and message history. Sending more of it is not automatically better.
Kelly Hong and colleagues at Chroma tested 18 models and found that quality degrades unevenly as input length grows, well before the advertised context window is full. Drew Breunig named the four failure modes plainly: poisoning, distraction, confusion, and clash.
Context is a product decision and a security decision at the same time. It sets what data leaves regulated boundaries, what history the agent can act on, and what an attacker could smuggle inside a retrieval result. Compression, summarization, and retrieval scope all sit on the routing surface. For a longer treatment of the pattern, we cover it in context engineering for AI.
The Agent Runtime, Tools, and Permissions
Intelligence and agency are not the same thing.
A model reasons.
An agent acts.
The runtime is the code that turns reasoning into a sequence of actions with state, retries, and handoffs. The tool and permission layer is the code that decides what those actions may touch.
Simon Willison named the risk shape in June 2025. Any agent with private data, exposure to untrusted content, and an outbound channel is vulnerable to indirect prompt injection. You must break at least one leg of the trifecta. That is a permissions decision, not a prompt decision.
The OWASP Top 10 for LLM Applications 2025 places excessive agency, prompt injection, and system prompt leakage at the top of the list for the same reason.
Least privilege is the default. External tool output is treated as untrusted input. Consequential actions like payments, deletions, and outbound messages require explicit approval, not a confidence threshold. Every tool call is logged with input, output, and decision context.
Verification: The Executor Cannot Be the Only Judge
An agent that grades its own homework will pass more often than it should. Jie Huang and coauthors at ICLR 2024 showed the failure directly. Large language models cannot reliably self-correct their reasoning without an external signal. And self-criticism often makes results worse.
Verification is a separate layer with its own inputs.
Deterministic checks run first, since tests, schema validation, and policy checks are the cheapest way to catch a bad output.
Structured output validation catches malformed tool calls before they reach a tool. An independent judge model, following the pattern Yuntao Bai and coauthors at Anthropic formalized in Constitutional AI, reviews outputs against a written rubric. A human reviews anything consequential. Failure recovery covers rollback, escalation, and the honest failure message.
Execution and verification must be separable, even when they live in the same stack. Completed is a runtime state. Safe to act on is a verification state. Confusing the two is what turns a helpful agent into an incident.
From Request to Approved Action
Consider a common workflow. A support engineer wants the agent to turn a batch of customer feedback into a scoped product change and open a pull request against the internal roadmap.
The task intake layer classifies the request as medium risk. Data is customer-identifiable. The output creates a change record.
Context selection retrieves the last thirty days of tagged feedback from the vector store, strips PII, and pins the roadmap taxonomy into the system prompt.
Routing sends the task to a mid-tier model with retrieval, a scratchpad, and read-only access to the roadmap. The escalation rule promotes to a frontier model if the plan touches more than three roadmap areas.
The runtime plans the change, drafts the scope, and calls a summarization tool. State is checkpointed on disk, following the harness pattern Justin Young at Anthropic documented for long-running agents.
Verification runs three checks. A schema validator confirms the change record is well-formed. A judge model scores the proposed scope against the rubric. A search over past roadmap items flags duplicates.
Human approval gates the write. The product manager confirms the scope before the pull request is opened.
The full trace, verification scores, and approval decision are captured for evaluation and future routing.
Every layer of the stack is visible in this run. No layer is optional.
The Operating Loop
The stack improves through a closed loop: route, execute, verify, approve, observe, evaluate, and update the routing rules.
Observability is the substrate that makes the loop possible.
Model non-determinism forces teams to shift from unit tests to production-trace-driven development, since the ground truth lives in the run, not the code.
Evaluation datasets are built from real runs and their human corrections.
Routing rules move as evidence moves. A rule that promotes to a frontier model on complexity above a threshold is only justified when the trace record supports it. For a longer treatment of the loop, see why observability is non-negotiable.
Why Observability Is Non-Negotiable for Multi-Provider RAG Systems
When you are developing an AI product, you need a much narrower approach than a generalist approach. A narrower approach is much more aligned to your product’s vision, the problem that you are solving, and the ICP that you are targeting. A generalist approach is where you make an app or a product for a wide spectrum of users. They cater to users across …
A self-improving agent is not one that changes itself in the dark. It is one whose stack learns from measured outcomes and controlled updates.
An Implementation Checklist for the First Week
Define three task-risk levels, and write the routing rule for each.
Draw the data-handling boundary. Name what can leave the regulated store and what cannot.
Restrict tool permissions to the minimum set that the current task classes need.
Add one verification step before a consequential action.
Define the two events that must trigger human approval, and instrument them.
Turn on full trace capture with a common schema.
Build a small evaluation set from the last twenty real runs, including the failed ones.
Measure correction rate, tool failures, cost per successful task, and time to human resolution.
Change one routing rule based on evidence, and log why.
This is not a maturity model. It is a week of work that separates a demo agent from an accountable one.
The Right Question
Return to the opening. The question was “Which model should power our agent?” and it was the wrong architectural question.
The right question is longer and worth the extra breath.
What routing, context, permissions, verification, and learning loops must be in place before this agent can be trusted with the work in front of it?
The model is a component. The stack is the product.