Eval-First Product Design For Frontier AI Products

The evaluation suite is the specification your frontier product actually runs on. Everything else is intent.

TL;DR: AI products developed using frontier AI models surpassed many traditional product requirements, best practices, and norms altogether. These days, products are iterated at lightning speed, resulting in continuous releases. This is all because of LLMs with agentic capabilities that are also being iterated, refined, and released at a faster speed. These agentic LLMs or AI models help develop better, more aligned releases of AI products. As such, evaluating whether these products are being developed in a safe and aligned manner is crucial. Eval-first product design treats the evaluation suite, including the rubric, as an important specification itself. This blog covers the five major instances that improved the PRDs, there are six things an eval specifies, etc. If you are an AI PM, product builder, or founder shipping agent, vision, and voice features, this blog will help you a lot.

Three Failures With One Shared Root Cause

Failures and shortcomings are very common when you are developing any product, be it an agentic product or any traditional product that does not use AI. But when you are developing an agentic product, you will come across some really annoying behaviors. For instance, a voice agent keeps talking after the customer interrupts, or it will start talking when you pause for a little while but haven’t conveyed the entire problem.

And then we have vision models that return unnecessary edits in the image. For instance, when you are editing or modifying an image, it will subtly change the facial structure of the person or add an object that doesn’t seem natural.

This can happen because an agent sometimes picks the correct tool at the wrong moment in a conversation.

Now, lately we have seen a lot of improvement in these voice and vision models, but I am just pointing out some common errors I see often.

Let me explain how these three failures share a common root cause.

Consider that you wrote a rubric and it scored each one as a “PASS”. Here, there can be some of the following conditions as to why the rubric passed the output:

1. The rubric evaluated whether what the agent said were the correct words [in semantic and syntactic coherence].

2. It confirmed the tool was called with valid arguments.

3. It confirmed the response followed policy and guardrails.

4. It did not evaluate whether the waiting time was long enough for the customer to still be willing to speak.

5. It did not evaluate whether the image it edited preserved the facial consistency, etc.

This is the recurring failure of the frontier model.

The eval scores the artifact, but the user judges the interaction. In other words, the user (who is a user) judges the output based on their experience and feelings. Whereas the eval scores whether the overall output satisfies the rubric.

Now, the issue with the rubric is that it tends to be superficial at times, and “looks correct” is not a specification.

What Eval-First Product Design Actually Means

So the eval-first product design is not a habit of writing rubrics before you write features. Features come first, and based on them you write rubrics or evaluators. This is essentially a design practice where the evaluation suite works as the product specification.

Why?
Because the suite defines what acceptable output looks like on every user-facing UI. It becomes the source of truth for what the product should do, not the prose requirement.

The traditional PRD for traditional products specifies behavior in natural language for deterministic systems. It says or defines things like “when the user clicks submit, validate the form and show a confirmation modal.”

That works when the system is deterministic or has one correct path and one correct output. A test either passes or fails against a fixed return value. The PRD is legible because the underlying software is legible — meaning to say it is straightforward or linear.

Probabilistic systems like the LLM (with a softmax function) break this rule. The same input produces a range of outputs across runs, models, and prompt revisions. Prose cannot pin down what “helpful,” “accurate,” or “on-brand” means at the token level.

Andrej Karpathy framed this issue in his Sequoia Ascent 2026 talk: “LLMs and reinforcement learning automate what you can verify.”

Now, just think for a moment, if verification is the substrate, then the verifier is the specification, isn’t it? The eval suite becomes the rule or constraint that the model is built to satisfy.

This is why you should always write the evaluation after you have built the feature. Because the main aim of the feature is only known to the builder, product leaders, and AI engineers. Similarly, they are the ones who will know the function of the feature. When you build something based on user feedback, you know what the user wants and what the actual architecture of the feature is. Hence, you can only write an evaluation or a rubric based on the combination of the user feedback as well as the architecture of the feature

Why Frontier Products Broke The Old PRD

Frontier models are very capable these days for doing any tasks. From recent releases such as GPT 5.6, Claude Opus 5, Claude Mythos, Kimi K3, etc., we see that they are capable of hard-core cyber attacks, running overnight to complete a certain task, etc.

Now, let’s see what today’s frontier did with old PRD. They literally moved past that assumption of one model, one modality, and one deterministic behavior rule. Below I have explained how they did it.

Start with modality.
A 2024 PRD could specify “text in, text out” and move on. Google’s Multimodal File Search now lets a single retrieval call span PDFs, images, and audio. The input contract is no longer a string. It is a mixed bundle of file types the product owner has to reason about.

Model choice is now a portfolio decision.
OpenAI’s GPT-5.6 release split the default into a three-tier Sol, Terra, and Luna family, priced from $1/$6 to $5/$30 per million tokens at launch. Naming a single model in a PRD is stale the day it ships.

Runtime routing followed.
Cognition’s Devin Fusion took model selection into the runtime. The post reports a 35% cost reduction on FrontierCode, 41% with Fable 5, and 88% of merged PRs driven by the router rather than a fixed model. Their line lands hard: “The age of using one model for all of your work is coming to an end.”

The action surface is not fixed either.
Anthropic’s mid-conversation tool changes documentation lets an agent add or remove tools between turns while preserving the prompt cache. Capabilities expand inside a single session.

Voice and generation transformed.
Anthropic’s Claude Voice Mode expansion now spans Opus, Sonnet, and Haiku with access to Gmail, Slack, and Calendar. Karpathy’s Sequoia Ascent talk reframes generation itself, positioning vibe coding as complementary to agentic engineering. The design pattern under those five shifts is its own thread, covered in how to design AI features for nondeterminism.

What An Eval Suite Specifies That A PRD Cannot

A PRD paragraph can specify a certain intent. That intent stops making sense the moment the same input can produce many different outputs, which is the nature of probabilistic models.

Consider this: product logic used to compile into branches that a reviewer could read and compare. But agent logic compiles into a range of different outputs.

A written line like “the assistant should refuse unsafe requests” reads well in a doc. But it will definitely not succeed in production. Now, if you properly consider it, then an eval case captures four things: the input you tested, the user segment it represents, what an acceptable answer looks like, and the specific mistake that would block a release.

• The input you tested: The inputs are essentially the prompts. But along with that, there are the edge cases you have already seen fail, production requests, and adversarial prompts like jailbreak attempts, prompt injection strings, and malformed tool responses.

• The user segment it represents: An eval suite also specifies a user group, language, document type, or workflow the case stands in for. So pass rates hold per slice and not just in aggregate.

• What an acceptable answer looks like: In this case, you might be interested in the output pattern you accept, including when the agent should decline or hand off, what share of claims must trace back to a retrieved source, which model or tool should handle the request, and how the bar shifts across text, code, image, audio, and video.

• The specific mistake that would block a release: The tagged issue or error you refuse to ship past, whether that is a hallucinated claim, a misrouted call, a policy break under an adversarial attack, or a modality-specific slip like a receipt total the agent read wrong.

Now, consider vision. Anthropic’s Claude Opus 5 launch shipped a long context with thinking on by default. A screenshot-to-summary agent can now ingest a full onboarding flow in one call. “Looks correct” no longer works as an eval criterion. Locality and preservation need their own eval slice.

The Methodology In Five Moves

1. Tag/label the failure that matters, not the feature.
   Write down what a bad output looks like before you write a spec. Focus on the failure a user would report, not the feature you want to ship.

2. Build the eval from real failures and not speculation.
   Run error analysis on real outputs first, then let the evaluator codify what you actually found. Hamel Husain and Shreya Shankar make this the core of their Evals FAQ: you assemble the spec iteratively, from evidence. Speculating about failure modes invents problems that never occur in production.

3. Pick the smallest grader stack that can defend the release.
   Use code checks where you can. Use LLM-as-judge only for the calls a human cannot script. Track cost per successful task as your honest number, as covered in the complete guide to LLM and AI agent evaluation. A cheap grader that catches the top three failure modes beats a fancy grader that catches everything but costs more than the feature earns.

4. Store prompts, fixtures, and graders in one repo.
   OpenAI’s Migrate from the prompt object guide says it directly: “Move prompt content into source code so prompt changes go through the same review and release process as product logic.” Treat evals the same way. One repo, one review process, one release.

5. Convert every escaped failure into a permanent case.
   When a user hits a bug the eval missed, add that exact input to the fixture set. The suite grows with production. It never shrinks.

Four Ways Eval-First Fails In Practice

Now, let’s discuss how eval-first fails in practice.

The first failure is treating evals as engineering infrastructure. Eval scores are saved and kept inside the repo, and they never get caught in the product review.

This unawareness can lead the engineering team to default to intuition on the release call. Meaning, the product ships/released to the public when the demo feels good and not when the eval says it is good. This is the demo-first pattern documented in building AI products, not prototypes.

The second failure is aggregating a multi-span product into one single quality score. For instance, consider a billing agent that contains multiple roles like financial and compliance risk, and also a summarization feature. When you evaluate multiple parts of the product to the same number, you either ship the billing agent before it is safe, or you block the summarizer for problems it does not actually have. This is where the product starts to fail because multiple components of the same product need their own isolated eval.

The third failure is prompt-first debugging when the important component is model choice, retrieval, or routing.

Cognition’s Devin Fusion post reports 88% of merged PRs came from router-driven model selection. Which model to call is now a runtime decision. It is also important to learn that using one model is not a good idea. But having a model stack where smaller models execute the plan that the larger frontier makes. The diagnostic question has to start there before anyone rewrites a system prompt.

The fourth failure is judging voice and vision products by how the output looks or reads and not by how the user experiences it.

There is no doubt that a voice agent can read back a confirmation number letter-perfect at a speed no caller can copy down in real time. Likewise, a document parser can extract every line item on page one of an invoice and never notice page two. Here, the transcript check passes, as does the field-level pass rate. But the product still fails the person on the other end because expectations are not met.

What This Changes In The Next Eighteen Months

I guess within eighteen months or maybe even less, “show me your eval suite” will be the frontier-product equivalent of “show me your wireframes.” I think it’s time for many PMs and founders, AI engineers to just move on from these kinds of eval statements and refrain from using them at all. It is important that AI PMs, AI engineers, and builders show some accountability and make evals more hyper-specific to the feature that they have introduced. Two best practices push toward that outcome.

The first is ownership.
Eval rubrics now live in product code and not just in vendor dashboards. OpenAI’s deprecations page marks the Evals platform and reusable Prompt objects as deprecated on June 3, 2026, read-only on October 31, and fully shut down on November 30. Any product storing its behavioral contract in that dashboard has to migrate it into a repo, a CI job, and a review process.

The second is routing.
Runtime routing is here, and the eval suite is now the arbiter of routing correctness. Cognition’s Devin Fusion post reports 88% of merged PRs came from router-driven model selection. Every routing decision now needs a test that catches regressions across a portfolio of models.

The specification for your product now lives in the tests, not the tickets. If you do not own it, you do not own the product.

---