Proof-Driven Requirements: The New Agile for Building AI Systems
This post is a war story.
Over the last few weeks, I built and operated an agentic AI system that does something unforgiving: it takes a school district’s usage and assessment data and writes the activity-and-impact report an account manager puts in front of a senior K-12 administrator. The report has to be safe, compliant, and right. A wrong number isn’t a bug; it’s a credibility breach at the exact moment we ask a customer to renew.
I am a founder. Two of my degrees are in Electrical and Computer Engineering, but I haven’t written code professionally since my first corporate job fifteen years ago. I came to agentic engineering because of the roughly 50x increase in agency I experienced the first time I worked this way: things I would have had to brief, queue, and wait for, I could now just build, production-grade.
I built this system directly using frontier LLMs, starting from a single top-level business objective: “build a company second brain for renewals.” In plain terms: one system that pulls together everything we know about a customer — CRM, support tickets, product usage, meeting notes — and turns it into the report and the story for that customer’s renewal.
I planned extensively: brainstorming sessions became specs, specs became implementation plans, plans I executed against. That produced the business logic: what a renewal story is made of, what the report should say. But no amount of planning produced the document that makes the system’s output trustworthy. It exists now — dozens of hard rules, enforced by code and by AI agents reviewing each other’s work — and almost every one was paid for by a failure on a real run.
That experience has a name now.
What changed: requirements used to come first
Traditional software is deterministic. The same input produces the same output, which is what made it possible to specify behavior before building it. Agile already taught us not to specify everything upfront — requirements emerge through iteration. But Agile kept one assumption so safe it never needed stating: once a behavior passes its test, it stays passed. A failure announces itself. AI systems break that assumption. The same prompt can produce different answers depending on context, phrasing, or what got loaded into the window. And the failures don’t look like failures. Nothing crashes. No error fires. The system produces a plausible, confident answer that happens to be wrong — and it looks identical to a correct one.
There’s a measured cost to that. In a Wharton/BCG field experiment, consultants using AI on tasks the model handled well were faster and better across the board; handed a task deliberately chosen to sit just past the model’s ability, they scored 19% worse than colleagues working without AI — the model sounded exactly as confident on both, so people stopped checking. (Mollick calls that boundary the jagged frontier.) Creating an effective harness is how you stop a system from falling asleep at the wheel.
For the consultants, the cost was a weaker deliverable. In a system reporting to institutions, a wrong interpretation isn’t just a failed test; it’s a trust breach, a governance gap, or a compliance failure. That is what makes upfront specification structurally insufficient, not just inconvenient. I’ve seen the alternative called Proof-Driven Requirements (PDR), and the name fits: in an AI system, the most important requirements cannot be written before execution. They are discovered through it.
Notice what the second sequence keeps: the objective still comes first, and so does real planning. What changes is how the behavioral layer gets defined: by proof, not by prediction — because passing once proves almost nothing about passing next time. Naming the pattern is the easy half. The hard half is what you do with a requirement once a failure proves it — that’s what this post is about.
The one idea that organizes everything
Looking back across everything this system forced me to fix, one move kept repeating — not the only fix, but the one that did the most work each time:
Everything starts as prose. You turn the deterministic parts into code. The prose that remains is true LLM judgment — and that isn’t a compromise. It’s the superpower.
Concretely: an agent skill usually begins life as one long prose file — prompt-based instructions the model reads and follows. But look closely: most of those instructions are work that should happen the same way every single time — which query runs next, what shape a record takes, where a file gets written, whether the parts sum to the whole. Written as prose, that is a program in English executed by a probabilistic reader — an instruction the model chooses to follow, and on a long enough run, eventually doesn’t. So you turn it into code.
What you deliberately leave as prose is the true LLM judgment: synthesizing data into a story, writing the narrative, choosing which analyses to run for this customer, deciding whether an anomaly is signal or noise. This is where the model’s non-determinism stops being a liability and becomes the point; and that work is the reason the system is worth building at all.
That conversion takes you most of the way. But it sets up the harder question, the one the rest of this post answers: how do you trust a system where some steps — and some hand-offs between steps — still run on a model reading prose? You build robust tracking and watch where it breaks.
The catalog of failures that wrote the spec
While building our renewal storytelling engine, I encountered failure modes that no planning session had surfaced. Below is a sample of key issues: nineteen, grouped into five families. Each one forced an architectural change. Each one produced a requirement that was proven into existence, not planned.
Every example steps through what failed, the proof of that failure, the requirement it forced into existence, and how I fixed it. I expect any team building multi-step agentic workflows to encounter variants of those failures.
The evaluation loop is the spec
If requirements follow proof, then evaluation stops being a QA stage at the end and becomes part of the specification itself. In practice, that means every run — every district, every regenerated report — is held against explicit criteria:
- Did every phase prove it actually ran — or did something quietly hollow out?
- Did every number in the narrative trace back to a query executed during this run?
- Did the parts sum to the whole at every join and rollup?
- Did the telemetry itself capture reality at write time — or was the recorder dead and self-healed at the end of the run with confident guesswork?
A failure against this list is not just a bug to fix. It is evidence that the specification is incomplete — and the fix is a new permanent gate, not a patch.
The skill this builds
If the catalog has a punchline, it’s this: the work of building reliable AI systems is mostly not prompting. The frontier keeps moving up a layer — 2024 was the year of prompt engineering, 2025 the year of context engineering, 2026 the year of harness engineering — and the next layer up is loop engineering: running the agent against a goal on a schedule, evaluating the result, and feeding the improvement back in. Each layer is the prerequisite for the one above it.
The loop is the optimizer; the harness is the verifier it iterates against. A loop without a harness has nothing checking what it produces — it converges on garbage at machine speed.
The competitive advantage is shifting from “can the agent do the task?” to “can the system prove the task was done correctly?” The second question is harder — and far more defensible. As models commoditize, raw execution stops differentiating anyone. Trust infrastructure doesn’t commoditize, because it can only be earned the way this catalog was: one proven failure at a time.
Who gets to design trust
Here is the shift hiding under everything above: the people building these systems are no longer just engineers. Everyone at a company now needs to be building their own agentic systems — sales, account management, support, product, marketing, ops. As , CEO of Replit, put it: “The roles are collapsing. We have designers shipping code, engineers shipping code, and sales people shipping code. The particular skill is not the bottleneck anymore. It is how ambitious you are, how generative you are, how creative you are, and how good you are at utilizing these tools.”
And whoever builds one is no longer only defining features, workflows, and UX. They are defining acceptable uncertainty, evidence requirements, escalation policies, and risk allocation. Should the model decide autonomously? Should uncertain cases escalate to a human? Fail open or fail closed? Those are not purely engineering questions anymore — they define trust, liability, and customer confidence. You are no longer only designing experiences; you are designing how the system earns trust.
Engineering leadership changes underneath this shift, too: from managing the engineers who write code to solve the company’s problems to enabling the entire company to write code to solve their own — often better than a handoff could, because the person closest to the problem now has the agency to build — while engineering focuses on what genuinely needs them: security, scale, and the core product. Enablement is what makes a non-engineer’s work verifiable instead of forbidden. The org that shares trust-boundary decisions between product and engineering ships; the org that hoards them stalls — and the harness is what makes that sharing safe.
Bottom line
In the agent era, a product requirement can no longer just describe behavior. It has to define verification, evidence, containment, observability, and escalation — the product requirements document evolves from a feature document into a proof document. Because once agents can reason, retrieve, synthesize, and act, the real product question is no longer “what can the system do?” It becomes “what can the system reliably prove?”
Planning produced my intent: a company second brain for renewals. Proof produced everything that made it trustworthy enough to put in front of a customer. The evaluation loop becomes the infrastructure through which the product learns what reliability actually demands.
¹ Some places I’ve seen this emerging term include HUMAIN’s product team and Architecture of Proof. No relation to proof-driven development, a formal-methods practice (2015) of machine-proving code against an upfront spec.
Author: Mohannad Arbaji, ChalkTalk Founder
A note on punctuation: Every em-dash in this post is mine, not an AI’s. I founded an education company whose flagship products include SAT and ACT courses — with entire lessons on em-dashes, semicolons, and proper punctuation. I was using em-dashes long before AI made them suspicious, and I refuse to replace them with commas. A moment of silence for those who did.
