AI-assisted analysis. See our editorial policy.
Human editorial review not recorded
The story that should outlive this session is about a question that worked perfectly and cost 1.3 million tokens (33:07).
The details are what make it valuable. The answer was good. It passed their question-and-answer checks and their acceptance testing. The tool calls the agent made were sensible. By every measure of correctness the team had built, the system performed exactly as intended — and consumed enough tokens on a single query to have destroyed throughput had it reached production at scale (34:00).
What that failure mode actually is
This is a category of defect that traditional software mostly does not have. A function returning the right answer inefficiently is a performance problem: measurable, profileable, bounded. An agent returning the right answer after pulling far more context than it needed is something else, because the cost is incurred inside a reasoning process with no fixed shape.
The team's response was to extend their agent framework for per-turn reasoning visibility (32:10), and then set limits on how much data the agent could pull and under what conditions.
The sequence matters more than the fix. They could not have written that guardrail before observing the behaviour, because nothing in the requirements suggested a question like that one would go wide. Observability was not an operational nicety added after launch. It was the instrument that revealed a class of failure their correctness testing was structurally unable to detect.
Why they split the agent from the data
The architectural decision that made the rest possible was separating agent logic from data logic (15:44).
The reasoning is practical. Their data layer required parallel processing to be efficient, and embedding that inside the agent code would have made it hard to maintain and hard to scale (15:44). Splitting it lets each side scale independently — sometimes the agent computation needs more capacity, sometimes the data layer does, and traffic is genuinely hard to predict (16:40).
That last point is specific to their situation in an instructive way. Fantasy football traffic is not merely spiky; it is spiky on a published schedule, concentrated in the minutes before kickoff. Most systems would kill for that predictability, and they still describe the load as hard to forecast — which suggests the variance comes from what users ask rather than when they ask it.
Which loops back to the 1.3-million-token question. If a single query can consume three orders of magnitude more resource than a typical one, then the distribution of question complexity dominates the distribution of question timing, and capacity planning against traffic patterns will not save you.
The performance target and what it forced
Their commitment is a first response inside five seconds and full analysis inside thirty at the 95th percentile (6:27).
A latency budget of that shape constrains architecture more than most requirements do. It rules out serial tool calls beyond a small number, rules out reasoning depth that cannot be bounded, and makes every retry expensive. The interesting consequence is that it pushes in the same direction as the token guardrail — both punish an agent that explores widely before answering.
Their actual advice
The recommendation they end on is deliberately unglamorous: when you are trying to reach production, find the simplest pattern that delivers the benefit and pursue it, then analyse with real data once you are running (38:06).
Set against everything that preceded it, this is a stronger claim than it sounds. They are not saying sophistication is unnecessary. They are saying they could not have designed the right sophistication in advance, because the failure that mattered most was invisible until a real system produced it.
Which is a reasonable argument for shipping early, and an uncomfortable one for anyone whose organisation requires the failure modes to be enumerated before the system is allowed to run.
Key numbers
Talk chapters
Key takeaways
- 01
A single question consumed 1.3 million tokens while passing every correctness check the team had, including acceptance testing and tool-call review. 33:07
- 02
The guardrail bounding how much data the agent could pull was only writable after observing behaviour, not derivable from requirements. 34:00
- 03
They extended the agent framework for per-turn reasoning visibility, which is what surfaced the failure their testing could not. 32:10
- 04
Agent logic and data logic are separated so each scales independently, because embedding a parallel-processing data layer inside agent code does not maintain or scale. 15:44
- 05
Their closing advice is to find the simplest pattern that delivers the benefit, ship it, and analyse with real production data afterwards. 38:06
Entities mentioned
Related talks
The practical counterpart to the argument made elsewhere this season that specification is what contains model entropy. Raval and Harris name the failure they are addressing precisely — a prompt-and-pray loop in which working code arrives with no record of what the model assumed, which requirements were fuzzy, what design was chosen or why, leaving nothing to review and nothing to iterate against when a defect surfaces months later. Their answer is three committed markdown artefacts: requirements written in a structured requirements syntax with acceptance criteria attached to each user story, a design document carrying technical decisions together with the reasoning behind them, and a task list whose entries cite the requirement numbers they satisfy. The traceability is the point — a reviewer questioning a decision in a pull request can follow it back through the task to the design to the requirement, all in the same repository. Notably they keep the human between each phase rather than after it, with the agent surfacing ambiguity as questions before proceeding.
Drawn from a year of engagements with more than a hundred companies, this is the most direct challenge in the season's programme to the assumption that faster code generation produces faster delivery. Mishra and Raja open with external evidence rather than their own: an industry study putting realised velocity gains in the ten to fifteen per cent range, and a controlled experiment in which developers using AI estimated themselves roughly a fifth more productive while measurement showed them a fifth slower. Their diagnosis is that both prevailing working styles fail for opposite reasons. Handing an ambiguous problem to an agent and awaiting a finished result produces a volume of code the developer must nonetheless sign for and cannot confidently review, so it stalls before production. The senior engineer's alternative — decomposing the work personally and inserting AI into narrow slots — keeps the intellectual load exactly where it was, and leaves the surrounding process untouched, so hours saved in editing are consumed by the meetings that process still requires.
Brooker builds the definition from the bottom up rather than asserting it, using a deliberately absurd arithmetic task to separate three categories: what a model computes reliably as a fixed function of its input, what merely needs to arrive in the system prompt, and what genuinely requires reaching into the world. Only the third category justifies a tool, and the distinction matters because most production disappointment comes from tools built for the first two. His working definition follows — a system given a goal that loops between inference and tool calls until it reaches one — with the observation that modern agents increasingly embed code in their definitions, not for expressiveness but because replacing inference steps with deterministic code improves reliability while lowering both latency and cost. The remainder covers what production actually demands around that loop: somewhere to run, memory that persists preferences, a gateway to internal and external tools, evaluation, and formal methods applied to policy.
The most concrete attempt this conference season to answer a question the agentic coding sessions mostly leave open: if commit counts and hours saved are the wrong measures, what replaces them? Otto's account is unusually specific about why the obvious alternative fails — summing the small time savings a platform team delivers produces figures exceeding a hundred per cent of a developer's time, and a minute returned is not code in production. Their replacement borrows from Amazon's retail supply chain, where cost to serve measures what it takes to place a package on a doorstep, and applies the same shape to software: total cost divided by units of delivery, with the unit chosen to fit the team. The supporting research is the more quotable finding — across tens of thousands of developers over five years, individual velocity reverts to the team's mean, making team velocity the strongest predictor of both individual output and perceived productivity, which is the empirical case against measuring individuals at all.
The framing statistic is organisational rather than technical: around eighty per cent of organisations expected to have platform engineering teams going into 2026, up from about forty-five per cent a couple of years earlier. The interesting part is the doubling. The problem described is teams solving the same problems separately, producing inconsistency and redundancy — dangerous not because of duplicated effort but because each independent solution has its own security properties, and the organisation's real posture is the weakest rather than the average. The most valuable content is that two financial services organisations went in diametrically opposite directions on workload identity and both are described as working, which implies the choice is determined by context rather than by a general answer. The honest note follows immediately: even with standardised patterns the result remains fragmented.
The line that explains this session comes from the customer in the last ten minutes: they are preparing for a world where metadata is how agent-based systems find the data they need and access it through the controls being built. That relocates a function — governance has spent two decades as compliance activity describing data that people locate by other means, and if agents navigate by the catalogue then the catalogue stops describing the access path and becomes it. An incomplete catalogue is a documentation problem when humans can ask a colleague; an agent has no such workaround. The most honest moment addresses the perennial failure that rules get written and ignored, with enforcement rather than publication as the argument. Generated descriptions and greyed-out classification suggestions divide the labour correctly, keeping a person accountable while removing the burden of finding candidates.