AI-assisted analysis. See our editorial policy.
Human editorial review not recorded
Joe Cudby asks the room a question and answers it himself, because he already knows what the honest answer is.
How do you measure developer productivity today, before any AI tooling? What baseline exists? His suspicion, stated plainly, is that most organisations do not have one — that the current approach is ad hoc, partial, assembled from whatever was available (14:26).
That is the problem underneath every disappointing AI impact report. Not that the effect is hard to detect, but that there is nothing to detect it against.
Adoption is not an outcome, and treating it as one hides the failure
The session's structural argument is about sequencing, and it corrects an error that is nearly universal in adoption programmes.
Organisations want the return figure. They want to know what the licences bought, expressed in something a finance function recognises. So they skip to it — and arrive at a number that cannot be interpreted, because they never established whether the tools were adopted, whether people engaged with them, or whether the experience was good enough to change behaviour (15:04).
Those are leading indicators. The financial impact is a lagging one. Reading a lagging indicator without its leading indicators produces a result you cannot act on: a poor number might mean the tools do not work, or that nobody uses them, or that they are used for the wrong tasks. Three different problems with three different remedies, indistinguishable from the outcome alone.
Cudby's related point is that engineering productivity measurement — cycle time, team velocity, delivery throughput — is a prerequisite rather than a parallel effort (27:36). An organisation that could not measure its delivery before cannot measure a change to it now.
Enablement is not optional, and the research says so
The finding he draws from this year's industry research is the one that should reset expectations for anyone planning a rollout: structured enablement determines outcomes. You cannot switch these tools on and hope (4:23).
This matters because the default deployment pattern assumes the opposite. Tools are procured, licences distributed, and adoption is expected to follow from capability — the reasoning being that developers who find something useful will use it. It is a reasonable assumption and it does not hold, and the sessions across this conference season converge on why.
The talks describing successful outcomes are consistently describing changed practice, not changed tooling. Specification-first workflows. Deliberate context management. Review discipline. Each requires someone to teach it, and none of it arrives with a licence key.
Why this deserves attention despite the format
A session featuring a vendor, a measurement partner and a joint customer carries an obvious framing, and the framing is visible.
What survives it is the diagnostic sequence, which is independent of any product. Establish a baseline before deploying. Instrument adoption and engagement before reaching for return. Treat enablement as the intervention rather than as documentation. Accept that the leading indicators are the ones you can actually act on while the lagging one is still resolving.
Placed alongside the measurement work Amazon's own platform organisation presented at this conference — where individual velocity was found to revert to the team mean, and time-saved arithmetic broke down entirely — a consistent position emerges from an unlikely direction. The vendors selling AI development tooling are, at this conference, being noticeably careful about what can be claimed.
That care is itself information. A category confident of a dramatic effect does not spend its stage time on baselines and leading indicators.
Talk chapters
Key takeaways
- 01
His opening question is what productivity baseline existed before any AI tooling, and his stated expectation is that most organisations have none — which is why impact reporting is unreadable. 14:26
- 02
Organisations skip to the return figure without first establishing adoption, engagement and whether the experience was good enough to change behaviour. 15:04
- 03
Adoption and engagement are leading indicators while financial impact lags, and a poor lagging figure alone cannot distinguish between three different problems with three different remedies. 15:11
- 04
This year's industry research points to structured enablement as decisive: these tools cannot simply be switched on in the expectation that usefulness produces adoption. 4:23
- 05
Measuring engineering delivery — cycle time, team velocity — is treated as a prerequisite for measuring AI impact rather than as a parallel exercise. 27:36
Entities mentioned
Related talks
The practical counterpart to the argument made elsewhere this season that specification is what contains model entropy. Raval and Harris name the failure they are addressing precisely — a prompt-and-pray loop in which working code arrives with no record of what the model assumed, which requirements were fuzzy, what design was chosen or why, leaving nothing to review and nothing to iterate against when a defect surfaces months later. Their answer is three committed markdown artefacts: requirements written in a structured requirements syntax with acceptance criteria attached to each user story, a design document carrying technical decisions together with the reasoning behind them, and a task list whose entries cite the requirement numbers they satisfy. The traceability is the point — a reviewer questioning a decision in a pull request can follow it back through the task to the design to the requirement, all in the same repository. Notably they keep the human between each phase rather than after it, with the agent surfacing ambiguity as questions before proceeding.
Drawn from a year of engagements with more than a hundred companies, this is the most direct challenge in the season's programme to the assumption that faster code generation produces faster delivery. Mishra and Raja open with external evidence rather than their own: an industry study putting realised velocity gains in the ten to fifteen per cent range, and a controlled experiment in which developers using AI estimated themselves roughly a fifth more productive while measurement showed them a fifth slower. Their diagnosis is that both prevailing working styles fail for opposite reasons. Handing an ambiguous problem to an agent and awaiting a finished result produces a volume of code the developer must nonetheless sign for and cannot confidently review, so it stalls before production. The senior engineer's alternative — decomposing the work personally and inserting AI into narrow slots — keeps the intellectual load exactly where it was, and leaves the surrounding process untouched, so hours saved in editing are consumed by the meetings that process still requires.
Brooker builds the definition from the bottom up rather than asserting it, using a deliberately absurd arithmetic task to separate three categories: what a model computes reliably as a fixed function of its input, what merely needs to arrive in the system prompt, and what genuinely requires reaching into the world. Only the third category justifies a tool, and the distinction matters because most production disappointment comes from tools built for the first two. His working definition follows — a system given a goal that loops between inference and tool calls until it reaches one — with the observation that modern agents increasingly embed code in their definitions, not for expressiveness but because replacing inference steps with deterministic code improves reliability while lowering both latency and cost. The remainder covers what production actually demands around that loop: somewhere to run, memory that persists preferences, a gateway to internal and external tools, evaluation, and formal methods applied to policy.
The most concrete attempt this conference season to answer a question the agentic coding sessions mostly leave open: if commit counts and hours saved are the wrong measures, what replaces them? Otto's account is unusually specific about why the obvious alternative fails — summing the small time savings a platform team delivers produces figures exceeding a hundred per cent of a developer's time, and a minute returned is not code in production. Their replacement borrows from Amazon's retail supply chain, where cost to serve measures what it takes to place a package on a doorstep, and applies the same shape to software: total cost divided by units of delivery, with the unit chosen to fit the team. The supporting research is the more quotable finding — across tens of thousands of developers over five years, individual velocity reverts to the team's mean, making team velocity the strongest predictor of both individual output and perceived productivity, which is the empirical case against measuring individuals at all.
The framing statistic is organisational rather than technical: around eighty per cent of organisations expected to have platform engineering teams going into 2026, up from about forty-five per cent a couple of years earlier. The interesting part is the doubling. The problem described is teams solving the same problems separately, producing inconsistency and redundancy — dangerous not because of duplicated effort but because each independent solution has its own security properties, and the organisation's real posture is the weakest rather than the average. The most valuable content is that two financial services organisations went in diametrically opposite directions on workload identity and both are described as working, which implies the choice is determined by context rather than by a general answer. The honest note follows immediately: even with standardised patterns the result remains fragmented.
The line that explains this session comes from the customer in the last ten minutes: they are preparing for a world where metadata is how agent-based systems find the data they need and access it through the controls being built. That relocates a function — governance has spent two decades as compliance activity describing data that people locate by other means, and if agents navigate by the catalogue then the catalogue stops describing the access path and becomes it. An incomplete catalogue is a documentation problem when humans can ask a colleague; an agent has no such workaround. The most honest moment addresses the perennial failure that rules get written and ignored, with enforcement rather than publication as the argument. Generated descriptions and greyed-out classification suggestions divide the labour correctly, keeping a person accountable while removing the burden of finding candidates.