AI-assisted analysis. See our editorial policy.
Human editorial review not recorded
Philipp Schmid gives a reason for a design decision that most API documentation would state without justifying, and the reason is the interesting part.
Google's Interactions API abandons the alternating user-turn, model-turn structure that language model APIs inherited from chat. Not because it is inelegant, but because an agent trace was never a conversation. It contains reasoning, function calls, environment responses, compaction — steps that are not turns, and forcing them into a turn-shaped container distorted what was actually happening (19:30).
The replacement is a steps model, where each action inside an interaction is its own entry with its own type. Schmid's example of what was wrong before is small and telling: returning a function result required labelling it as a user turn, which it plainly was not.
State moved to the server, and what that removes
The second departure is more immediately practical.
Conversation history no longer lives in the client. A call returns an identifier; passing it into the next call continues the thread with the accumulated context intact (15:31). The array of prior turns that every application maintained — appending, trimming, worrying about what got cut — is gone (16:21).
This is a smaller change technically than it is operationally. Managing that array was where a category of subtle bugs lived: context silently truncated at the wrong boundary, a tool result dropped, history diverging between retries. Moving it server-side does not make those concerns disappear, but it moves them to one place with one implementation.
The same call shape covers models and agents — you change a field rather than an integration (14:29). Server-side tools such as search and URL retrieval combine with client-defined functions in a single tools field, with the server-side ones executing without a client round trip (18:16), so a model can decide to search and then call your function without the intermediate hop.
The environment is the substantive change
The part that alters what is buildable is the one that sounds like infrastructure detail.
Managed agents receive their own remote environment rather than executing on the caller's machine (21:42). Schmid frames the motivation in terms of scale rather than convenience: if you want to offer agent execution to your customers, each needs isolation, and providing that yourself means operating infrastructure that has nothing to do with your product.
Environments persist and are addressable, which produces the capability worth noticing. A research agent runs, writes files, and returns an environment identifier. An application-builder agent is handed the same identifier and reads those files directly (22:23). Context passes between agents through a filesystem rather than through a conversation history.
Schmid is explicit about the consequence: files scaffolded into the environment are deliberately not part of model input (23:10). A large video, a repository, a dataset can sit where the agent can read them without occupying the context budget. That is a different resource model from everything built on stuffing material into a prompt, and it removes the ceiling that shaped a generation of retrieval architectures.
The other half
Thor Schaeff's opening section covers the real-time path, and one property of it deserves separating from the demonstration.
The live interface runs over a stateful socket carrying text, audio and video frames, with the model reasoning about what it sees rather than only what it hears (5:48). Voice is handled natively speech to speech across ninety languages in preview, with interruption supported and transcripts produced for both directions (6:26).
Native speech to speech is the detail that matters. The conventional arrangement — recognise speech, run a text model, synthesise a reply — loses everything that is not words at the first step and cannot recover it at the last. A model working in audio throughout keeps the material that makes a conversation feel like one, which is why the language switching in the demo happens without a mode change.
What the three changes have in common
Read together, the steps model, server-side state and persistent environments are the same decision applied three times: stop pretending an agent is a chatbot with extra abilities.
A chatbot's state is a transcript, its unit of work is a turn, and its context is whatever fits in the prompt. An agent's state is a task in progress, its unit of work is a step, and its context is an environment it can read from. Each of Google's departures replaces a chat assumption with an agent one, and the fact that they arrive together suggests the accumulated cost of the old model had become obvious internally before it was announced.
For anyone building on top of this, the practical question is which of their own abstractions still carry the chat assumption. History arrays, turn-based logging, retrieval that exists to compress material into a prompt — each was a reasonable answer to a constraint that has now been moved somewhere else.
Key numbers
- 90 languages
- native speech-to-speech coverage in preview 6:26
Talk chapters
Key takeaways
- 01
The real-time path runs over a stateful socket carrying text, audio and video frames, with the model reasoning over what it sees rather than only what it hears. 5:48
- 02
Voice is handled natively speech-to-speech across ninety languages in preview, with interruption supported and transcripts produced for both the incoming and outgoing audio. 6:26
- 03
The Interactions API deliberately gives models and agents the same call shape, so switching from one to the other is a field change rather than a different integration. 14:29
- 04
Conversation state moved server-side: a call returns an identifier that continues the thread, removing the client-side history array developers previously had to assemble and trim. 15:31
- 05
Server-side tools such as search and URL retrieval combine with client-side custom functions in a single tools field, with the server-side ones executing without a client round trip. 18:16
- 06
The data model replaces alternating user and model turns with discrete steps, because reasoning, function calls, environment responses and compaction were never chat turns and forcing them into that shape distorted the trace. 19:30
- 07
Managed agents each receive their own remote environment rather than running on the caller's machine, which is what makes per-customer agent execution possible without operating that infrastructure yourself. 21:42
- 08
Environments are persistent and addressable, and can be shared between agents — a research agent's files become an application builder's inputs through the filesystem rather than the context window. 22:23
- 09
Scaffolded environment files are deliberately excluded from model input, so large artefacts stay readable by the agent without consuming the context budget. 23:10
Entities mentioned
Organizations
Related talks
Four of Google's model, product and search leads on what changes once agents run for hours rather than seconds, and the most quotable argument comes from Dean: the constraint is moving out of the model and into the tools around it. By Amdahl's law, an agent spending half its time in tools built for human-speed interaction cannot gain more than a doubling however fast the model becomes — which reframes a great deal of current infrastructure work as latency debt. Their internal response is concrete: rewriting Python tooling into Go, framed as a fully specified translation task rather than an open prompt, produced order-of-magnitude speedups overnight. Reid supplies the counterweight from Search, where acceptable latency turns out to scale with how much work is being taken off the user rather than being a fixed budget. Woodward's detail is the quietest and perhaps the most telling: teams that have stopped writing product documents for humans and now write context files for models to act on directly.
Pichai's framing of Google as the buffer between people and the raw internet is offered as continuity — search did it, browsers did it, agents do it more — and it is also the most contested claim in the industry, because a buffer decides what passes through. He reaches immediately for the counterweight, the connection people feel to creators they follow, which is precisely the tension the company is currently managing without resolving. Two answers are sharper than the format usually produces. On competition he describes participants running on different pre-training and release cadences rather than at different speeds in one race, which is a more honest account than the leaderboard framing and comes from someone with an interest in leaderboards. On security he acknowledges models improving at cyber work, which is the one domain where better capability does not obviously net out positive, since an attacker needs one vulnerability and a defender needs all of them.
The anecdote that opens the panel does the work: a robot asked to pick up the extinct animal selected a dinosaur toy, with nothing in its training data connecting the phrase to the object. That transfer from language models into machines with hands is the premise of the current wave. What the practitioners then describe is where it stops. Physical intelligence is about exerting force and using a body to do it, which is knowledge about consequences — the one thing a corpus of internet images contains almost nothing about. The humanoid question gets an honest treatment: not that human shape is optimal, but that the world is already built for it, plus a development-loop argument about collecting data and deploying on the same hardware. The most useful passage is scepticism about the field's favourite shortcut: generated video looks realistic and does not hold up for dexterous manipulation, because looking right and being physically consistent are different properties.
The observation that organises this session is not about capability but about attention. Engineers increasingly file a ticket rather than opening an editor, and the code comes back — which changes what the surrounding tools are for. If the agent works while you do something else, the conversation between you is no longer a workspace; it is the mechanism by which the agent surfaces a question it cannot resolve alone. Interfaces built for continuous conversation optimise for flow, and interfaces built for interruption should optimise for the opposite. A runtime constraint follows immediately: an agent that starts a long-running job cannot block until it finishes, which turns out to be a workflow-engine problem rather than a model one. The panel's closing formulation — that deciding what to build is the hard skill and always was — reads as reassurance and functions as a warning, since that judgement is downstream of exactly the work now being delegated.
The most honest moment here is an aside about how the presenters have tracked their own projects: plans in documents, plans in spreadsheets, plans in bug comments, and once a plan written on a receipt. That describes the actual category being addressed — not software nobody has built, but the small internal tool every team improvises badly because building it properly was never worth the effort. The demo turns on a single question: the generated app is strictly single player, so what happens when you want to share it with the team? That boundary is where improvised tools historically died, because it is where accounts, shared storage and access rules begin. Here it is crossed in one step, with the access rules generated and deployed automatically — which is convenient, and is also the moment the application acquires obligations nobody reviewed.
The case for local inference is made in three clauses representing different kinds of constraint: cost, where a local model removes an API call; availability, where the application works on a flight; and data, where requirements prevent information leaving the device. Only the third changes what is buildable rather than what is affordable. What makes this newly practical is unglamorous — instruction set extensions integrated into the runtime rather than a modelling breakthrough — with around thirty per cent improvement reported in image editing functions. The guidance on fine-tuning inverts the usual advice: adaptation matters most for the smallest models, because their capability budget is already spent and getting them to perform on your problem means spending some of it there. The two examples do the real work, since neither is a cheaper version of a cloud application.