.@neilzegh and I are doing a fireside chat at @aiDotEngineer World's Fair next week about building the next-generation voice interface. Neil comes to this work from a machine learning researcher's perspective, and I approach it from a system engineering perspective. There's obviously a lot of overlap. We both think a lot about, and work on, inference infrastructure at scale, for example. But there are real differences in perspective. Which is useful! I've been updating the Voice AI guide for the World's Fair, and the new section on turn detection is an example of this. I mostly think about turn detection as a systems problem: we stream data into a multi-stage, graph-structured processing pipeline. We combine several components to build the most natural possible turn taking behavior. The new guide section describes the turn detection approach we use in most production code today: - a VAD model - a highly optimized native audio end-of-turn classifier - transcription - single-token EOT tagging by the conversation LLM. A few important ideas here: - These components are designed together. The native audio classifier model was designed from the ground up to work together with VAD and run in parallel with the transcription model. We've worked with all the trancription model providers to evolve the transcription APIs to do what this multi-component approach requires. - Each component has to be very good, for this to work. So the components are co-designed over time, but we can only do this if we create good enough components, judged on their own merits. The modular, co-design approach gives us a big design space to play in. But there's a lot of elbow-greese engineering and engineering "taste" to make this kind of modular system. For example, we trained an open source turn detection model specifically with modularity in mind, starting in 2024, because there wasn't a good turn detection model. (Today, this model, Pipecat Smart Turn, is the most widely used turn detection model. You can contribute data and PRs! Working on this model is a fun way to get started with small audio models.) - As all the models get better, the combined design evolves. @mark_backman had the insight a few months ago that the LLMs we use for most voice agents had gotten good enough at single-token tagging that we could lean on this technique very heavily. (We do lots of benchmarking to track model behavior on isolated performance metrics.) Single-token tagging removed a lot of complexity and improved the cost, by using the main conversation LLM instead of a parallel inference call. - This approach is very, very flexible. For example, you can prompt the conversation LLM to take into account your particular format of account ID numbers when making the single-token tagging decision. In Pipecat, this is implemented as a "user turn strategy," which is a pluggable system that allows you to modify every component and every parameter of turn detection. Different use cases benefit from different turn detection implementations. At the risk of putting words in Neil's mouth, I think by default he approaches problems like turn detection as opportunities to solve hard things at the model level. You can bake turn detection into a transcription model and collapse this whole graph processing structure I just described into the weights! My response to that is basically, yes please, as long as the "bigger model" approach yields behavior that's as accurate, flexible, and configurable as the "systems engineering" approach. Or, at least, can be turned off for the situations where it's not the right tool. One of the super fun things about working on @pipecat_ai, for me, is getting to work closely with researchers like Neil. Come hang out with us next week in San Francisco.

When I was in grad school, "Doctor Flemson or something" was one of our catch phrases. Funny only to us, but that's why you go to grad school, to find the people who share your obsessions and think the same niche jokes are funny. That phrase is from the 1987 Apple "Knowledge Navigator" concept video. It's a great video. Squarely in the lineage of Vannevar Bush, J. C. R. Licklider, and Alan Kay, but prescient in its own right. The video shows a foldable tablet, a touch-screen interface, a conversational voice assistant with a strong personality and access to both personal and global information, realtime video generation, realtime computer vision, seamless video call integration, delegation of complex tasks for autonomous execution, and what we might today call "continual learning." None of these ideas are particularly surprising to us in 2026. But Apple made this video 40 years ago. Before the Internet. Before most computer users had even seen a mouse, much less a touch screen. What *will* be surprising to some people is that we can actually build this today. The team at @tavus re-created "Knowledge Navigator" using their conversational video agent tech stack and posted a video, captured in a single take. "Re-interpreted" is probably an even better description of this project than "re-created." The original video features an academic (portrayed by an actor) preparing a lecture. The Tavus video captures actual use of this real personal assistant application: some recreational 3d printing, some vibe coding with Claude Code. "Let me model you a tapered adapter right now," delivered in my terrible version of a British accent, is my new catch phrase. Response times are very fast, including the 3d model generation. Tavus worked with @cerebras on the video. I posted recently about how fast Kimi K2.6 is on Cerebras and how well the model does on my benchmarks. If you haven't spent much time with K2.6, it's worth watching the video just to get an intuitive sense of the speed, natural conversation dynamics, and structured data strength of this model. I'm always, always impressed by the Tavus team. (After grad school, you move to San Francisco to find even more people who share your obsessions.) @hassaanraza wrote a wonderful, meditative blog post on the experience of building and using this new version of a knowledge navigator. Link in thread.

My favorite annual AI event is next week in San Francisco. Come hang out with me at the @aiDotEngineer World's Fair. Go buy a ticket. Or, if you're a student, there's a great volunteer program. The organizing team does a great job making the World's Fair a conference with great talks, an opportunity for deep-dive learning across multiple AI engineering domains, a gathering of people building the infrastructure for AI adoption, and an opportunity to see early what everyone will be talking about next year. I'm doing a couple of talks. One on voice interfaces, with @neilzegh. And one about experimenting with new building blocks for "AI-native" software, software patterns for applications we couldn't have built at all before now and that we'll take for granted in a few years! I get more requests than I can keep up with, via LinkedIn and email, to meet and talk about AI engineering, voice AI, and @pipecat_ai. Most of the year, my default response is, "I would love to but I can't, if you make a PR I'll try hard to look at it." AI Engineer World's Fair is the one time of year when I can say: if you'll be at the World's Fair, come find me. We'll geek out about what you're building and all the crazy ideas we both have!