Model routing is a hot topic but there are two challenges to doing routing for coding agents: 1) Even if different models can pass the task, there are subtle differences in behavior & style that mean they aren't perfectly interchangeable. 2) The initial agent prompt isn't enough to know the difficulty of the task. "Fix xyz bug" could be a one-line edge case or could require rearchitecting your entire product; you can't know until you've actually investigated the code. How do you solve these problems? Well, you need evals that account for style and behavior, not just pass/fail. And you need the agent to be able to dynamically update and re-route. We built Devin Fusion with both of these points in mind and found that it reduces costs by 30-40% while still maintaining the frontier intelligence "smell":

Happy 4th! Proud to be an American today and every day.

If you’ve ever wondered why we will need 100X more AI inference in the future, and what it’s going to be driven by, this is another good example. Devin pushes forward an idea of agentic mapreduce, which means we’ll now have swarms of agents that are processing large amounts of data (code) to handle tasks that humans never could have done before. “Devin maps relevant signals across the repo, fans out focused agents over bounded shards, reduces their findings into one report, then verifies serious vulnerabilities in isolated sandboxes before marking them confirmed.” In this case it’s code security, but there are tons of other use-cases in code and knowledge work. We see this at Box with customers that want to process and understand millions of documents for risk, insights, relationships, and more. This will play out in pharma, banking, and many other industries across all forms of unstructured data. As an aside, these types of capabilities are generally only possible when you can deploy a variety of models (both the frontier and lower cost) because of the sheer amount of tokens that go into these use-cases. This is going to be a major value proposition for the applied AI layer.

Demand for inference compute will grow exponentially at least as fast as the METR curve grows (and arguably faster). Excited to have more great chips on the market - congrats to the @Etched team!

Frankly I think this is the reason Devin is having such a comeback Nobody is really doubting the productivity gains of AI, and I would guess that companies would still be willing to pay the exponential if they must... But token spend is scaled and open source is now really good. It makes sense we are now spending energy to curb the runaway train Extreme high-growth startups are only now thinking about token spend, but this has been an enterprise (read: Publicly Traded Company) concern since day 1 Want to understand how Cognition so quickly grabbed all the big banks and giant Fortune 100 enterprises as customers? Aligned incentives is the answer. 1. Being an independent company Because we are not a model lab with $100B raised and $1T of data center commitments, we don't need to "catch up" by selling increasingly more expensive tokens Nor do we need to push a specific model family to make margins. Our only calculus is - "Is this the best model for the job?" - "Can we make the user more productive?" - "Can we save the user money?" (increasingly) This comes in the form of post-training research (building cheap specifically tuned coding models) new coding evals (FrontierCode benchmarks) model routing (a lot behind-the-scenes of Devin's cloud harness). You should be skeptical of an Italian restaurant pushing the expensive market price specials. Just like you should be skeptical of a model lab pushing the newest most expensive model 2. Enterprise cost controls As a pre-requisite to selling enterprise contracts to the biggest companies in the world, you need really good spend controls. These banks and big conglomerates have been token-sensitive since day 1. They saw the writing on the exponential. For this reason, Devin has the most complete & robust spend controls of any coding agent on the market. The boring stuff of orgs, users, scopes, limits. But it matters. 3. AI Productivity alignment Cognition has an "AI Productivity Guarantee" That means if Devin delivers less engineering value than you’re paying for, Cognition will fund your usage until it does, up to $10 million. This is the tip of the iceberg and the one thing about Cognition that has been most novel to me since joining. Everything (and I mean everything) in our GTM motion is oriented around ROI. Every conversation is rooted in the actual engineering tickets we are taking off the backlog. I can only imagine what it would be like if instead conversations were rooted in "how can we entice users to burn through tokens"

first day of work at @cognition excited to build the future here

In 1958 Ian Donald published what is now the foundational paper on medical ultrasound for obstetrics. He was so widely ridiculed by his colleagues at the time that they nicknamed him Mad Donald [1], and one said ultrasound would be useful only to "a gynecologist who was blind and had lost the use of both hands" [2]. Another noted that he'd invented a £10,000 device to undertake a task that could be accomplished with a £0.02 rubber glove. [3] Last week, my wife and I welcomed our first child into the world. She had a rare pregnancy complication that until recently would have meant only a 28% intact survival rate for our newborn. But in 2013 US guidance was updated to add regular preventative screening for her condition at the 20-week ultrasound, and with early detection the survival rate is ~99%. (In the UK, preventative screening is still not recommended, for reasons like "it is not known how accurate screening tests are" [4].) The entire history of radiology is people expressing skepticism about the work done by innovators. I for one am grateful for folks like @DavidSHolz building new classes of devices that can help us see things in new ways, and I'll be rooting for their success. Hand in hand with my wife and our healthy baby boy.

huge shoutout to @ryanbai who spearheaded our AI Productivity Guarantee work!

Congratulations to @axliu42 on receiving the ACM Doctoral Dissertation Award for his work on learning-theoretic foundations for understanding quantum systems. Allen joined Cognition's research team this week while continuing as a professor at NYU. We’re honored and excited to work with him. x.com/TheOfficialACM/status/…

we are going to need up our own human orchestration as much as we're upping our agent orchestration:

Really thoughtful piece by @saranormous !

A new top scorer just one day after our benchmark released! Especially strong on the hardest tasks: 13.4% -> 29.3% on FrontierCode Diamond compared to Opus 4.8.

SWE-Bench style grading has been the standard for years now - you ask the agent to solve an issue and then run its code on a pre-constructed unit test. The problem is that passing a unit test is only one part of writing production-ready code. You also want to evaluate agents on a number of other axes, including scope, coding style, and unintended side effects. The result is our new benchmark FrontierCode - which has ~80% fewer false positives and for which the best model (Opus 4.8) only scores 13%! "Where others grade like a CI, FrontierCode grades like a tech lead."

Measuring someone's productivity by their token usage is a horrible idea. Giving everyone the same fixed token budget isn't much better. So what's the right way to roll out AI across your org? We built a system to measure how many productive engineering hours every Devin task is worth, validated against a dataset of real engineers’ times estimates. The goal is to answer the fundamental question that companies are grappling with: how much real value are you getting from each of your agent sessions? On top of that, we're giving an AI productivity guarantee! Now if Devin delivers less engineering value than you're paying for, we fund your usage until it does. The whole industry needs to move from measuring activity to measuring output. We hope to see more AI companies taking this approach.

Devin pushing the frontier on ECDSA circuits!

At @harvey, the engineering team integrated Spectre — their internal background agent — into Devin Desktop. Now Spectre's organizational context can live on every engineer's laptop and flow across their favorite agents.

Excited to partner with @Carahsoft! An important step for more of the public sector to have access to the same tools and technology that the private sector is adopting en masse today.

Standalone IDEs have about 6 months left to live. An interface for manually editing and refactoring doesn’t need to exist if you're not manually editing and refactoring anymore. So what's the right interface for a dev to be working in for 8h / day? Some parts are obvious: you want to be able to spin up agents (either local or cloud agents) and to have a clean interface to keep up with all of your parallel running agents. Then you want to be able to get into the weeds whenever needed for last-mile fixes and review. But as software engineering continues to evolve we will see more and more of the lifecycle get reinvented. How do you build a single surface that allows you to plan, spec, prototype, debug, review, QA? Bringing Devin and Windsurf together has been our vision ever since the acquisition. Devin Desktop is our first shot at what this looks like. Excited to make this a reality today!

This one doesn't fit in a wave either! See you tomorrow