Imagine every pixel on your screen, streamed live directly from a model. No HTML, no layout engine, no code. Just exactly what you want to see. @eddiejiao_obj, @drewocarr and I built a prototype to see how this could actually work, and set out to make it real. We're calling it Flipbook. (1/5)

New realtime diffusion experiment: ever had an image gen output that you love and wished you could edit it directly? Turns out you can live edit image gen outputs in realtime if you apply enough inference optimization! Just click to move objects around, change their color, even edit text:

Fit check w @eddiejiao_obj - come say hi #config

Hivemind has been sitting in my texts for months scheming on not my but our behalf (arranging dinners, intros, and ofc gossip) like a nosy well-meaning village elder v excited for the world to see what this feels like! Huge labor of wit, wisdom, and love from @jasonyuan

we started a company!! so, we’re tackling continual learning: what’s the learning algorithm to take arbitrary data — documents, conversations, the models’ own experience — and make better models? how do we scale compute in the same way we’ve already seen with pre-training and inference time, but scaling on the same data we see as humans, day after day with no labels, no rewards? A lot of the ingredients are out there already (rl, distillation, long-context, sparse / param-efficient architectures, etc.). our team is at the frontier of these topics, and we’re singularly focused on this. we want to understand this problem better than anyone else in the world. nobody’s solved this problem yet, but even today it’s extremely greenfield opportunity to co-develop research & useful products. in our space, how people interact with the models defines what the data distribution is - and working on this problem end-to-end, from core science to end user, gives us incredible freedom to define the problem and imagine new kinds of experiences. i expect we’ll use models that continually learn much differently than we’re using them today. it’ll feel different when the models _just know_, and build on our thinking and direction in ways we can’t even imagine. we don’t even know the queries we’re not asking, the things we would do but aren’t able to today. i’m so excited to share what we’re doing with the world in the coming months!! and the team is extremely cracked :) tackling this grand challenge and working alongside @jxmnop @EyubogluSabri @dan_biderman @MayeeChen @__howardchen @shizhehe and many others has made every day so fun. come work with us!

Blew out my birthday candles Rube Goldberg style this year! Thanks to my ingenious, fun-loving friends and family for this contraption and so much more. Onwards 🚀

Latest latentspace pod is excellent. @EthanHe_42 really gets it and lays out a lot of the thinking that led us to Flipbook and how to think about the future of generative UI agents 🤝 video gen 🤝 users

Proud of my bro @suedadam for already shipping some crazy voodoo mac os magic in just his first few weeks @OpenAI ! Check out remote computer use in Codex, it even works when your computer is locked 🔓🪄

awesome work from the @thinkymachines team. Flipbook is inspired by the same idea; for human AI collaboration to be better than the model alone, we need higher bandwidth interaction. Not a scaffold/harness for turn based interaction, but where the model is the interface itself

wow I just learned @FastCompany did a writeup on us for Flipbook! Love how our vision actually made it through to people

(totally unrelated to Flipbook) My friends and I organize a creative make-a-thon summer camp for adults every summer in SF called Fidget Camp and we just opened applications for this year! If you are a creative nerd who loves to learn and teach and make art and be surrounded by others who love the same please apply by May 10th! fidgetcamp.com/

thank you all for bearing with us as we scale Flipbook! I know some of you had to wait literally 2 hours to get in yesterday 😭 huge huge thanks to @modal for stepping up to sponsor us 🙌 we now have way more compute available and wait times are down to just a few minutes!

unfortunately we don't have the GPUs to keep up with how much people want this rn! I have added a little waiting room. Please be patient if you end up in there and you'll be able to try it soon 😇

To get to the Flipbook visual style you see here, we had to go through literally hundreds of iterations. Here are some of my favorite in thread

Wow y'all. We are blown away by the response to Flipbook. Thank you. It means the world that so many of you want computing to feel this alive. That is all we ever wanted with getting this out there. To set expectations: Flipbook is a project from a tiny team; the site is a house of cards of APIs and open models tied together with duct tape and string. We did not expect it to blow up like this, and I've been glued to my laptop for 24 hours trying to keep it running! Thank you to everyone reaching out, sharing what you're making, and offering ideas, support, and collabs. Please keep it coming! we're reading all of it, and it's helping us figure out what's next for this.

oops - @modal 's magic serverless GPU infra!

All of this is live! it's early and slow. many of the demos above are sped up/edited, but we can't wait to see what you think. Try it yourself at flipbook.page (5/5)

Today, Flipbook is limited, so we designed it around visual explanations. As the models get more accurate and more stateful, the set of things worth doing this way will expand. Even ones you'd assume need structured UIs like coding: (4/5)

To bring the imagery to life, we heavily optimized @LTXStudio's video model. Enough to stream live 1080p video at 24fps directly to your screen, connecting directly via websockets to @modal_labs serverless GPU infra. (3/5)

Because there's no strict layout engine, illustrations reshape themselves to fit your window. And any region of the image can become interactive, not just the parts someone decided to make a button (2/5)