Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how.
This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code a couple APIs. The visuals are all AI-generated.
I'll be sharing the full workflow all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon!
Till then, this is the high-level idea:
1. Get a clean image of the food (or whatever your asset is)
- In my app, the user describes foods via text, or attaches images (or both)
- If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through
@fal
- If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design
- Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton
2. Gaussian splatting (2D image → 3D model)
- I tried various 2D-to-3D options on
@fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below)
- The app displays the 2D image while our backend generates the 3D splat
- We "groom" the splat to reduce size and load time by culling low-opacity/scale points
3. Render efficiently on device
Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR:
- SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable
- Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below)
- We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render
Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :)
Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs.
If you have any questions, drop em below!