building robots - one at a time. incoming assistant prof @nyuniversity

Joined September 2014
6 Photos and videos
Pinned Post
If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale! We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.
8
39
230
32,844
yisha reposted
RL training for LLMs involves exposure to problems in the “Goldilocks zone” of difficulty: not too hard, not too easy. But difficulty is not the only thing that matters. Problem type matters, too. And LLMs do not see or organize problem types the way humans do. This is the starting point of Manifold Bandits. The fundamental issue is this: when using RL to train LLMs, there will always be problems that are more productive for the policy at a given training iteration. Across a dataset, there are an astronomical number of possible batch orderings; uniform sampling is unlikely to consistently land on the most useful ones by chance. This is why many adaptive curriculum learning and difficulty-aware methods focus on the policy’s “optimal learning zone,” avoiding examples that are already trivial or still impossible for the LLM to solve. But problem type is usually treated differently. Task decompositions within a training set are often manually defined according to human semantics, or ignored entirely, with an entire dataset/domain treated as “one task,” despite the immense heterogeneity seen even in small training datasets. It is generally understood that LLMs may not share our concept of difficulty. But it seems less recognized that LLMs may not share our concept of problem type, either. In both cases, training the LLM according to our own semantics can obscure fine-grained learning dynamics that could otherwise be exploited. In this work, we seek to derive an adaptive curriculum learning method that caters not only to the policy’s ability, but also its perception: its latent organization of tasks. Driven by the manifold hypothesis and the Bitter Lesson, we leverage computation to derive curriculum structure from the policy’s latent geometry, then use Bayesian inference over that structure to guide search and learning over the training set. The result is Bayesian Manifold Curriculum (BMC): an algorithm that does not just try to find problems of the right difficulty, but instead orchestrates training effort across diverse and interacting problem types. More technically, we frame problem sampling for LLMs as a manifold-structured bandit problem with endogenous non-stationarity. The tl;dr is that the problems/arms are diverse, structured, and interact through policy updates, so the typical non-stationary bandit framing is not quite the right fit for training LLMs. BMC is derived from this new framing. 🌐Website: darrienmckenzie.com/manifold… 📰Paper: arxiv.org/abs/2606.19750 💻Code: github.com/DarrienMcKenzie/m… 🧵1/N
3
38
201
10,253
yisha reposted
Train on one tactile and deploy on another
A robot policy trained on one tactile sensor usually breaks the moment you swap the sensor. New hardware → recollect data → retrain. We built TACTX: train tactile policies on one sensor, deploy zero-shot on a physically different one. The catch, these sensors don't just look different, they measure contact through totally different physics: 👁️ Daimon — vision-based · 🧲 eFlesh — magnetic · ⚡ FlexiTac — resistive Same contact but completely different raw signals. 🧵
2
25
5,241
We show that the policy trained with one tactile sensor can be transferred to another sensor! Also, great thanks to @binghao_huang and @venkyp2000 for donating sensors :D
A robot policy trained on one tactile sensor usually breaks the moment you swap the sensor. New hardware → recollect data → retrain. We built TACTX: train tactile policies on one sensor, deploy zero-shot on a physically different one. The catch, these sensors don't just look different, they measure contact through totally different physics: 👁️ Daimon — vision-based · 🧲 eFlesh — magnetic · ⚡ FlexiTac — resistive Same contact but completely different raw signals. 🧵
2
22
3,248
This so fun to play with!
Super excited to share the last paper of my PhD: "Hallucination in World Models is Predictable and Preventable"✨ We train a 350M-param generative world model on a large dataset w/ 210 tasks and show that we can predict *when* hallucination happens and use that to fix it! 🧵1/n
1
1
719
If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale! We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.
8
39
230
32,844
We also find that by simply making the hardware optimizable with control policy, the fingertip tracking error can go down to less than 1mm - not possible for the commercial robot hands we tried of the same dof
3
421
This was quite a weird project when I started a year ago🦭 but the designs that came out of 3d printers are so fun to play with!
3
610
with my awesome co-authors @ncklashansen @baicrystal25 @carlo_sferrazza Mike Tolley @xiaolonw
3
741
yisha reposted
Human demos contain rich hand-object skills. The challenge is the embodiment gap. Human and robot hands differ in shape, joints, and contacts. Introducing ConTrack, our latest work that turns contact-rich human hand-object demos into robot motions.
6
31
164
25,649
My first time going to CVPR! Looking forward to sharing our work and meeting everyone 🦉
What senses does embodied AI need beyond vision? Join us at Sense of Space Workshop @CVPR — a full-day workshop on multi-sensory modeling for embodied intelligence. 📅 June 3rd 📍 Mile High 2C, Denver 🖼 Posters: Hall A #275–282 Featuring invited talks from leading researchers across robotics, vision, sensing, and embodied AI. @NimaFazeli7 @notmahi @mapo1 @wojmatusik Christian Theobalt @mangahomanga @Boyiliee @yswhynot @LingjieLiu1 @pliang279 @Ismini_L Website: sense-of-space.github.io/ #CVPR2026 #EmbodiedAI #Robotics #MultimodalAI #TactileSensing #SpatialIntelligence Powered by the amazing team! @RaoFu79761158 @LiGuankfd2 @alex_kai2020 Kun He @tomhodan Ergys Ristani Jessica Yin @AntheaYLi @yining_hong Devin Murphy Ray Song @xyz2maureen @ericyi0124 Qi Ye @YunzhuLiYZ @haoshu_fang @ruoshi_liu Vatsal Mehta @LuoYiyue @MengyuLearner
5
972
I’ll be at CVPR for one day tomorrow presenting some of our tactile projects in the Sense of Space workshop! Happy to catch up with old and new friends 🙌
Can we bridge the Sim-to-Real gap in complex manipulation without explicit system ID? 🤖 Presenting Contact-Aware Neural Dynamics — a diffusion-based framework that grounds simulation with real-world touch. Implicit Alignment: No tedious parameter tuning. Tactile-Driven: Captures non-smooth contact events. Consistent: Stable predictions in contact-rich tasks.
2
2
33
6,738
Here’s the workshop link: sense-of-space.github.io/
101
yisha reposted
Excited to share FreeForm☁️: Reduced-Order Deformable Simulation from Particle-Based Skinning Eigenmodes at #CVPR2026 FreeForm enables fast elastodynamic simulation for robotics and beyond, directly on messy data (no mesh needed)!
1
14
69
10,775
yisha reposted
VLA/VAs are doing well on short skills like pick-and-place. But real tasks rarely stop after one action, they require 1) many interdependent steps, 2) progress tracking, and 3) recovery from mistakes. In our paper LoHo-Manip, we address long-horizon manipulation with trace-conditioned VLA planning: a task manager tracks what’s done, plans what remains, and guides execution with visual traces.
1
44
237
26,054
yisha reposted
𝘏𝘰𝘶𝘴𝘦 𝘰𝘧 𝘋𝘦𝘹𝘵𝘳𝘢 - a generative robot hand and learned control co-design framework is out now! Code and modular robot hand build guide are on our website. We built four generated robot hands with learned control sim-to-real. End to end design and control < 24 hours
2
4
41
2,306
yisha reposted
I am attending ICLR 2026 in Brazil tomorrow! DM if you want to meetup. 📅ICLR Poster presentation in Rio at Pavilion 4 on Sat, Apr 25, 3:15–5:45 PM. Hope to see you there! 🤖Project Website w/ Github: an-axolotl.github.io/Houseof… 🦾Build Guide: an-axolotl.github.io/Houseof…
1
7
372
yisha reposted
Ever want to have a single policy to control diverse robots as well as different dexterous hands, or to observe the emergent behavior under cross embodiment training? Introducing our #CVPR2026 paper XL-VLA, Cross-Hand Latent Representation for Vision-Language-Action Models.
11
44
273
88,301
Why is one robot headless 😨
"Cross-embodiment" is a sign of generalization. We’ve seen huge progress in manipulation and navigation — but what about humanoid whole-body control? Can ONE policy control multiple different humanoids? Meet our #ICRA2026 work 🦅EAGLE: Embodiment-Aware Generalist Specialist Distillation for Unified Humanoid Whole-Body Control. Instead of brute-force URDF / morphology domain randomization, we iteratively distill specialists into one generalist. We also find that embodiment-aware representations matter for policy learning. 🔗 website: eagle-wbc.github.io/ 📜 arXiv: arxiv.org/abs/2602.02960
1
3
355
yisha reposted
Learning world models from touch instead of pixels. Led by @cwj99770123 and @yswhynot
Can we bridge the Sim-to-Real gap in complex manipulation without explicit system ID? 🤖 Presenting Contact-Aware Neural Dynamics — a diffusion-based framework that grounds simulation with real-world touch. Implicit Alignment: No tedious parameter tuning. Tactile-Driven: Captures non-smooth contact events. Consistent: Stable predictions in contact-rich tasks.
1
7
72
12,034