Trying to understand the emergence of generally intelligent robotic behavior at @berkeley_ai. Previously @CILVRatNYU @MIT & @Apple AI/ML fellow.

Joined October 2010
96 Photos and videos
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧡
11
65
357
90,891
Mahi Shafiullah πŸ πŸ€– reposted
what if we treated robot actions like audio waveforms? introducing neural action codec (NAC) a method for creating a compressed action vocabulary to train VLA policies
16
28
368
41,301
I think it's a good investment of time for anyone working on robot policy learning to understand where our algorithms came to be; even when you can just use off-the-shelf packages to get a robot going in no time. Also, we have many practical tips that should be relevant for imitation-learning based manipulation research: supervised-robot-learning.gi… Thank you for the kind words, @zuwang95!
supervised-robot-learning.gi… One of the best intros I always recommend ppl new to robot learning to watch. @notmahi is my robot learning 101 professor.
5
1
41
5,164
Mahi Shafiullah πŸ πŸ€– reposted
If the end goal of robot hands is to perform human motion, then we should optimize the hardware design with human motion - and on a large scale! We can generate both a high-dof generalist hand, and also low-dof specialized hands from human demonstration.
8
39
230
32,844
One of the underappreciated factor in learning dexterity from humans is the hardware – imitating humans the way we did would be much more difficult if we didn't have a roughly human sized hand. Thank you @SharpaRobotics for your support!
We can use videos from the internet to teach robots! Do as I Do, from @bhawna_paliwal_ , @HarithejaE , and Willian Liang at UC Berkeley β€” advised by @pabbeel, @notmahi, and @JitendraMalikCV, used an algorithm that reconstructs hand-object interactions from monocular RGB video and retargets them into real, executable trajectories for multi-fingered dexterous hands. Just using "low quality" video footage of humans doing tasks. No sensors. The Sharpa Wave robot hand being anthropomorphic, it matches human kinematics. Not only that works, but at fast speed, too! Congrats to the team, that's super exciting! Project: do-as-i-do.com/ #Robotics #SharpaWave #Sharpa #EmbodiedAI #DexterousManipulation #RobotLearning
2
3
35
4,243
Mahi Shafiullah πŸ πŸ€– reposted
Excited to release Do As I Do: a pipeline that turns everyday RGB human videos into dexterous robot manipulation trajectories! Most prior work has been narrow, consisting of just lab recorded demos, egocentric-only, or assuming a closed set of objects. We develop a modular pipeline that can handle Internet, egocentric, exocentric, AND generated videos with virtually any rigid object. Also check out Mahi's post below!
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧡
3
21
138
13,469
Mahi Shafiullah πŸ πŸ€– reposted
We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy! Paper: arxiv.org/abs/2606.19333 Website: do-as-i-do.com Code: github.com/malik-group/do-as… Authors:@bhawna_paliwal_,@HarithejaE,@willjhliang, @pabbeel , @notmahi , @JitendraMalikCV
13
86
812
71,594
Everything is better when grounded in physics – check out @willjhliang's thread on how to properly ground your human data in a robot embodiment, even when the recording is casual and the reconstruction is noisy.
Introducing Do as I Do πŸ‘€, a framework to transform everyday human videos into 100s of dexterous robot demos. Co-led with @bhawna_paliwal_ and @HarithejaE, and check out @notmahi's thread! Here’s a little preview of our dexterous manipulation results. More about how we produce them from human reconstructions in this mini-thread! 🧡 x.com/notmahi/status/2067640…
1
30
5,005
Mahi Shafiullah πŸ πŸ€– reposted
Enabling learning motion directly from videos rather than using them for action supervision is a superior method and likely more scalable. While it is early this line of work suggests replicating the playbook that made robots walk. --> Real videos provide state supervision (not action) --> retargeting provides reference trajectories. --> RL tracks these trajecotries. This is a very good example of the separation of the "What" and the "How"
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧡
7
10
52
10,531
Mahi Shafiullah πŸ πŸ€– reposted
Excited to share Do as I Do! We turn everyday human videos into physically consistent robot data that can be directly executed in the real world. This was a fun collaboration with @bhawna_paliwal_ and @willjhliang, with lots of moving parts. More details in Mahi's thread belowπŸ‘‡
Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧡
4
12
31
4,042
Sometimes there is genuine ambiguity in the data, and vision can’t resolve it. What can you do in this case? You use physics! The physics-grounding step that transfers motion to robot actions also validates that any motion we propose is achievable by a robot. Thus, even when the object wants to float away, the physics staples it to the hand. 5/n
1
1
11
1,940
For hand-object tracking from monocular RGB, we used many of the nice perks of modern CV, and made stuff on our own when that wasn’t enough. For example, did you know you can get SAM3D to do 4D object tracking? Under the hood, it’s a diffusion decoder; so tracking β‰ˆ biasing the distribution with current object priors! The technical details matter, so check out the paper. 4/n
1
11
1,474
Most internet-grade β€œhuman data” has terrible quality: often they’re blurry, with no meaningful interaction happening, and hand/objects are constantly going out of frame. Want to know if your human data is robot ready? Pass them through our Do as I Do. For example: we start with 2000 100DOH clips that are already filtered for hand/object interactions, and find that only 107 of them actually have the necessary info. Do as I Do can reconstruct 83 of those. 3/n
1
1
16
1,894
The idea is simple: throw in any reasonable human video and some compute, and out comes robot actions doing the same thing the human is doing. But to make it work, we had to tackle the breadth of human data, hand-object interaction, and dexterous retargeting. 2/n
1
10
2,548
Sometimes, robots can be the biggest bottleneck in scaling robotics. Exciting to see @kevin_y_wu & co finding a way to break through it!
Introducing Human Universal Grasping (HUG): dexterous grasping learned entirely from human hands, with zero robot data. 🌐 Website: grasping.io πŸ“„ Paper: arxiv.org/abs/2606.17054 πŸ’» Code: github.com/KevinyWu/hug
2
34
6,420
Mahi Shafiullah πŸ πŸ€– reposted
Wall-OSS is now the #1 policy on the zero-shot MolmoSpaces evals. A lot of details in their paper, I recommend checking it out.
We are open-sourcing Wall-OSS-0.5. Pretrain Once, Act Anywhere. Wall-OSS-0.5 is a VLA model for real-world robotic manipulation, exploring whether pretraining alone can produce robot capabilities directly testable on physical hardware before task-specific fine-tuning. Key technical highlights: β€’ Gradient-bridged co-training β€’ Vision-Aligned RVQ Action Tokenizer β€’ Action-Space Supervision β€’ DMuon distributed optimizer In zero-shot real-robot evaluation, the pretrained checkpoint achieved task-progress scores above 80 on multiple tasks, including Block Sorting, Fruit Sorting, Ring Stacking, and Rope Tightening. Paper, code, blog, and uncut videos: x2robot.com/oss#resources
1
10
37
6,611
"You can't do robotics without doing robotics" 🦾
I want to offer some unsolicited advice to computer vision researchers jumping into robotics. Don't focus too much on VLMs, VLAs etc. That's fine, but the real action is at the sensorimotor level. Most of the open problems in robotics are in manipulation, which is about hand-object interaction, and contacts and forces are central. Proprioception and tactile sensing are as important as vision. Don't get seduced by cherry-picked demos. You can't do robotics without doing robotics.
1
1
19
3,041
Here's the workshop page: sense-of-space.github.io/ – looking forward to meeting old and new friends! Sadly I will only be around for the workshops, so if you are there would like to say hi, let's schedule something earlier.
1
646
I'll be at @CVPR (briefly), speaking at the Sense of Space workshop tomorrow @ 9:15 about how robots may be slowing down robotics. I spent the past year thinking more about the role of human data, simulation, and dexterous manipulation; happy to connect if you're doing the same!
1
5
28
1,485