Awesome, love to see it working on your setup so quickly! Glad the blog post helped.

Preparing to run evals without the sun sabotaging us! In earlier experiments the same policy performed differently depending on the lighting conditions. So we built a lightbox for consistent lighting.

I keep running into brilliant people in Zurich working on robot learning who've never met the person doing the same thing three tram stops away. So I've started a paper club to fix that: present your own work, flag a paper worth reading, and trade ideas with people beyond your lab or company. Last edition drew 60 sign-ups, and we had six presentations by Liam, Jessie, @mar_baga, @HKydlicek & @gui_penedo, @aurel_arnold and me. The next one is Thursday, July 9th, 7 PM. Got your own work to show? → Present it. Read a paper worth knowing? → Bring it. Just want to listen and ask questions? → Come anyway. Drinks, pasta & pesto are on us. Luma sign-up link below.

Robots 🤖struggle at new long-horizon tasks. One trick to improve performance is using granular subtask labels ("grab the plate" instead of just "do the dishes"). Today we're releasing our first blog 📜: using VLMs to mine subtasks at scale (19x cheaper than human annotators!)

Does collecting human intervention data (HG-DAgger) actually beat just collecting more demonstrations? I tested this myself: - interventions reached 72% success - more demos plateaued at 38% success I wrote up the experiments: aurelarnold.xyz/blog/human-i…

We spent last week in Zurich speaking with robotics companies. It was great to catch up with everyone and see the immense progress 🤖. It was also a useful reminder of how early we still are. Until recently, many teams were mostly trying different approaches to data and models and seeing what sticks. Now, more of them are starting to scale, and the bottlenecks are shifting from demos to data infrastructure. Some of the questions that kept coming up: - How should I store all my source data? Do I align sensor values upfront, or keep raw logs and convert later? - Now that I have a lot of data, how do I select what matters and how to sample it? Should I deduplicate? How do I run QA when I can no longer just vibe-check every episode? - How do I quickly get annotations like instructions, subtasks, and failure labels? Should I outsource them, or can I rely on models? These are the kinds of problems teams are starting to face as they move from experiments to real scale and it's something we are heavily focusing now. Many of primitives now exist in the refiner, but many are missing so there is a lot of work ahead of us. Finally, I was really impressed by the talent in Zurich: students working on wildly ambitious projects at @ethroboticsclub, labs like @mimicrobotics building at the frontier, and focused integrators like @DominiqueCAPaul turning robotics into something useful for real customers. It might not look as bright for Europe in LLMs, but after last week I’m increasingly hopeful that Europe will be at the forefront of robotics.

Most of the stack transfers to other manipulation setups. Write-up: aurelarnold.xyz/blog/vr-tele… Code: github.com/Dream-Machines-Ro…

To make teleoperation more convenient I pull down the headset to my chest and look at the robot directly. A piece of tape over the proximity sensor keeps the headset awake.

Two more features can be accessed through buttons on the controllers. One is a precision modifier i hold to lower the translation and rotation gains (1st video) and one that sends the arm back to its home pose (2nd video).

Mapping the controller’s absolute pose to the robot means I’d need to start every session in exactly the robot’s pose, or it jumps. The same problem we have with our passive GELLO leader arms. Instead I map relative motion while holding a button (1st video). This also lets me intervene during a policy rollout (2nd video).

Since the robot now tracks position at the wrist, I want to read my hand’s position at the wrist too, but the controller reports it at its center. Twisting my hand swings that point through an arc, which reads as a translation and moves the robot’s wrist. (left, blue sphere = read-out point) A short calibration (2nd video) finds the wrist pivot of my hand and shifts the read-out point, so a pure twist barely moves the robot’s wrist.

Making the control feel natural took a few tricks in the IK. With a standard approach, twisting your hand without moving your wrist swings the whole robot arm (left). I split the joints so joints 1-3 track position of a point at robot’s wrist and joints 4-6 track the gripper’s orientation (right).

I implemented VR teleoperation for our robot arm because I wanted to do human intervention data collection and our GELLO leader arms didn’t allow it. I wrote all my learnings up and open-sourced the code…

I find it quite hard to decide when to stop the policy and how to intervene for HG-DAgger style data collection. How would you do it? > A: right before failure, a small nudge > B: rewind and re-demonstrate the hard part > C: correct after failure > D: record the rewind and correct after failure > E: a mixture > or something different? Gonna train a couple of policies with these different intervention styles to compare performance and the learned behaviours…

Nothing beats the feeling when things weren’t working for a while and they do! Policy finally started doing what we want it to. It’s fun to get excited about these kinds of things (as you can hear in the video). Doing some evals and will share results soon.

there are people in zurich building some really weird but cool things @DominiqueCAPaul

Finally finished the blog post on my first week of imitation learning at Dream Machines, going into detail on how i trained my first ACT and Diffusion Policy baselines, what worked and what didn’t, and how I would approach it differently if I were to start over: aurelarnold.xyz/blog/first-w…

Collected 50 episodes of a fairly dexterous assembly task today and trained an ACT policy for 1.5h locally on an RTX 5090 (20k steps, chunk size of 32) to collect intervention data afterwards (HG-DAgger). As you can see from my reaction, the policy already did surprisingly well after the first training run. It struggled most on the subtasks I struggled with myself and probably didn’t provide very clean demonstrations. Ran out of time to train on the aggregated data before packing up. We’re already heading back home tomorrow. Testing the effect of human interventions on performance will be done back in Zurich.

Second day at the electronics factory working on robot learning. 1/ We started with a 3-hour tour of the new factory by the production manager. So many new tasks that I hadn’t seen on my last visit. Many are still out of reach for the current SOTA, but there are a few new ones, like kitting tasks (placing multiple objects in a bag or box), packaging, and individual assembly steps that are automatable. 2/ We invited workers to try teleoperating the arms. This was the first time showing factory workers the product. Would they worry about being automated? No — they actually liked it a lot. We just wanted a 5-minute trial, but the first worker kept operating for 30 minutes, and a group of nine workers started forming around him, making jokes and laughing. When he stopped teleoperating, the man said, “I see a big future for this” — let’s wait and see if he changes his opinion when he sees the first policy running tomorrow. 3/ @aurel_arnold and I then each picked a task and started collecting data to train a minimal policy on. Aurel chose an assembly task for lights; I am working on a kitting task. We had some connection issues throughout the day with the SSH sessions. Regular SSH via the local network keeps dropping, so we connect over Tailnet instead. Sometimes the robots freeze for half a second while teleoperating, and it’s hard to identify the routing issue. There hasn’t been a single revelation yet, but a lot of micro-learnings. Some examples: > Our grippers don’t have a lot of friction on the plastic bags, and they easily slide off. The TPU should compress together more. For tasks requiring precise movements, teleop is hard. Seeing more reasons to try UMI. > Adding a rubber mat to a workstation desk can be useful, e.g. for opening a plastic bag: you can push the bag across the mat with a gripper, and it creates a motion similar to a pinching movement that separates the two plastic sides, like a finger sliding motion would do. > A non-actuated mobile base would be very useful for moving the arms from desk to desk and also placing them in front of machines that don’t have a clamping affordance like a desk does.