For a robot to learn a movement, it needs to know where the hand went at every moment, and along what trajectory. But a YouTube video does not contain that trajectory, the robot executable action label. Video shows you the outcome.
The information of "what action was performed" is written nowhere in it. The research community is explicit about this: human videos are not action data for robots, and they lack aligned, robot executable action labels and proprioceptive context.
So researchers run hand pose estimators, bolt on depth models, and mix in separate robot data to try to recover that trajectory after the fact. They are filling in missing information with guesswork, and the noise from that process piles up.