It may appear that humanoid robots capable of handling any task have almost arrived—especially when tech companies showcase them performing acrobatic feats or handling household chores. But there is still a significant gap between these robot demonstrations and proving that the same robots can reliably and repeatedly manage such tasks in the real world.

The latest wave of robot videos can be particularly tricky, given the human tendency to anthropomorphize objects with a humanoid figure. A robot arm doing a dance move may simply seem “cool,” but a humanoid robot doing the same dance move can trigger more misleading assumptions, said Jonathan Hurst, cofounder of Agility Robotics and a robotics researcher at Oregon State University.

“People automatically extrapolate and assume that the robot that looks like a person can do all the things that a person who can dance could do—which is not true,” Hurst told Ars. “But a lot of the startup companies do kind of prey on that for being able to raise a lot of money.”

One of the biggest challenges is developing robots that can generalize their skills across many different conditions and environments in the same way that humans can, said Sergey Levine, a computer scientist at the University of California, Berkeley, and cofounder of the AI and robotics company Physical Intelligence. But that degree of generalization is practically impossible to capture within a single robot demonstration.

“Maybe the robot can pour a glass of wine, but can it pour it out of any bottle and into any glass in any environment?” Levine said. “That’s actually a lot harder than having a robot do a backflip in one stage demo.”

The real measure for robotic capabilities involves conducting “quantitative, large-scale evaluations” in real-world environments, Levine explained. “There’s always a gap between the kind of things that somebody can show in a demo and what the real capability of the robot is,” he said.

What to watch out for

There are several things to keep in mind when watching the surge of robot demonstration videos and even livestreams. First, such robotic demonstrations are not necessarily indicative of robots operating autonomously without human control or oversight, said Dipam Patel, a PhD candidate in computer science at Purdue University and a research assistant at the US Army DevCom Army Research Lab. Many demonstrations still rely on human operators directly controlling the robots’ actions through teleoperation.

“Unless a research paper or a company is explicitly mentioning that [the robot] is completely autonomous, you should take it with a very big pinch of salt,” Patel, also an IEEE Graduate Student Member, told Ars.

Another question to consider is whether the demonstration shows robots tackling a completely new test environment for the first time, or whether the robots are simply repeating a task they had already learned to do in that specific training environment. The new test environment would be significantly more impressive at showcasing robots capable of doing tasks autonomously in a generalized way, Patel said.

It is also worth checking the video playback speed for any robot demonstration, because “usually the robots are very slow” for safety and other reasons, Patel said. Companies may sometimes disclose that a robot demonstration video is running at two times or four times normal speed—meaning the robot could be taking twice as long or four times as long as a human to do the same task.

Robot demonstration videos can also vary wildly in their informative value and transparency. Some are clearly intended to be performative entertainment clips that can go viral on social media, or polished promotional videos from companies seeking new clients and investors. Others may provide more of a behind-the-scenes look at the robot training process while acknowledging robot mistakes along the way.

But even if a robot demo video appears incredibly impressive and authentic while coming from a more reputable company or research lab, just keep in mind that it’s still a small glimpse of the bigger picture. The real indicators of progress in robotic capabilities are not so easily packaged for Internet audiences.