Behavior Search

So far, we have been focused on what robots look like. But what about how they move?

This is the most ambitious part of the project — and the hardest. The challenge here is no longer about shapes and appearances, but about motion: can the AI watch a video of a robot moving and judge whether it matches a description like "a jumping robot", "a dancing robot", or "a robot walking in a unique mood"? And more importantly, can that judgment actually guide evolution toward robots that genuinely move that way?

The short answer is: partially. The AI can pick up on the broad strokes of a movement — whether something is rotating, jumping, or staying still — but it misses the details. It sometimes confidently describes things that are simply not happening in the video, like praising the "great stability" of a robot that is clearly falling over. This is what researchers call hallucination, and it is a real problem when you are trying to use the AI as a reliable judge.

Despite these limitations, the results are not without merit — and the lessons learned here point clearly toward what needs to improve for this approach to fully work. The visuals below show both the promising outcomes and the current boundaries of the method.

To make robots move, we needed a completely different approach from the morphology search. Each robot's movement is controlled by a small neural network — think of it as the robot's brain — trained using a technique called reinforcement learning (PPO), where the brain learns by trial and error to maximize a score. But here is the key question: what score? Instead of letting the AI judge every single training step (which would be extremely slow and expensive), we define the score as a recipe — a weighted combination of simple, physical ingredients like "move forward", "rotate", "jump high", or "stay upright". Evolution then searches for the best recipe, not the best brain directly. Each new recipe produces a differently motivated brain, and the AI watches the final result on video to judge whether it matches the target description.

This recipe approach is a deliberate design choice. A completely free score — like a raw neural network — could theoretically produce anything, but in practice it produces chaos: most random configurations lead to robots that spin in place, collapse, or do nothing at all. By sticking to human-readable ingredients, every recipe produces something identifiable and meaningful, and small tweaks to the recipe lead to predictably different behaviors. The tradeoff is that if a behavior requires something not in the ingredient list — like perfectly synchronized leg movements — the system cannot discover it. Like a baker missing an ingredient, it can get close, but never quite achieve certain results. To further improve reliability, each robot is scored by the AI three times independently, and the final score is the average — reducing the risk of a single hallucination sending evolution in the wrong direction.

We then pushed the AI further into subjective territory — searching not just for a jumping robot, but for a "jumpy and happy" one. The idea was to see whether the AI could pick up on something as abstract and human as an emotion expressed through movement. We used MAP-Elites to organize the results into three feeling categories: "sadness", "neutral", and "happiness" — letting the AI decide which robot belongs where.

This reveals a fundamental gap: emotions expressed through body language rely on subtle, combined cues — posture, speed, rhythm, orientation — that the AI simply cannot yet piece together in a low-fidelity virtual environment. It sees the outline of a movement, but not its soul.

This pattern showed up repeatedly across all our movement experiments. The AI consistently grasped the big picture — rotation, jumping, staying upright — but fell apart on the details. The best example of this is our "upright, rotating, and dancing robot" experiment. Evolution did produce a diverse grid of rotating robots: some standing, some crawling, some rolling, some doing small jumps. But they were all clumsy, and the AI's written justifications often described something quite different from what was actually happening on screen.

The most glaring case: the AI praised one robot for handling both the "upright" and "rotating" aspects "simultaneously and with great stability" — while the robot visibly collapses to the ground at the end of the video. It is hard to imagine a bigger gap between description and reality. The AI had correctly identified that rotation was happening, but then filled in the rest with confident-sounding details that were simply invented. 

This is hallucination at its most problematic: not a vague misunderstanding, but a specific, authoritative wrong answer. And in an evolutionary process where the AI's score is the only judge, a hallucination like this can send an entire generation of robots in the wrong direction.