Using VLMs to simulate subjective human perception and evolve interesting robots designs

Building robots that look like animals, feel ergonomic, or move expressively is harder than it sounds. The usual approach requires engineers to manually translate a vague creative vision into precise mathematical equations — which is tedious, and sometimes just impossible when the goal is subjective.

So we asked: what if an AI could do that job instead?

We explored using a Vision-Language Model (VLM) — the kind of AI that can look at an image and understand what it sees — as an automatic judge in a robot design process. Instead of writing reward functions by hand, we simply describe what we want in plain English, like "a kangaroo jumping high" or "an ergonomic lamp", and let the AI score how well each robot matches that description in the objective to find "interesting" solutions. To model the "interestingness" concept, we broke down the score into three criteria : coherence with the target, orginality and potential for further improvement. The final overall score is the weighted average of these three score.

The robots are evolved over many generations: the best ones survive, get mutated, and their offspring are scored again. We tested two strategies for managing this evolution. The first keeps only the top performers. The second, called MAP-Elites, also tries to maintain a wide variety of different solutions — and it consistently produced better and more diverse results.

We ran experiments in two settings. First, we looked at robot shapes: can the AI guide evolution toward a robot that visually resembles a target? The answer is yes — the AI's scores correlated reasonably well with human judgment (Pearson correlation of 0.5). Then we looked at robot movement: can the AI evaluate videos of robots and guide them toward a target behavior? Here the results were more mixed — the AI could grasp the general idea of a movement but missed many details and sometimes described things that simply weren't happening in the video.

Overall, this project shows that AI vision models can act as flexible, language-driven judges for robot design — no equations needed. But there is still work to do, especially for evaluating movement in video, where the AI struggles with spatial reasoning and consistency.

The VLM used for the study is Gemini 3 Flash, released in December 2025.