VLM Benchmarking

Before using the AI as a judge in the robot design process, we needed to make sure it was actually up to the task. We ran three validation experiments : Correlation, Guidance and Consistency. First, we compared the AI's scores to those of a human evaluator on the same robots, to check whether they agree — the closer the match (high correlation), the more the AI can be trusted to simulate human judgment. Second, we tested whether giving the AI explicit scoring guidelines actually helps it score more accurately. Third, we checked how stable the AI's scores are: if you show it the same robot twice, does it give the same score? This last point matters a lot — an inconsistent judge would make the whole evolution process unreliable. The results of these three experiments are shown below.

To measure the correlation between the VLM and a human, we asked one human evaluator to grade morphologies according to a target on the three criteria. We then asked the VLM to grade the same morphologies and compared the results. To quantify the comparison, we used the Pearson correlation coefficient to evaluate the linear correlation. The Pearson coefficient ranges from -1 (exact opposite linear correlation) to 1 (perfect linear match), where 0 means no linear correlation at all. As a reminder, the VLM output is 3 scores : coherence, originality and potential. The overall end score is the weighted average of those.

The correleation experiment was run multiple times and contains different morphologies and targets. It eventually achieved a Pearson correlation of around 0.5, which confirms a real potential for simulating a human evaluator for morphology. The gap to 1 may be explained by different factors : imperfect prompt, scoring bias or simply VLM scoring inability for complex concept.

Even if the correlation seems encouraging, we wanted to be sure that the VLM configuration we put with the score guidance in the prompt, had a positive influence on the scoring. To verify the actual impact of the score ranges, we tested the VLM evaluation with and without score range guidance to measure its effect.

Score range guidance example for coherence criteria :

coherence - How well does the morphology match the static target ({target_morph}) ?

0–29 = no recognizable similarity to a {target_morph}

30–49 = vague resemblance, very weak matching feature

50–69 = partial match, some features of {target_morph} are present

70–89 = some great resemblance, key features identifiable

90–100 = good match, structurally faithful to a {target_morph}

For "good" morphologies according to the human evaluator, the guidance actually pulls the scores toward the human baseline. On the other hand, for "bad" morphologies, the guidance goes the wrong way; both scoring methods tend to be too optimistic, which is related to the extrema avoidance tendency. The VLM is very sensitive to the prompt and the context given.

We should also ensure the consistency of the scores given, to avoid rare but potentially abnormal scores for individuals with average or poor performance. Indeed, the non-determinism properties of the models results in varying outputs for the same input. To verify consistency, we made identical requests multiple times and measured the standard deviation of the scores given. We tested this on morphology images and behavior videos, with and without target-fitting solutions.

The first experiment focuses on morphology image scoring. We asked the VLM 21 times to evaluate the same "car " morphology: once for an average-looking car and once for a random morphology. In both cases, the standard deviation remained around 10% of the mean, and 100% of this sample correctly discriminated the good morphology from the random one. This result highlights the precision of the model for image scoring, with consistent outputs.

Then we tried the same approach but for video scoring :

The consistency results are clear: for morphology evaluation through images, the VLM is precise and confident, with low variance in its scores. For dynamic behavior, it is another story. Indeed, the standard deviation is very high, reaching more than 50% of the mean, making the scores untrustworthy. 

One way to tackle the inconsistency issue and obtain coherent scores is to make multiple requests to average the outputs, avoiding score spikes as much as possible. This tradeoff is absolutely necessary; otherwise, the evolutionary process will not be able to select great solutions, lost among non-meaningful scores.

Different prompt variants have been used depending on the experiment configuration, but the overall structure and content were the same. The variant were on the presence of an image/video reference section, the presence of a batch context section, some modifications in the context and small tweaks in the scoring rules. The following example is an example for a morphology evaluation containing all section possible.

==== BATCH EVALUATION ====

    You will evaluate {len(robot_ids)} robot morphologies in one pass.

    Each image was labeled before being sent: {id_list}.

    Evaluate each one independently.

==== REFERENCE IMAGE ====

    The first image labeled "reference" shows the CURRENT BEST-PERFORMING robot from the previous generation.

    It is provided as a contextual baseline ONLY for comparison. 

    - Do NOT score the reference. Do NOT include "reference" as a key in your JSON output. 

    - The reference is likely to not be far from great, and may be improve a lot. Do not take it as a "great" reference, take it as a starting point for comparison and improvement.  

    Use the reference to better identify and reward: 

      • Genuine structural novelty: designs clearly different from the reference in limb count, body plan, attachment points, or overall stance. 

      • Real improvement: morphologies that address visible weaknesses of the reference (e.g. better ground contact, better simplicity / efficiency, more coherent body plan, more {self._prompt_config.target}-like structure). 

      • Interesting new traits the reference does not have. 

==== CONTEXT ====

    You are a strict and skeptical evaluator analyzing a static image of a MuJoCo robot morphology.

    Your job is to be PRECISE and reproduce human-like feedback on the robot's structural design.

    The scene: 

    - 2 simultaneous views of the same morphology: left = front/side angle, right = 3/4 perspective 

    - checkerboard floor and sky in background 

    - Robot has a white ovoidal torso and body part, and colored limbs (red, yellow, green, purple...) 

    - The robot's locomotion objective: {dynamic_target} (= dynamic target) 

    - The robot's morphology objective: looking like a {static_target} (= static target) 

==== ANALYSIS ====

    Step 1 --- Factual observation 

    Describe precisely what you see in both views: 

    - Overall shape, size and position relative to the ground (use the left view to see if the legs touch the ground) 

    - Number of limbs, their attachment points, articulations, segment lengths and approximate angles 

    - Overall stance: is the robot upright, crouching, sprawled, collapsed? 

    - Any asymmetry, specificity or unusual structural feature across the two views (shapes, connections, ..) 

    Step 2 --- Morphology interpretation 

    You are evaluating structural design.

    Based on the static pose and limb layout: 

    - Does the morphology resemble a {static_target}? Identify which features do or do not match. 

    - Does the structure suggest the dynamic target ({dynamic_target}) is even physically plausible? 

      Consider: center of mass, ground contact points, limb symmetry, limb articulation, joint range of motion (~90°). 

    - If the morphology shows originality or promising structural traits, state what they are

      and how they could support the static target and dynamic target. 

    - If the morphology is poorly designed, state specifically why

      (e.g. too few contact points, limbs too short to reach ground, torso too high). 

    Step 3 --- Score 

    Score each dimension using only the static image evidence.

    Be conservative. Do not infer runtime behavior from a single frame. 

    Avoid OVERGRADING, let space for better morphology further improvement (A medium morphology cannot have more than ~60/100 in overall). 

SCORING RULES 

    coherence  --- How well does the morphology match the static target ({static_target}) ? 

      0–9    = no recognizable similarity to a {static_target} 

      10–39  = vague resemblance, very weak matching feature 

      40–59  = partial match, some features of {static_target} are present 

      60–89  = great resemblance, all key features identifiable 

      90–100 = very good match, structurally faithful to a {static_target} (this category must be chosen only for real breakthrough) 

    originality  --- Is the structural design novel or inventive ? 

      0–9    = generic, indistinguishable from a randomly generated MuJoCo morphology 

      10–49  = basic organisation and minor variation on a standard body plan 

      40–59  = one interesting structural choice (unusual limb count, asymmetry, etc.) 

      60–89  = novel design with inventive features 

      90–100 = very creative, unexpected but clever combination of structures (this category must be chosen only for real breakthrough) 

    potential  --- Is the structural design have some potential for further evolution toward the target ? 

      0–9    = completely off-topic, very far from target, hopeless 

      10–39  = very poor design, hard to tweak toward the target 

      40–59  = plausible but inefficient, long evolution needed 

      60–89  = design that could works, could still have improvements 

      90–100 = very interesting design able to evolve toward the target (this category must be chosen only for real breakthrough) 

==== FEATURE DESCRIPTORS ==== 

    These structural features are used to CLASSIFY the morphology along

    diversity axes in the evolutionary archive (MAP-Elites).

    Each score places the robot in a distinct region of the design space,

    so rate them accurately and independently of the fitness scores above.

    A robot can score high on fitness but low on a descriptor (or vice-versa). 

    Rate each of the following 2 features (integer 0-10): 

    Scoring guide: 0 = not present / impossible to assess from this image,

                   5 = moderately present, medium amount

                   10 = strongly and unambiguously present / high amount

    overall looking: 

      How to classify the design of this tree-like robot ? Consider the body parts, the arms and its characteristics. 

      0 = no specific characteristics, generic tree 

      1-2 = a clumsy or broken tree 

      3-4 = a very tall tree of its type 

      5-6 = a very small tree of its type 

      7-8 = a very wide tree of its type 

      9-10 = a very narrow tree of its type 

    tree type: 

      How too characterized the type of the tree ?  

      0-1 = a bush or shrub 

      2-3 = a cold forest tree (pines, ...) 

      4-5 = a temperate forest tree (oaks, birch, ...) 

      6-7 = a tropical forest tree (palms, banana tree, bamboo, ...) 

      8-10 = other types of tree 

    Include these ratings as a top-level "descriptors" key in your output JSON. 

==== OUTPUT FORMAT ====

    Respond ONLY with valid JSON, no text before or after.

    The top-level keys must be exactly the robot IDs: {id_list}

    Each value follows this schema: 

    {

      "observation":    "factual description",

      "interpretation": "interpretation description and explanation",

      "coherence":      { "score": <int 0-100>, "reason": "..." },

      "originality":    { "score": <int 0-100>, "reason": "..." },

      "potential":      { "score": <int 0-100>, "reason": "..." }

   }