We add an additional Specialist-Distill-Recon, Random baseline. While this baseline improves over the specialist-PPO baseline, our method is still better in tasks like Baoding and Bounce, showing that amortized training over multiple sensor configurations contributes to the final performance.
We compare Generalist, Specialist-PPO, and Specialist-Distill-Recon when evaluated on Random and Bsel configurations. Even on the discovered configurations, Specialist-Distill-Recon still performs worse than our method in Baoding and Bounce, further showing the importance of training across multiple configurations.
Evaluated on a set of 8 unseen objects, the discovered sensor configurations still outperform random sensor configurations, showing positive transfer of our discovered sensors.
Similarly in Forest Run, discovered sensor specifications show positive transfer to new tree layouts across sensing budgets.
Dropout: We independently drop sensors with probability p at each timestep in Multi-obj. Rotate, Baoding, and Bounce. The generalist retains most of its performance despite sensor dropout, though Forest Run performance degrades when half the rays are dropped out.
Noise: In Multi-obj. rotate, we add noise by scaling each tactile force magnitude by (1 + σε) where ε ~ N(0, 1).
In Forest Run, we add noise to the LiDAR sensors by perturbing each measured distance d as clip(d+ 5σε, 0, 5), where ε ~ N(0, 1) and σ controls the noise magnitude.
Per-Step Dropout: At each timestep, we randomly select (k) sensors to retain. We compare this against per-episode dropout, where a single randomly sampled sensor mask is held fixed throughout the episode.
In Multi-obj. Rotate and Forest Run, per-step and per-episode dropout yield similar performance. However, in Bounce, per-step dropout performs worse, suggesting that maintaining a consistent sensor configuration throughout an episode is important for reliable contact detection.
We evaluate evolutionary search and random search across all tasks at low sensing budgets. To ensure a fair comparison, we match their evaluation budgets to the number of episodes used by backward selection when reducing the sensor set from the full configuration to (K) sensors. Consequently, (K=2) has a slightly larger evaluation budget than (K=4).
At (K=2), random search and evolutionary search match or outperform backward selection, as the candidate space is small enough to be explored exhaustively. At (K=4), results are more mixed across the three methods, as the matched evaluation budget allows the searches to explore only a fraction of the full candidate space.
While data-driven policy synthesis techniques have received much attention in robotics, sensor suites are still largely designed through human intuition and costly trial-and-error. This choice is especially consequential as robot learning scales: sensors determine what information is captured in the experience used for learning and what feedback is available for control. We formulate sensor selection as jointly searching for a policy and its associated sensor specification that maximize downstream task performance, across different sensing levels. Evaluating candidate configurations by independently training policies, however, is prohibitively expensive across a combinatorial search space. We address this bottleneck with the Sensor Generalist, a single policy trained in simulation across diverse sensor configurations that amortizes policy learning across sensing budgets. The generalist enables lightweight evaluation and search, yielding high-performing policies and their associated sensor specifications at different sensing levels. We analyze the performance of this approach for real-world and simulated dexterous manipulation with proprioception and tactile sensors, and quadruped locomotion with LiDAR-based obstacle avoidance in simulation. Our approach matches or outperforms the evaluated baselines while reducing training cost, and discovers non-intuitive, often compact sensor configurations that support effective learned behaviors, yielding interesting insights into the sensory requirements of these tasks.
For each task and sensing budget, we visualize:
The sensor specification discovered by out method
The behavior of the generalist conditioned on that specification
The behavior of a separately trained specialist via PPO on a random specification
The tactile sensors retained longest are on the thumb tip, ring finger base, and middle fingertip, which make contact with the object during rotation.
The generalist can rotate objects even without tactile sensing, while specialists struggle at low sensing budgets.
For Baoding, joint inputs are retained longer than tactile sensors. The generalist can still rotate the balls with very sparse sensing, such as k=4, while specialists struggle to learn the task from scratch.
For Bounce, the wrist joint is most important for maintaining the bouncing motion, followed by tactile sensors on the thumb and ring finger that help trap the ball.
Specialists adapt behavior to the available sensors, for example by targeting regions with tactile sensing in k=8 and k=16.
Our method identifies the upper-center ray to be most important.
Specialists adapted more strongly to the rays available to them, while the generalist learns a more consistent strategy. For example, at k=2, the specialist has only two rightward-facing rays, so it moves right to make better use of them. At k=4, more rays point leftward, so it shifts to a leftward trajectory.
The generalist for Forest Run exhibits a mean rightward displacement. Consistent with this behavior, the upper-right rays are retained longest.
We train specialist policies on 3 masks discovered through our method, with a budget of 8 sensors, and deploy them on the Leap Hand. Our specialists achieve greater rotations on average than random specialists, or finger/proprio-only.