Lossless Adaptation of Pretrained Vision Models for Robotic Manipulation