Towards Disturbance-Free Visual Mobile Manipulation