If you wish to remove the driver and not just the device from Device Manager, select Delete the driver software for this device