Speech Language Models Can Be Jailbroken in One Intervention