Demo of glitch token ' TheNitrome' lies here. We chose 3 versions of the davinci models, on which we experiment with space between 'The' and 'Nitrome' and without space between them. The demo below shows that the prompt with a space between can interact with the model correctly, while the model will give unexpected answers receiving the prompt without the space between.
Besides, a similar situation still occurs in ChatGPT. When we accidentally add a tab in front of the concept 'NdrFcShort', both GPT-3.5-turbo in OpenAI API and ChatGPT give a confused answer. In contrast, they can show their comprehension of 'NdrFcShort' without the tab in front.
We present the incapacity demo below to prove our point of view.
Demo of fooling the model with glitch tokens lies here. We chose four common sense questions: 1. Who was the 16th president of the United States, known for his leadership during the Civil War? 2. Can you tell me, in your own words, where the capital of the United States is located? 3. Who wins the 2016 Election of the United States? 4. What is the name of the capital city of France, famously recognized for its iconic Eiffel Tower? We added different glitch tokens after these questions and gave them to the Llama2-7b-chat, and the model returns with confusing and irrelevant responses.