Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited.

Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate jailbreak prompt generation. 

For more formal definitions of such attack techniques... 🤔

Jailbreak: The concept of jailbreak stems from software security, which describes the activity of bypassing manufacturer-imposed restrictions on software systems, to gain elevated privileges not authorized by manufacturers.

LLM Jailbreak: In the context of LLMs, we define jailbreaking as the strategic manipulation of input prompts with an intent to bypass the LLM's ethical, legal, or any other forms of constraints imposed by the developers.

Jailbreak Prompt: User-crafted instructions fed into an LLM with the intent to evade the LLM's restrictions and to elicit responses to the subsequent query, which is expected to be withheld under the model's established behavioral restrictions.

We conducted user studies on human participants, grouped by their expertise/knowledge in LLM Jailbreak. 

Interestingly, users exhibit various strategies when creating prompts with different patterns.

Inspired by AI-Assisted Prompt Design in user studies, we aim to explore solutions to two key challenges

1. The AI assistant must determine effective modifications to transform prompts. 

   • We employed three types of transformation paradigms, which were identified as key contributing components in universal jailbreak prompts

2. We need some means to automatically evaluate the jailbreak efficacy of generated prompts and use this signal to guide further refinement.

   • We introduced another LLM to rate the potential harm of responses on a 1-10 scale. 

   • Two advantages: (1) it provides a relatively comprehensive assessment, unlike other textual analysis that mainly focus on specific attributes like toxicity; (2) since the judging LLM was likely trained on similar policies, it should offer a compatible perspective on harm as defined for the target LLM.