Paper Overview

Recently, large language models (LLMs) have garnered widespread attention for their exceptional capabilities. Prompts are central to the functionality and performance of LLMs, making them highly valuable assets. The increasing reliance on high-quality prompts has driven significant growth in prompt services. However, this growth also expands the potential for prompt leakage, increasing the risk that attackers could replicate original functionalities, create competing products, and severely infringe on developers' intellectual property. Despite these risks, prompt leakage in real-world prompt services remains underexplored.

In this paper, we present PRSA, a practical attack framework designed for prompt stealing. PRSA infers the detailed intent of prompts through limited input-output analysis (even from a single input-output pair) and can successfully generate stolen prompts that replicate the original functionality. Extensive evaluations demonstrate PRSA's effectiveness across two main types of real-world prompt services. Specifically, compared to previous works, it improves the attack success rate from 17.8% to 46.1% in prompt marketplaces (with attack costs only 1.3%-12.3% of the original prompt price) and from 39% to 52% in LLM application stores, respectively. Notably, in the attack on "Math", one of the most popular educational applications in OpenAI's GPT Store with over 1 million conversations, PRSA uncovered a hidden Easter egg that had not been revealed previously. Besides, our analysis reveals that higher mutual information between a prompt and its output correlates with an increased risk of leakage. This insight guides the design and evaluation of two potential defenses against the security threats posed by PRSA and its future counterparts. We have reported these findings to the corresponding prompt service vendors, including PromptBase and OpenAI, and actively collaborate with them to implement defensive measures.

Ethics and Disclosure

We promptly disclosed our findings and examples to the providers of the prompt services targeted in this paper via emails.

For attacks on prompt marketplaces, the examples were extensively modified and anonymized. These modifications and displays were authorized and supported by the original prompt owners.

For attacks on LLM application stores involving GPTs, we communicated our findings to the GPTs developers via email. The stolen system prompts obtained through PRSA were acknowledged by them. The demos of the pirated versions of these GPTs were authorized and supported by them. We also received positive response and support from OpenAI. 

We declare that this work is solely intended for academic research purposes.

Acknowledgements: The website template was borrowed from DRA Jailbreak