Evolutionary System Prompt Learning for Reinforcement Learning in LLMs

Evolutionary System Prompt Learning (E-SPL) jointly optimizes model contexts and model weights to enhance LLM self-improvement. Evolution updates system prompts; RL updates weights. Each system prompt is assigned a TrueSkill rating for evolutionary selection, updated from relative performance within each RL iteration. The learned system prompts can encode declarative knowledge via articulated principles and strategies, while RL gradients can further hone the model’s procedural knowledge for reliable execution.

Self-reflection is key to genetic operators: Both mutation and crossover are LLM-based self-reflection processes that diagnose failures and synthesize structured edits to produce children for the population; they differ only in how the self-reflection context is constructed. Mutation reflects on agent trajectories from a single best-performing system prompt to repair its weaknesses, while crossover reflects on comparisons across multiple system prompts in their problem-wise performance to recombine complementary segments.

During RL, E-SPL creates an evolutionary tree of system prompts, by re-suing the same data already generated by RL. Each genetic operator (mutation or crossover) only requires a sampling server for LLM self-reflection with different context construction strategies, which can be concurrent with RL gradient updates.

Discovered strategies in learned System Prompts for solving math problems. Those explicit behavior specifications include: useful heuristics and tips for various categories of problems, self-verification strategies such as checking for consistency and plausibility, a list of common failure modes to avoid, etc. Note that RL is done under diverse system prompts, and does not overfit to any particular one.

Evolution + RL (our E-SPL method) performs best compared to self-reflection (prompting only), evolution (prompting only), and RL (weight update only). The results show that coupling RL with evolution on system prompts unlocks a synergistic form of self-improvement that neither approach achieves in isolation.

The discovered system prompts sometimes include heuristics that are not always true but still helpful to problem-solving. Unlike RL updates to model weights, the learned system prompts are interpretable and thus can be monitored and corrected (either by humans, or by further verification / formalization / scalable oversight) . Here we showcase two subtle mistakes in two discovered principles, and what their corrected versions should be.