Self-Preservation
in Language Models
Dongkyu Cho, Rumi Chunara, Ali Behrouz
Embedded Files
Summary
A new task gives a clear learning objective, whereas the model’s prior capabilities are harder to enumerate, and its original training data are often unavailable. A few retained examples or named tasks therefore provide only partial evidence of what should survive an update.
Yet forgetting can extend beyond the supplied targets. Generating more examples adds “Depth” within those targets. Broader protection also depends on which behaviors the replay represents, what we can think of as "Width"
Self-Preservation
Can a model use its own knowledge to protect capabilities beyond the supplied preservation targets?
Learning a new task can change behavior far beyond that task. We evaluate both new-task learning and protection on held-out tasks excluded from replay construction and target selection.
Proposed Method: Task Banks
A Task Bank lists behaviors we want the model to retain. Each entry contains a task description and optional examples. We select entries and ask a frozen pre-update model to generate replay data. The learner then trains on new examples mixed with this replay.
Combining entries increases “Width”, the range of targets represented in replay. Held-out tasks test whether protection extends beyond the bank.
Takeaways
Generated replay protects capabilities beyond its supplied targets.
Generated replay protects capabilities beyond its supplied targets.
A frozen model generates replay for predefined targets. This reduces held-out forgetting while retaining positive new-task gains.
Average accuracy can hide forgetting.
Average accuracy can hide forgetting.
Correct answers are lost on 3.56% of examples and gained on 3.61%. The resulting 0.05-point net gain conceals substantial answer turnover.
Combining Task Banks strengthens broader preservation.
Combining Task Banks strengthens broader preservation.
Combining two partial banks improves mean Retention within the inventory and on held-out tasks at all three tested sizes.
Incoming task information can improve replay selection.
Incoming task information can improve replay selection.
At 4B, incoming-aware selection improves mean held-out Retention over random selection by 2.75 pp, with similar new-task gains. Smaller models show no corresponding advantage.
Preservation can come at a cost to new-task learning.
Preservation can come at a cost to new-task learning.
Under a fixed example budget, replacing incoming examples with replay reduces the worst held-out loss but also reduces new-task gains.
Page updated
Google Sites
Report abuse