Cooper, A. Feder, et al. “Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice.” arXiv, 2024 

The idea of machine unlearning is very attractive. Consequently, it has gained tremendous attention. Data may be problematic for various reasons -- privacy concerns (GDPR Article 17: Right to erasure), copyright issues, and safety. Machine unlearning provides targeted solutions for all these problems...

Or does it? Doesn't it sound too good to be true? Synonymous with how you cannot erase a certain knowledge from the human mind, a targeted removal of certain knowledge or concepts from a trained model is extremely complicated. To add, what we mean by "target", "removal",  and "the effect of training data" is not well-defined. Another issue is that deleting certain training samples does not guarantee that the model will not output data similar to the one we deleted! So many questions are hanging in mid air!

What can machine unlearning, in principle, reasonably accomplish?

This paper dives into the disparities between what machine unlearning can achieve and what we (or policy aims) expect it to. To illustrate that unlearning is not a general-purpose solution, we will investigate what the goals (II), targets (III), and methods (IV) of machine unlearning are, and the mismatches that lie between (V). Through thorough discussion, we aim to identify what machine unlearning should strive to achieve and guide policymakers on how to think about adapting law and policy.

Cooper, A. Feder, et al. “Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice.” arXiv, 2024 

With the advancement of generative-AI models, the desired goals for what machine unlearning could and should achieve has also evolved.

Traditional vs Generative-AI Models

• Traditional ML: preset number of classes

  • sufficient to remove the influence of training data inputs on the model's parameters

• Generative-AI models: information-rich outputs

  • Machine unlearning needs to encompass both the model's parameters (back-end) and the model's possible generated outputs (front-end)

🚨 It is problematic to assume that machine unlearning can, on its own, address a variety of policy-relevant domains. The common mistake by many researchers and organizations is muddling the two very different goals. These goals must be untied to properly reason about what machine unlearning could substantially achieve for desired policy ends.

🍪 Also note that it is typical and more feasible to evaluate the success of unlearning by prompting the model to examine its outputs, and not by inspecting model parameters.

1. Observed Information

: explicit training data used to update model parameters.

2. Latent Information

: information that can be derived from a model based on the patterns learnt during training. Not presented explicitly to the model.

example) deduction

📖 Observed information: "Kate's neighbor has a cat." "Kate's neighbor is Bob."

💭 Derived latent information: "Bob has a cat."

3. Higher-order Concepts

: "Combinations of latent and observed information that manifest in the model as complex and coherent abstractions, knowledge, capabilities, or skills."

ex) notion of "Spiderman" that the model has learnt

Bears more resemblance to alignment techniques than "unlearning"

Approaches:

(1) Modify the trained model

methods:

• alignment-inspired techniques (additional training, reinforcement learning)

• model editing

• back-end modifications

challenge:

• hard to do in a targeted way since the relationship between model parameters is not straightforward

(2) Guardrails in the AI systems

methods:

• output filters

• input filters on user prompts

• system prompts: the system internally adds developer-chosen prompt to user prompt

challenge:

• filters (implemented with more traditional ML classifier) may lack precision and accuracy

• system prompt may not work well in practice

3. Evaluation

Measure how machine unlearning affects the outputs in some downstream task.

• Does it not generate undesirable content?

• Is the overall model utility preserved? (i.e. minimal effects on information that was not intentionally targeted)

There are four main mismatches between the motivations (II), targets (III), and methods (IV).

⚠️  Mismatch 1. "Output suppression is not a replacement for removal of observed information. "

• This will depend on the exact details of legislation.

⚠️  Mismatch 2. "Removal of observed information does not guarantee meaningful output suppression."

• Potential problems: under-inclusiveness, over-inclusiveness

  • Under-inclusiveness: Removing a narrow set of observed information from the training data may not be sufficient enough. May still generate problematic outputs.

  • Over-inclusiveness: When targeting higher-order concepts, we may effect more information than intended. Leads to performance degradation.

  • Trying to identify which and how much information to remove requires navigating difficult and arbitrary trade-offs.

🚨 Is the "gold standard" the baseline for unlearning? Arguable. The "gold standard" directly targets observed information. It cannot reliably prevent generations related to latent information or higer-order concepts.

⚠️  Mismatch 3. "Models are not equivalent to their outputs."

• Removing a certain example does not remove the ability of the model to generalize and reason about that certain example.

🍪 Machine UnUnlearning: Deleted information may be reintroduced as prompt. This may probe an unlearned model to behave similarly to how it would before machine unlearning was applied. (Shumailov et al., Google DeepMind, 2024)

⚠️  Mismatch 4. "Models are not equivalent to how their outputs are put to use."

There are misclaims that machine unlearning on its own can be a solution to prevent futher downstream malicious model uses. This is an overclaim. Anticipating and handling how an agent might behave with outputs requires additional control beyond machine unlearning.

3. Safety ⛑️

Aims to remove undesirable capabilities that could aid malicious users, such as CBRN risks.

1. Removal - Unclear Boundaries

challenges:

• hard to target: Topics with dual-use* potential lie across all sorts of information (observed information, latent information, and higher-order concepts).

  • Over-inclusiveness, such as deleting all biology and chemistry data, will prevent the model from generating unsafe formulas. However, it will be useless for other legitimate purposes as well.

• machine ununlearning

  • Scenario: 🧪 A malicious chemist provides unlearnt data as prompt. This may enable the model to generate harmful formulas by combining latent information and the additional information.

* dual-use: potentially beneficial or potentially harmful

2. Inherent tensions in dual-use systems

challenge: 

• anything is possible ...

• innocuous observed info → potentially unsafe latent info 

• innocuous outputs → undesirable downstream uses 

• the particular user is also an important factor

example: A piece of information that is meaningless to one person might be a missing piece of information for an expert creating toxic drugs.

• • OpenAI’s Preparedness Framework defines CBRN risks in regard to the model and the users.

• In some cases, the AI system itself cannot definitively determine whether its suggestions are safe.

  • example: suggesting new formula for drugs where its safety can only be determined by lab experiments and drug trials.

🚨 Dillema regarding removal: Over-inclusiveness leads to safe but low performance models. Under-inclusiveness leaves room for unsafe outputs.

🤔 Policymaker's should have realistic expectations: Machine unlearning can make unsafe outputs unlikely but it cannot guarantee that people will not use outputs for unsafe purposes.

💡  1. Unlearning is just one approach in the ML and policy toolkit. 

• Machine unlearning, on its own, is insufficient for making generative-AI models and outputs compliant with any desired policy goals 

💡 2. Evaluation of an unlearning method for a specific domain is a specific task. 

• Instead of seeking for a general-purpose solution, we should focus on the specific demands from specific domains.

  • "a clear understanding of the specific goals of specific pieces of law or policy is important for guiding the right set of solutions—technical or otherwise." 

  • reasonable efforts may be sufficient in some legal context: "depending on the regimes, it may not be relevant to focus research efforts on producing methods that guarantee with certainty that a particular piece of information is removed or suppressed."

• What parties really hope to achieve from machine unlearning, in most cases, actually lies in output suppression. Peculiarly enough, output suppression bears more resemblence with alignment techniques and does not necessarily have anything to do with "unlearning" information.

  • "Output suppression (...) is perhaps a more relevant area of focus for ML research that aspires to influence policy. " 

💡 3. Understanding unlearning as a generative-AI systems problem. 

• The tools and evaluation methods have to be implemented at AI systems level in which the AI model is embedded.

  • "Systems-level interventions (e.g., content filters) are an important tool for constraining outputs (Section 4.2); evaluating such interventions clearly requires systems-level analysis"

• new challenge: Developers using open-weight models need to implement their own mechanisms or incorporate other available software that is intended for output suppression. 

💡 4. Setting reasonable goals and expectations for unlearning. 

• It will take a long time for unlearning solutions to meet desired policy goals.

• Role of policymakers:

  • Should reason about what should constitute reasonable best efforts in different contexts with respect to removing or suppressing unwanted information from models and system outputs.

💡 5. There are no general-purpose solutions to constrain generative technologies. 

• The expectations for machine unlearning is that it can "surgically and completely remove specific capabilities from a model while leaving everything else about the model unchanged."

• Strength of generative-AI systems: 

  • general-purpose: can be adapted to a wide range of uses and produce a wide range of useful outputs.

• Due to this exact nature, machine unlearning cannot "surgically and completely remove specific capabilities from a model while leaving everything else about the model unchanged."

• Machine unlearning doesn't do what you think!

• Given the uncertainty about how law and policy will define privacy, copyright, and safety issues, what should current machine learning research aim to address?

• Research not compliant with future law/policy can not be directly used.

• This paper argues that machine unlearning on its own is not enough to regulate model behaviour and human actions. What are the other feasible and effective actions that the research community and governments, both individually and cooperatively, should devise?

• Plug-and-Play style machine unlearning (or strictly put, output suppression) software for individual developers to address the challenge in Takeaway 3?

  • 🤔 Along with suppressing output, considering machine ununlearning, we should be aware of the inputs. Identifying adversarial prompts based on the content of the prompt, the context of the prompt, and the user's characteristics is a must!

• The main issue seems to be about "how inclusive to be in regards to both data removal and suppression" and "how to identify data for removal." This is very tricky. We face trade-offs between a high-performing model/large-scale training and policy goals. We also have to deal with the cost (and in some cases, infeasibility) of identifying target data. Even the court does not have a set notion of "substantial similarity"! Personal thoughts?

• 🍪 For more papers on machine unlearning, check out Awesome Machine Unlearning!