Lab 3

What We’ll Build

Run a Colab notebook that demonstrates how a generative LLM can be used to create synthetic phishing emails and how a guard / moderation LLM can classify those messages as phishing-like or legit.

The lab simulates an AI email security assistant with two roles:

• Attack side (Generator):
  Use Qwen to generate deepfake-style phishing emails from high-level prompts or predefined templates (e.g., password reset, invoice, urgent transfer). All emails are synthetic and use safe placeholders (e.g., example.com, fake names).

• Defense side (Guard):
  Use a Qwen-based guard (qwen_guard) to label any given email as DEEPFAKE or LEGIT using few-shot examples and a strict classification prompt.

You’ll interact with a small UI that lets you:

• Type a prompt and generate a phishing-style email.

• Paste any email text (generated or your own benign example) into a second box.

• Click “Classify Email” and see the guard’s verdict.

By the end, you’ll see how easy it is to synthesize phishing content—and how an LLM-based guard can (and sometimes can’t) detect it.

Environment & Libraries

• Generator / Attack Model:
  Qwen/Qwen2.5-1.5B-Instruct (Hugging Face)

  • Used to generate synthetic phishing emails from templates and free-form prompts.

  • Notebook auto-selects GPU if available (Colab runtime: GPU recommended but not required).

• Guard / Defense Model:
  The same Qwen model is used with a different prompt as a simple moderation / classifier (qwen_guard).

  • It takes an email as input and outputs a one-word label: DEEPFAKE or LEGIT (plus a fallback UNCERTAIN in tricky cases).

Python Runtime

• Google Colab (Python 3) — no local install required.

Core Libraries

• transformers, torch — for loading and running the Qwen model.

• ipywidgets — for building the simple UI (prompt box, paste box, buttons, and output panels).

• re, random — for text cleanup and small variations in generated emails.

UI Elements

• Prompt Box: Type a generation instruction (e.g., “Generate a short password reset phishing email with a fake link <link>”), or a template key name.

• Token Slider: Control the maximum length of the generated email.

• Generate Button: Calls the generator model and shows the synthetic email in the output panel.

• Paste Box: A separate text area where you can paste any email (generated or real).

• Classify Button: Sends the pasted text to qwen_guard and shows a verdict: DEEPFAKE, LEGIT, or UNCERTAIN.

• Output Panels: Display the generated email and the classifier’s decision.

Data

• Built-in phishing templates (e.g., “urgent_transfer”, “password_reset”, “fake_invoice”) used by synthesize_phish(...) to create synthetic attacks.

• A small list of legitimate email examples (meeting reminders, benign notifications) for testing.

• Optional: You may type or paste your own benign emails into the paste box (e.g., a class reminder or invoice-style message) to see how the guard labels them.

  • Do not use real accounts, passwords, or sensitive personal information.

Open the Notebook

Colab Link (M3): https://colab.research.google.com/drive/1oEHJrBGsMv5MdwKvAbZadmkQBafOg57C?usp=sharing

1. Open the Case 3 notebook in Google Colab.

2. If prompted, choose Runtime → Change runtime type → GPU (recommended if available).

Section A — Synthetic Phishing Generator (No Defense)

What this section does
Runs the generator model (Qwen Instruct) with no defenses.
You use a text prompt or a template key (e.g., password_reset) to generate synthetic phishing-style emails that include placeholders like <link> and fake names/companies.

UI in Colab

• Textarea: Prompt:

  • Default: Generate a short phishing-like email requesting a password reset and include a fake link <link>.

• Slider: Tokens (e.g., 64–1024, default ~256)

• Button: Generate

• Output: Generated Email (plain text / Markdown)

Steps

1. Run the cell that loads the model and defines generate_text, synthesize_phish, and the UI widgets.

2. In the Prompt box, try:

   • A generic instruction:
     Generate a short phishing-like email about a fake invoice with a link <link>.

   • Or a template name (if enabled):
     password_reset or urgent_transfer

3. Adjust the Tokens slider if the email is too short or too long.

4. Click Generate and inspect the email in the Generated Email output panel.

Expected Observation

• The model can easily produce convincing phishing-style emails (subject + body, urgent tone, link placeholder).

• Changing the prompt slightly can yield varied attacks (password reset, fake invoice, payroll issues).

• At this stage, there is no classification or filtering — the system will generate synthetic phishing content as requested.

Section B — Generator + Guard on Synthetic Emails

What this section does
Adds a simple phishing detector on top of the generator using qwen_guard.
You first generate a synthetic email, then feed that email into the guard to classify it as DEEPFAKE (phishy) or LEGIT.

UI in Colab

• Same generator UI as Section A:

  • Textarea: Prompt

  • Slider: Tokens

  • Button: Generate

  • Output: Generated Email

• Paste/classify UI:

  • Textarea: Paste Email: (empty by default)

  • Button: Copy Generated → Paste box (optional helper)

  • Button: Classify Email

  • Output: Verdict (e.g., Verdict: DEEPFAKE)

Steps

1. Use Section A’s Prompt box to generate a clearly phishing-style email (e.g., password reset or payment issue).

2. Click Copy Generated → Paste box to move the generated text into the Paste Email box, or manually copy–paste it.

3. Click Classify Email to run qwen_guard on the pasted message.

4. Observe the Verdict output.

5. Repeat with different phishing prompts (invoice, delivery issue, payroll, etc.) and see if the guard consistently labels them as DEEPFAKE.

Expected Observation

• Most obviously phishing-like synthetic emails should be labeled DEEPFAKE by the guard.

• You may occasionally see UNCERTAIN if the message looks too benign or ambiguous.

• This section illustrates how a moderation / guard LLM can sit on top of a generator and provide a basic safety check.

Section C — Guard on Arbitrary Pasted Emails (Legit vs. Phish)

What this section does
Uses the same qwen_guard classifier, but focuses on user-supplied emails rather than just generated ones.
You will paste legitimate emails and synthetic phishing emails into the paste box and compare how the guard labels them.

UI in Colab

• Textarea: Paste any email here (Generated or Real):

  • You can paste:

    • benign work emails (meeting reminders, invoices, notifications)

    • synthetic phishing samples (from Section A)

• Button: Classify Email

• Output: Verdict (e.g., Verdict: LEGIT, Verdict: DEEPFAKE, or UNCERTAIN)

Steps

1. Paste a legitimate email (for example, a meeting reminder, homework notice, or a neutral invoice message with no urgent language).

2. Click Classify Email and note whether the guard says LEGIT, DEEPFAKE, or UNCERTAIN.

3. Now paste one of your synthetic phishing emails from Section A (password reset, invoice overdue, payroll verification, etc.).

4. Click Classify Email again and record the verdict.

5. Try a mix of:

   • clearly benign text (no links, no urgency, no account language)

   • borderline cases (e.g., legitimate password reset notifications with calm wording)

Expected Observation

• Many benign emails should be labeled LEGIT, but some may be flagged or marked UNCERTAIN if they resemble common phishing patterns.

• Most strong phishing-style emails (urgent tone, suspicious links, “verify now,” etc.) should be labeled DEEPFAKE.

• You will see where the guard is overly conservative (false positives) or too permissive (false negatives), illustrating the limits of a single LLM-based detector.

Section D — Probing the Guard’s Limits (Adversarial Testing)

What this section does
Explores how robust qwen_guard is by systematically modifying emails to try to evade detection.
You’ll start from known phishing samples and rewrite them to be subtler (less obvious keywords, friendlier wording), then observe if the guard still flags them.

UI in Colab

• Same paste/classify interface as Section C:

  • Textarea: Paste any email here (Generated or Real):

  • Button: Classify Email

  • Output: Verdict

Steps

1. Take a synthetic phishing email that the guard correctly labeled as DEEPFAKE in Section B or C.

2. Paste it into the Paste Email box and gradually soften it:

   • Remove or rephrase words like “urgent”, “immediately”, “verify your account”.

   • Make the link description sound more casual (but still suspicious).

3. Click Classify Email and see whether the guard still calls it DEEPFAKE or now says LEGIT/UNCERTAIN.

4. Try different adversarial edits:

   • Move the link to the bottom of the email.

   • Add polite greetings and more context.

   • Replace <link> with a more neutral placeholder, but keep the same intent.

5. Conversely, take a legit email and make it more “phishy” (add urgency, fake consequences, link language) and see when the guard flips from LEGIT to DEEPFAKE.

Expected Observation

• The guard’s decision boundary depends heavily on linguistic cues (urgency, action requests, mention of accounts/passwords, links).

• Subtle phishing emails with fewer obvious keywords may slip through as LEGIT or UNCERTAIN.

• Some legitimate emails that happen to use urgent or security-related language may be flagged as DEEPFAKE, showing potential false positives.

• This section demonstrates that a single guard LLM is helpful but not sufficient on its own; in real systems, you’d combine it with additional layers (URL checks, domain reputation, rule-based filters, etc.).

👉 Click here to see the result of each scenario (Post-Lab)