Post-Lab

Methods

We evaluated how attackers could exploit generative AI to create highly convincing phishing emails and how defenders can classify and block them using an LLM-based guard.
Case 3 consists of two main components:

Attack Model (Qwen-Instruct):
Used to synthesize phishing emails from prompt templates (e.g., password reset, invoice reminder, urgent financial transfer). The model produces fully AI-generated, non-actionable phishing content for safety.
Defense Model (Qwen-Guard-Style Classifier):
A second LLM that classifies any email as DEEPFAKE, LEGIT, or UNCERTAIN using few-shot examples, strict labeling instructions, and fallback heuristics. The guard is applied to both generated emails and user-pasted samples.

This case demonstrates how easily LLMs can generate phishing messages and how guard models can help mitigate risks, while also showing where purely LLM-based detection still fails.

Section A — Phishing Generator (Baseline)

Produces fully synthetic phishing emails from natural-language prompts or predefined templates.
No restrictions or content filtering: whatever the user asks the model to generate will be produced (within safety placeholders like <link>).
Emails include realistic urgency cues (“immediately”, “verify your account”, “payment failed”), making them effective social-engineering simulations.

Observations:

The generator easily created convincing phishing messages from simple prompts.
Slight phrasing changes generated entirely different attack styles (password resets, invoices, delivery notices).
Without safeguards, a generative model can rapidly produce large volumes of phishing-style text.

Section B — Generator + Guard (Synthetic Emails)

Adds an LLM-powered classifier (qwen_guard) that receives the generated phishing text and attempts to label it as DEEPFAKE or LEGIT.
The classifier prompt includes a small set of few-shot labeled examples showing what phishing vs. legitimate emails look like.

Observations:

The guard successfully detected most synthetic phishing messages, especially those containing urgency, links, or IT/security language.
Some subtle or overly polite phishing emails returned UNCERTAIN or were misclassified as LEGIT, revealing the guard’s sensitivity to wording.
LLM-only classification is helpful but not consistently reliable across ambiguous or softly-worded attacks.

Section C — Guard on Arbitrary Pasted Emails

Extends the guard to any email text pasted by the user (realistic work emails, generated phishing, or borderline cases).
This tests the classifier’s practical usability on mixed content.

Observations:

Legitimate emails were usually labeled LEGIT, but certain routine IT messages (e.g., real password-reset notices) were occasionally flagged as DEEPFAKE, indicating false positives.
Subtle phishing messages with mild tone or minimal keywords could slip through as UNCERTAIN or LEGIT, indicating false negatives.
The classifier focuses heavily on linguistic cues rather than deeper semantic understanding of intent.

Section D — Adversarial Testing

Students modified phishing emails to gradually reduce obvious keywords, or strengthened benign emails to look more suspicious.
Purpose: identify where the guard breaks, slips, or fails to detect softer attacks.

Observations:

Removing or rephrasing urgency signals significantly reduced detection accuracy.
Adding friendly context or reducing explicit calls-to-action often caused the guard to return LEGIT despite malicious intent.
Conversely, adding a single suspicious keyword to a benign message could trigger DEEPFAKE, showing overfitting to surface patterns.
LLM-based guards alone remain vulnerable to prompt manipulation and adversarial rewriting.

Summary

Lab Objective:
Demonstrate how generative models can synthesize phishing emails and how an LLM-based guard can classify them, revealing both strengths and weaknesses of AI-driven email security.

Key Results:

Phishing Generator: Quickly produces plausible phishing messages with minimal prompting.
Synthetic Email Detection: Guard successfully flags most overt phishing attempts.
Pasted Email Detection: More variation—legitimate emails sometimes misflagged; subtle phishing sometimes missed.
Adversarial Testing: Guard can be evaded with strategic rewrites or deceived by legitimate wording with suspicious cues.

Lessons Learned:

Generative AI significantly lowers the barrier for producing phishing-style content.
Simple guard LLMs improve safety but are not sufficient alone; their decisions depend heavily on writing style and keywords.
False positives and false negatives highlight the need for additional layers (URL analysis, domain reputation, rules-based filters).
Guard prompts and few-shot examples greatly influence outcomes; tuning them is critical for real-world performance.

Reflection:

Purely LLM-based detection offers convenience and flexibility but suffers from inconsistency on edge-case emails.
Stronger defenses require multi-layer systems, not a single classifier.
Using synthetic data is effective and safe for training and experimentation.
Understanding how attackers can subtly rewrite messages is essential for defending against LLM-augmented phishing.

Conclusion

The Case 3 pipeline shows that while LLMs can generate highly convincing phishing emails, a guard LLM provides a meaningful first line of defense.
Although not foolproof, Qwen-Guard significantly reduces risk by identifying many malicious patterns.
However, fully reliable email security requires layered defenses—combining LLM guards with traditional detection systems—to prevent adversarial manipulations and classification failures.

Page updated

Google Sites

Report abuse