Overview

What is Generative AI?

Generative AI (GenAI) refers to machine learning systems capable of creating new content—text, code, images, audio, or synthetic data—based on patterns learned from massive datasets. Unlike traditional AI that simply classifies or predicts, GenAI produces.

Examples include:

• Large Language Models (LLMs) such as GPT, Qwen, and Llama

• Diffusion Models for image synthesis

• GANs (Generative Adversarial Networks) for deepfakes and synthetic samples

• Autoencoders / VAEs for reconstruction and anomaly detection

GenAI is now deeply integrated into cybersecurity—both as a defensive tool and an offensive threat vector.

GenAI in Cybersecurity: A Dual Role

Defensive Applications

GenAI strengthens cybersecurity by enabling:

• Enhanced anomaly and threat detection

• Advanced log analysis and incident triage

• Automated malware analysis and reverse engineering

• Creation of synthetic attack samples for safer model training

• Intelligent intrusion detection systems leveraging LLM reasoning

Offensive Applications

At the same time, attackers exploit GenAI to:

• Generate deepfake phishing emails and scam URLs

• Create polymorphic malware that mutates to avoid detection

• Produce fake websites, storefronts, and crypto “rug pull” landing pages

• Automate social engineering scripts and victim profiling

• Perform adversarial attacks such as jailbreaks and prompt injection

This dynamic contributes to the Cyber-AI Arms Race, where both attackers and defenders leverage AI to outpace the other.

Key Terms

GenAI — Generative Artificial Intelligence

Models capable of producing synthetic content.

GAN — Generative Adversarial Network

System of two neural networks (generator + discriminator) that produce realistic fake data.

Prompt Injection

An attack that inserts malicious instructions into a model’s prompt to override its safety rules.

Indirect Prompt Injection

A more dangerous variant where hidden instructions are buried inside external files (metadata, comments, hidden text, OCR’d images) and silently influence the LLM’s behavior.

Deepfake Phishing

Highly convincing phishing content or URLs created using GenAI.

Typosquatting & Homograph Attacks

Using visually similar domains or Unicode characters to mimic real sites (e.g., paypa1.com).

Rug Pull

A crypto scam where developers abandon a project and steal investor funds; GenAI can help generate fake whitepapers and landing pages.

Cyber-AI Arms Race

The growing cycle where GenAI-powered attacks require GenAI-powered defenses.

About the Modules (M1–M6)

This lab series explores six major cyber threats amplified by Generative AI.

M1 — Prompt Injection (Direct Attacks)

Demonstrates how attackers can coerce an LLM into revealing “sensitive” data using manipulative or override-style prompts.

M2 — Indirect Prompt Injection

Shows how malicious instructions hidden in DOCX/PDF files or metadata can silently override a model’s intended task.

M3 — Phishing Email Detection

Uses GenAI to generate synthetic phishing emails and a guard model to classify them as LEGIT or DEEPFAKE, highlighting both strengths and detection gaps.

M4 — Scam/Phishing URL Detection

Explores how attackers generate scam URLs (fake storefronts, crypto scams) and how AI classifiers categorize them as Safe, Suspicious, or High-Risk Fraud.

M5 — Code Hallucination & Over-Reliance

Examines how LLMs hallucinate insecure or incorrect code, and how blindly trusting AI-generated code can introduce vulnerabilities.
Focuses on:

• unsafe file operations

• insecure cryptography recommendations

• subtle logic flaws

• developer over-reliance on AI suggestions

M6 — Malware Component Reconstruction (Honor)

Demonstrates how an LLM might unintentionally assist in generating components of malware when prompted strategically, even without explicitly “breaking” rules.
Covers:

• evasion wording

• high-level malicious component generation

• reconstruction of payload logic

• bypassing LLM safety through fragmentation

This final module emphasizes the ethical, legal, and security risks of LLM-assisted malware creation.

Why This Matters

Generative AI is reshaping cybersecurity:

• Defenders gain powerful tools for automation, detection, and analysis.

• Attackers gain scalable, highly adaptive methods for phishing, social engineering, malware, and evasion.

• Organizations must understand both sides to build robust security systems.

• Students must learn how GenAI can fail, be misused, or be attacked.

By exploring each module, you learn how GenAI strengthens cybersecurity while simultaneously creating new vulnerabilities, attack paths, and risks that modern systems must address.