Post-Lab

Methods

We tested how generative AI can be used to rapidly create scam URLs across multiple fraud categories and how an LLM-based guard can analyze and classify those URLs based on risk patterns.
The experiment consisted of:

Attack Model (Qwen Instruct):
Used to generate synthetic malicious URLs (e.g., fake storefronts, crypto rug pulls, fake virus alerts).
The model outputs structured scam URLs with fake brands, invented TLDs, and recognizable social-engineering patterns.
Defense Model (URL Guard):
The same LLM, prompted differently, serves as a classifier.
It labels URLs as Safe, Suspicious, Likely Fraud, or High-Risk Fraud, and provides a short rationale.

This case demonstrates how attackers can automate scam infrastructure generation and how defenders can rely on lexical-pattern LLMs to detect risk indicators.

Section A — Scam URL Generation Baseline

The generator produces scam URLs with no filtering or defensive constraints.
Each category produces distinct linguistic and structural patterns:
- Fake storefronts mimic checkout/login flows
- Crypto scams reference token claims, swaps, and “airdrop” keywords
- Tech support scams mimic system warnings

Observations:

Generative AI can output believable scam URLs instantly.
Even vague prompts yield structured, realistic formats.
Minor prompt changes shift the type of scam (e.g., crypto vs storefront).
Demonstrates how AI can scale attacker capabilities effortlessly.

Section B — Guard Classification on Generated URLs

The guard inspects each URL’s structure, spelling, domain components, subdomains, and TLD.
Outputs 3 fields: risk, category, rationale.

Observations:

Most synthetic scam URLs were accurately labeled Likely Fraud or High-Risk Fraud.
The rationale often highlighted:
- Misspellings
- Suspicious subdomains
- Scam-like keywords (“support-alert”, “token-swap”, “verify-login”)
Some borderline URLs were labeled Suspicious instead of High-Risk.

Overall, the guard is effective at detecting overtly fraudulent patterns.

Section C — Guard Classification on Arbitrary Pasted URLs

User-pasted legitimate URLs, synthetic URLs, and ambiguous URLs were tested.

Observations:

Legitimate domains (google.com, github.com) were consistently marked Safe.
Benign but unusual URLs (odd parameters, tracking IDs) occasionally triggered Suspicious.
Very subtle scam URLs sometimes slipped through as Suspicious instead of Likely Fraud.
The guard relies heavily on lexical cues and spelling, meaning:
- Strong signals → good detection
- Soft signals → reduced confidence

Shows how classification accuracy depends on URL wording rather than deeper semantic analysis.

Section D — Adversarial URL Testing

Students attempted to bypass the classifier by modifying URLs with typosquats, Unicode characters, or less obvious indicators.

Observations:

Typosquats (paypa1, arnazon) were usually caught.
Unicode/homograph attacks had mixed results; some were detected, others not.
Parameter-based trickery (adding “verify-session” inside URL parameters) sometimes bypassed high-risk classification.
Some harmless URLs with strange patterns triggered false positives.

This demonstrates both the power and limitations of LLM-based lexical detection in adversarial conditions.

Summary

Lab Objective:

Show how generative AI can produce synthetic scam URLs and how an LLM-based guard can classify those URLs for defensive analysis.

Key Results:

Scam URL Generation: Fast, diverse, and highly scalable—LLMs generate domains that mimic real fraud patterns.
Guard Classification: Strong at identifying overt scams; understands common risk indicators.
Real-world URLs: Safe sites correctly identified; ambiguous URLs sometimes misclassified.
Adversarial Testing: Unicode tricks, subtle typos, and soft indicators can reduce the guard’s certainty.

Lessons Learned:

LLMs can both amplify attacker capability and provide powerful defensive analysis.
URL guards are effective but rely heavily on textual cues—attackers who manipulate structure subtly can evade detection.
Defenses should combine LLM classification with:
- DNS reputation
- Endpoint scanning
- URL pattern heuristics
- Blocklist/whitelist systems

Reflection:

The balance between detection accuracy and false alarms is delicate.
While LLMs are adaptable, they are not infallible and can be fooled by adversarial inputs.
Attackers using AI can rapidly mutate URLs, reinforcing the concept of a Cyber-AI Arms Race where both sides escalate with AI tools.

Conclusion

The Case 4 experiment shows that generative AI can effortlessly construct realistic scam infrastructure, but LLM-based guards offer a promising defensive layer.
Although not perfect, the guard classifier effectively identifies many fraudulent URL patterns and provides useful semantic explanations.
However, subtle adversarial inputs reveal gaps that require additional layers of cybersecurity beyond LLMs.

AI-driven detection is powerful—but must operate within a broader security ecosystem to counter increasingly AI-generated threats.

Page updated

Google Sites

Report abuse