Post-Lab

Methods

We tested three different ways of how attackers could manipulate AI output through indirect prompt injection. For the fourth case, we implemented a moderation LLM to block unsafe/unsanitized inputs and shut down attempts to override the model.

Section A — DOCX Baseline

Ingests all text from a Word document: body, headers, footers, tables, textboxes, comments, footnotes, endnotes.
No filtering or sanitization; hidden payloads are treated as legitimate input to showcase the dangers of unsanitized input
Observations: Malicious instructions in comments or invisible text easily override the model’s task.

Section B — PDF Baseline

Extracts text from PDF files using multiple methods (pdfplumber, PyMuPDF, OCR)
Again, no filtering; all extracted text is fed directly into the model
Observations: Hidden payloads embedded in fonts, images, or OCR text successfully influenced the model’s output. Vulnerable to indirect injection

Section C — Metadata Injection

Demonstrates how metadata fields (Title, Subject, Comments, Creator) can also be used to carry hidden instructions
Metadata is concatenated with resume text and ingested by the model
Observations: Metadata injection proved especially dangerous because humans rarely review metadata fields, yet the model obeyed hidden instructions.

Section D — Moderation LLM

Specialized guard model classifies inputs as Safe / Unsafe / Controversial and assigns categories (e.g., Jailbreak, PII, Unethical Acts)
Blocks unsafe inputs before they reach the responder model.
Observations: Reliable against metadata and indirect injections. Slight latency increase, occasional false positives, but strong overall defense.

Summary

Lab Objective: Demonstrate how indirect prompt injection via resumes and external files can override model instructions, and how moderation LLMs mitigate these risks.

Key Results:

- DOCX Baseline: Vulnerable; hidden payloads easily influence outputs.
- PDF Baseline: Equally vulnerable; OCR text and embedded payloads bypass defenses.
- Metadata Injection: Dangerous attack vector; metadata instructions override tasks.
- Moderation LLM: Most robust; classified and blocked unsafe inputs; acceptable latency trade‑off.

Lessons Learned:

- External files (DOCX/PDF) pose unique risks: hidden payloads in comments, textboxes, or metadata can silently override instructions.
- Metadata injection is especially insidious because it bypasses human review.
- Without guards, both DOCX and PDF ingestion are highly vulnerable.
- Moderation LLMs provide specialized, reliable defense when combined with careful input selection.

Reflection:

- Trade‑offs exist between coverage, speed, and complexity.
- Baseline ingestion is fast but unsafe.
- Metadata injection shows how subtle attacks can bypass human oversight.
- Moderation LLMs are strongest but add latency. Choosing the right defense depends on the sensitivity of the task (e.g., resume screening vs. high‑security applications).

Conclusion
The Moderation LLM is an effective guard to counteract indirect prompt injection. Not only does it provide complete protection against hidden payloads in both text and metadata, it also has acceptable trade‑offs in speed and complexity

Page updated

Google Sites

Report abuse