Methods

We explored the risks of Overreliance on AI-generated code and demonstrated how prompt engineering can mitigate security vulnerabilities.

The experiment utilized the Hugging Face Inference API (running openai/gpt-oss-120b) to simulate three distinct personas:

1. The "Helpful" Assistant: A standard LLM optimized for functionality, representing how most developers interact with AI.

2. The "Secure" Assistant: The same model conditioned with a strict "Security Expert" system prompt to prioritize defense.

3. The Code Auditor: A comparison bot that acts as a Static Application Security Testing (SAST) tool to analyze and explain the differences between the two outputs.
   
   

This case highlights that AI models default to "working" code rather than "secure" code unless explicitly guided otherwise.

Section A — The "Helpful" Assistant (Insecure Defaults)

• Simulates a standard developer workflow where a user asks for functionality (e.g., "Write a file upload function" or "Create a login query") without specifying security constraints.

• The model prioritizes brevity and functionality over robustness.

Observations:

• Generated code was functionally correct but security-flawed.

• Common vulnerabilities appeared immediately: SQL Injection (using f-strings instead of parameters), Directory Traversal (no filename sanitization), and XSS (unsanitized inputs).

• Demonstrated that "default" AI output is often unsafe for production.

Section B — The "Secure" Assistant (Prompt Engineering)

• Uses the exact same user prompts as Section A but applies a system prompt guard: "You are a security expert. Prioritize secure coding practices..."

• Forces the model to consider the "attack surface" before generating code.

Observations:

• The output code significantly improved in security posture.

• The model automatically included necessary libraries (e.g., werkzeug.utils.secure_filename or sqlite3 parameterization) that were missing in Section A.

• The response often included comments explaining why specific security checks were added.

Section C — The AI Code Auditor (Comparison)

• Acts as a judge, taking the outputs from Section A and Section B side-by-side to perform a comparative analysis.

• Identifies specific vulnerabilities in the first snippet and explains how the second snippet mitigates them.

Observations:

• The Auditor successfully identified the flaws in the "Helpful" code (e.g., "Code 1 allows SQL injection...").

• It correctly validated the defensive measures in the "Secure" code.

• demonstrated that while AI can write bad code, it is also highly effective at auditing code when tasked specifically to look for flaws.

Summary

Lab Objective:

Demonstrate the dangers of blindly trusting AI-generated code (Overreliance) and show how context/persona manipulation (Secure Prompting) is required to generate secure software.

Key Results:

• Insecure Defaults: Standard prompts consistently produced code vulnerable to OWASP Top 10 risks (SQLi, IDOR, Path Traversal).

• Secure Prompting: Simply changing the system persona reduced vulnerabilities by forcing the model to validate inputs and use safe libraries.

• Automated Auditing: The model proved effective at explaining why code was insecure, serving as a powerful educational tool for developers.

Lessons Learned:

• Context is King: An AI's output is only as secure as its system prompt.

• functionality != Security: Just because AI-generated code runs without errors does not mean it is safe to deploy.

• The "Human-in-the-Loop": Developers cannot treat AI as a "black box" that produces perfect code; they must act as reviewers and auditors.

Reflection:

• Overreliance is a behavioral vulnerability, not just a technical one.

• If developers do not know what to ask for (e.g., "sanitize this input"), the AI will not do it for them.

• The gap between Section A and Section B illustrates why prompt engineering is becoming a critical security skill.

Conclusion

Case 5 proves that AI-generated code is inherently risky when generated without security context. The "Helpful" Assistant consistently introduced critical vulnerabilities like SQL injection and directory traversal simply because it prioritized convenience. However, by establishing a "Secure" persona and utilizing AI as an Auditor, we converted the LLM from a liability into a security asset. This reinforces the principle that organizations must verify every line of AI-generated code and never rely on default outputs for critical systems.