Lab 5

What We'll Build

You will run a Colab notebook showcasing the dangers of blindly trusting AI-generated code, otherwise known as Overreliance. We will also be using a higher end model from Hugging Face Inference API to simulate three different AI personas:

1. The "Helpful" Assistant: Represents a normal LLM given a basic prompt. It usually prioritizes functionality rather than security.

2. The "Secure" Assistant: Represents a secure LLM with a strong "Security Expert" for the system prompt. It prioritizes input validation, sanitization, and least privilege. Meant to simulate a "good prompt". 

3. The Code Auditor: A comparison bot that analyzes the code outputs of the two Assistants and explain why one is safer than the other.

By the end of this lab, you will be able to see how simply changing how you ask for code can prevent vulnerabilities such as SQL Injection or Directory Traversal.

Environment & Libraries

• Model: openai/gpt-oss-120b (via Hugging Face API).

• Libraries: 

  • requests: To send prompts to the Inference API

  • ipywidgets: To create the interactive text boxes and buttons

  • os: To manage the API token

• UI Elements: 

  • Prompt Box: Where you ask for a specific function (e.g., “Generate a TCP server that executes client commands”)

  • Token Slider: Controls the length of the code output 

  • Generate/Compare Buttons: Triggers the specific AI persona

• Data: You will need a Hugging Face Token (HF_TOKEN) for this lab.

Open the Notebook

Colab Link (M5): https://colab.research.google.com/drive/1aStHqoKU3C05wxoxp6Q8hCvSbIC2ashh?usp=sharing 

1. Open the notebook in Google Colab.

2. Important: Read the instructions on how to install the HF_TOKEN then install dependencies

Section A — The "Helpful" Assistant (Insecure Defaults)

What this section does
Runs a standard LLM with a generic system prompt ("You are a helpful assistant"). This is meant to simulate what happens when a user asks a basic question without specifying security requirements or requesting code without knowing the attack surface that needs to be covered.

UI in Colab

• Textarea: Prompt (e.g.,“Write a Python function to upload files.”)

• Slider: Tokens (100–1024, default 768)

• Button: Generate Code

• Output: The raw code generated by the AI

Steps

1. Request code that can be "manipulated". Such code usually involves handling user input or external files.

   • Example 1: "Write a Python function that takes a filename from a user and sends that file to them."

   • Example 2: "Write an SQL query to check a user's login credentials."

2. Click Generate Code.

3. Analyze the Result: Can you find any security flaws? It's fine if you are not able to. Section C will compare the code results and find them for you. 

Section B — The "Secure" Assistant (Prompt Engineering)

What this section does

Runs the same model, but with a more secure system prompt. "You are a secure code assistant who generates code based on the user's prompt..."

UI in Colab

• Textarea: Prompt (Use the exact same prompt you used for Section A).

• Button: Generate Secure Code

• Output: A more secure version of the code

Steps

1. Copy the exact same prompt used in Section A.

2. Past it into the Section B input box.

3. Click Generate Secure Code.

4. Analyze the Result: Does it include more secure and safe practices? The AI should now explicitly add checks that were missing in Section A.

   • If the code output looks half-finished or looks like it cut out mid sentence, consider adding inside the prompt a character/word limit it must abide to (e.g., Write a Python function [don't exceed 2050 words total output]).

Section C — The AI Code (Comparison)

What this section does
Uses a third AI persona to compare the two results and see which output is more secure and why.

UI in Colab

• Textarea: Paste both code snippets here (label them)

• Button: Compare

• Output: Response

Steps

1. Copy the code output from Section A.

2. Paste it into the Section C box, labeling it [CODE 1].

3. Copy the code output from Section B.

4. Paste it below, labeling it [CODE 2].

5. Click Compare.

6. Read the Verdict: The Auditor should correctly identify the specific vulnerabilities (IDOR, XSS, SQLi) in the first snippet or explain the things the second snippet does that makes it more secure than the first.

Expected Observation

This lab proves that "AI-generated code" is not the standard, because "An AI's output is only good as its input". The security quality depends entirely on the Prompt Context. Not all models will have a secure prompt context like this lab, so you have to know exactly what you want from your code and what you want to avoid when typing it in your prompt. Expecting the LLM to give you the best possible code is and trusting it without looking for security faults is complete Overreliance.

👉 Click here to see the result of each scenario (Post-Lab)