Pre-Lab

Learning Objectives

After completing this learning module, you will be able to:

1. Define Hallucination and Overreliance in the context of Gen AI 

2. Explain why AI-generated code can often contain Insecure Defaults and Vulnerable Logic

3. Recognize the risks of blindly trusting AI outputs, including Package Hallucinations and supply chain risks

4. Demonstrate how refining your prompt can improve the quality of the output (like code) of the model

Prerequisites

• Python 3 (via Google Colab — no local install required)

• Basic familiarity with LLMs (system vs. user messages)

• A couple of test files to upload: One benign and one malicious containing a synthetic instruction hidden in text, comments, or invisible characters.

Key Terms

• Overreliance: A common and dangerous practice where the user incorrectly assumes AI-generated code is correct, secure, or legal without verifying it first

• Hallucination: When an AI model generates information that looks genuine, but in reality it is false, misleading, or even non-existent

• Insecure Defaults: Settings or code configurations generated by AI that prioritize convenience and functionality over security (e.g., allowing no authentication or unrestricted file uploads)

• Vulnerable Logic: Code patterns generated by AI that, while function correctly, introduce security flaws such as lacking input validation

• Package Hallucination: A type of hallucination where the AI recommends a made up software library or package that is not real. Attackers can exploit this by making the fake package "real"

Background

Generative AI tools like ChatGPT and GitHub Copilot have revolutionized coding by speeding up development. However, these models do not "know" how to code in the way a human security expert does; they predict the next word or line of code based on patterns in their training data.

Because LLMs are trained on vast datasets from the internet—which include both high-quality code and insecure, buggy code—they often reproduce the mistakes found in their training data. If the training data contains software vulnerabilities, the generated code will likely have them as well.

This module focuses on Overreliance. If developers assume code is safe simply "because the AI wrote it," they skip critical review steps. This can lead to the deployment of applications with hardcoded secrets, insecure database queries, or logic flaws that hackers can easily exploit.

The Risks of AI-Generated Code

1. Insecure Defaults and Vulnerable Logic

AI models are optimized for helpfulness, not necessarily security. When asked to "write a file upload function," the AI will provide the simplest code that works. Often, this code lacks necessary security checks because the AI does not understand the context in which the code will be used.

Example: Directory Traversal. If you ask an AI to write code to fetch a file based on a user's input, it might generate Python code using os.path.join.

• The Code: return send_file(os.path.join(“files”, filename))

• The Vulnerability: While this works for legitimate files like report.pdf, it does not sanitize the input. An attacker could input ../../../../etc/passwd to escape the directory and steal sensitive system files. 

• The Fix: The AI should have filtered for dangerous characters like .. or /, but it often won't unless explicitly told to.

Example: Type Juggling (Insecure Comparison)In languages like PHP or NodeJS, AI might generate code using "loose comparisons" (like ==) instead of "strict comparisons" (like ===).

• The Vulnerability: If an AI writes code to check a password hash using ==, an attacker can exploit "Type Juggling." By submitting a specifically crafted hash (like 0e654321), the system might treat the hash as scientific notation ($0$) and grant unauthorized access 17.

2. Hallucinations and Supply Chain Risks

Beyond writing bad code, AI sometimes invents things entirely.

• Factual Hallucinations: Asking an AI for a specific fact (like a birthday or a law) may result in a plausible but completely false answer because the model is "filling in the blanks" of sparse data.

• Package Hallucinations: A dangerous trend is when AI suggests importing a code library that does not exist. Attackers have learned to identify these hallucinated package names and upload malicious code with those exact names to public repositories. When a developer copies the AI's code and installs the package, they unknowingly infect their own computer

Defensive Measures: Prompting and Verification

The output of an LLM is only as good as the input it is given. To defend against these risks, we must change how we interact with the model.

"Secure" Prompt Engineering

We can manipulate the "Context" of the Ai to improve its output.

• Standard Prompt: Generate a python script that creates a configuration file."

  • Result: Likely insecure. For example, it might grant SYS_ADMIN privileges to a container, effectively giving it root access

• Secure Prompt: "You are a security expert. Generate a secure Python script for a configuration file. Ensure you drop all unnecessary privileges and follow the principle of least privilege."

• Result: The AI is now primed to prioritize security over convenience.

Verification & Human-in-the-Loop

You cannot rely on the AI to fact-check itself. 

• Code Review: Treat AI-generated code as untrusted user input. It requires expert review to ensure logic flows are secure.

• SAST Tools: Use Static Application Security Testing tools (like Snyk or JFrog) to scan generated code for known vulnerabilities before deployment.

• Check Licenses: AI might suggest open-source libraries that have restrictive licenses which could cause legal issues for your project.

In this lab, you will generate code using a standard prompt, observe the vulnerabilities, and then attempt to fix them by changing your prompt to a "Secure" persona.