In this work, we implement CoT by instructing LLM to follow our defined steps (in Fig.2) while providing explicit reasoning for each step’s output.
Step 1: Attack Steps Extraction The LLM analyzes the syntactic structure of input attack report text (I) to identify subject-verb-object relationships, forming triples (T = ⟨ actori, actioni, targeti⟩), where each triple represents a dis tinct attack step. As shown in the Fig.2, from “The threat actors using IP 104.223.34.98 gained initial access to Victim 2’s pro duction environment”, we form (⟨ threat actors, Network Request, Victim 2’s production environment⟩).
Step 2: Attack Effect Iden tification Since attack effects refers to the impact of an action on targets, we focus on (⟨ actioni, targeti⟩) pairs from (T). For each pair, the LLM analyzes the contextual information to determine its corresponding attack effect (Ei). For example, (⟨Tool Execution, PowerShell⟩) triggers (E = Program Execution), indicating a behav ioral effect on the system.
Step 3: Monitoring Items Generation For each (⟨actioni, targeti, Ei⟩), the LLM generates corresponding monitoring item (Mi). As shown, when (E = Program Execution), the LLM generates (M = Process creation monitoring).
You are a security analysis assistant. Given an unstructured security incident report, follow these steps to analyze the incident and generate appropriate monitoring recommendations:
Step 1: Attack Steps Extraction
Analyze the input text and break down the attack sequence into individual steps. For each step, identify:
- Who (threat actor)
- What action they took (verb)
- Against what target (object)
Output format: List each step as (actor, action, target)
Step 2: Attack Effect Identification
For each attack step identified above:
- Focus on the (action, target) pairs
- Analyze the technical impact of the action
- Determine the attack effect category
Output format: Map each (action, target) to its effect category
Step 3: Monitoring Items Generation
For each attack effect identified:
- Generate specific monitoring recommendations
- Focus on actionable detection methods
- Consider multiple data sources when relevant
Output format: List specific monitoring items for each effect
Let's solve this step by step:
Input: [Security incident description]
Step 1: Attack Steps Extraction
Given the input, I identify these attack steps:
1. (Actor: {who}, Action: {what}, Target: {target})
2. ...
Reasoning: I extracted these steps by...
Step 2: Attack Effect Identification
Analyzing each action-target pair:
1. ({action1}, {target1}) → Effect: {effect1}
Because: {reasoning}
2. ...
Step 3: Monitoring Items Generation
Based on the identified effects, here are the recommended monitoring items:
1. For {effect1}:
- Monitor: {specific_monitoring_item}
- Data source: {source}
- Justification: {why}
2. ...
Summary:
Critical monitoring points:
1. {key_point1}
2. {key_point2}
...
Example:
Input 1:
"Disco has achieved initial access and execution through content injection into DNS, HTTP, and SMB replies to targeted hosts that redirect them to download malicious files."
Output 1:
Step 1: Attack Steps Extraction
I identify the following attack steps from the incident:
(Actor: Disco, Action: Achieved, Target: Initial Access)
Reasoning: "Disco has achieved initial access" indicates the first step in the attack chain where the attacker gains access.
(Actor: Disco, Action: Content Injection, Target: DNS, HTTP, SMB replies)
Reasoning: The phrase "content injection into DNS, HTTP, and SMB replies" describes an attack action targeting the network protocols.
(Actor: Disco, Action: Redirect, Target: Targeted hosts)
Reasoning: "Redirect them to download malicious files" refers to the attacker redirecting the hosts to download the malicious payload.
(Actor: Disco, Action: Download, Target: Malicious files)
Reasoning: This step involves the targeted hosts downloading malicious files as a result of the redirection.
Step 2: Attack Effect Identification
Now, let's analyze the attack effects of each action-target pair:
(Achieved, Initial Access) → Effect: Initial Access
Reasoning: The action "achieved initial access" leads to the attacker gaining unauthorized access to the target system.
(Content Injection, DNS, HTTP, SMB replies) → Effect: Network Traffic Manipulation
Reasoning: Content injection into network protocol replies is a form of manipulation of legitimate network traffic, enabling redirection of hosts.
(Redirect, Targeted hosts) → Effect: Malicious Redirect
Reasoning: The redirection of targeted hosts to a malicious destination represents a manipulation of traffic flow, leading the hosts to an unwanted or harmful resource.
(Download, Malicious files) → Effect: Malicious File Execution
Reasoning: The downloading of malicious files directly results in the execution of malicious payloads, enabling further exploitation.
Step 3: Monitoring Items Generation
Based on the identified effects, here are the recommended monitoring items for detection:
For Initial Access:
Monitor: Network-based access attempts
Data source: Firewall logs, VPN logs, IDS/IPS
Justification: Detect unauthorized access attempts that may indicate initial exploitation.
For Network Traffic Manipulation:
Monitor: Anomalous DNS, HTTP, and SMB traffic patterns
Data source: DNS logs, HTTP proxy logs, SMB traffic logs
Justification: Track abnormal content or behavior in network protocol responses to identify potential traffic manipulation.
For Malicious Redirect:
Monitor: DNS queries and redirects to suspicious domains
Data source: DNS logs, HTTP logs
Justification: Monitor for unusual redirections to external malicious IPs or domains indicating a compromise.
For Malicious File Execution:
Monitor: File download activity from untrusted sources
Data source: Web proxy logs, endpoint security logs, file integrity monitoring
Justification: Detect downloads from untrusted or suspicious domains or IPs and any execution of files from these sources.
Summary:
Critical monitoring points:
Initial Access – Monitor network access logs to detect unauthorized access attempts.
Network Traffic Manipulation – Watch for abnormalities in DNS, HTTP, and SMB traffic.
Malicious Redirects – Monitor DNS and HTTP logs for suspicious domain redirections.
Malicious File Execution – Track file downloads and executions, especially from untrusted sources.
Input2:<input our attack report text for analysis>
Output:
The threat actors using IP 104.223.34.98 gained initial access to Victim 2’s production environment in late January 2022, or earlier. These actors likely obtained access by exploiting Log4Shell in an unpatched VMware Horizon server. On or around January 30, likely shortly after the threat actors gained access, CISA observed the actors using PowerShell scripts to callout to 109.248.150.13 via Hypertext Transfer Protocol (HTTP) to retrieve additional PowerShell scripts. Around the same period, CISA observed the actors attempt to download and execute a malicious file from 109.248.150.13. The activity started from IP address 104.155.149.103, which appears to be part of the actors’ C2 infrastructure.
Use the example attack report above as input to CoT Prompt
The following output can be obtained by analyzing the preceding attack report example using LLM.
Step 1: Attack Steps Extraction
The attack steps are:
(Actor: Threat actors, Action: Gained, Target: Initial access to the production environment)
Reasoning: The attackers gained initial access, likely through Log4Shell exploitation.
(Actor: Threat actors, Action: Used, Target: PowerShell scripts via HTTP)
Reasoning: They used PowerShell scripts to contact a malicious server, likely to retrieve additional tools.
(Actor: Threat actors, Action: Attempted, Target: To download and execute a malicious file)
Reasoning: They downloaded and attempted to execute a malicious script to progress their attack.
(Actor: Threat actors, Action: Conducted, Target: C2 activities)
Reasoning: C2 activities indicate ongoing communication with their infrastructure.
Step 2: Attack Effect Identification
The effects identified for each step and corresponding actions are:
(Gained, Initial access) → Effect: Network Request
Reasoning: Indicates an external network connection to gain initial access.
(Used, PowerShell scripts) →
Effect 1: Command Execution (PowerShell)
Reasoning: Represents the execution of PowerShell commands.
Effect 2: Network Request
Reasoning: Involves an outbound HTTP request to a malicious server.
(Attempted, To download and execute malicious file) →
Effect 1: Tool Download (Malicious Script)
Reasoning: Malicious script downloaded for further exploitation.
Effect 2: Tool Execution (Malicious Script)
Reasoning: Attempted execution of the downloaded malicious script.
(Conducted, C2 activities) → Effect: Network Request (C2 Connection)
Reasoning: Communication with the attacker’s C2 infrastructure.
Step 3: Monitoring Items Generation
Based on the effects, the monitoring items are:
For Network Request:
Monitor: Network connection monitoring
Reasoning: Detect unauthorized external connections.
For Command Execution (PowerShell):
Monitor: Process creation monitoring
Reasoning: Track PowerShell execution to detect malicious commands.
For Network Request:
Monitor: Network connection monitoring
Reasoning: Identify suspicious HTTP requests to external servers.
For Tool Download (Malicious Script):
Monitor: Target file/folder monitoring
Reasoning: Detect downloads of malicious scripts in specific file paths.
For Tool Execution (Malicious Script):
Monitor: Process creation monitoring
Reasoning: Track the execution of malicious tools/scripts.
For Network Request (C2 Connection):
Monitor: Network connection monitoring
Reasoning: Detect ongoing communication with C2 infrastructure.
Summary of Critical Monitoring Points
Network connection monitoring – Identify unauthorized or suspicious external connections.
Process creation monitoring – Track execution of scripts and commands to detect malicious activities.
Network connection monitoring – Detect suspicious HTTP requests.
Target file/folder monitoring – Monitor downloads and suspicious file modifications.
Process creation monitoring – Detect execution of downloaded malicious files.
Network connection monitoring – Identify C2 communication with external infrastructure.
Based on the real-word attack reports collected in the Attack Reports Source , we have organized and analyzed them using the CoT-guided LLM, resulting in the following set of monitoring items.