This blog is a summary of my talk at the Offbyone 2026 cybersecurity conference. It records some of my thoughts as a cybersecurity researcher after spending a year using LLMs for research.
Slides: https://github.com/edwardzpeng/presentations/tree/main/offbyone%202026
Chinese version: https://sites.google.com/site/zhiniangpeng/blogs/Hacking-with-LLMs
Over the past year, I have spent a great deal of time on one thing: trying to make LLM genuinely participate in cybersecurity research. The results have continued to surprise me.
In 2023, our first attempt to use an LLM for bug huntingfailed. By the end of 2025, we were already using LLMs with agents to analyze vulnerabilities, generate PoCs, and validate them on real devices. After entering 2026, some vulnerability research had even become close to highly automated. I then applied the same methods to cryptography and other cybersecurity projects, where the performance of LLMs was equally astonishing.
## First Attempt: Bug hunting with Code Llama
The story began in 2023.
Our idea at the time was very straightforward: if an LLM could understand code, then after seeing enough vulnerable code, it should be able to determine whether a piece of code contained a vulnerability and identify the specific bug.
We collected approximately 40,000 Python vulnerability samples, 37,000 Java vulnerability samples, and 40,000 clean-code samples to fine-tune Code Llama. We hoped that, given a piece of code as input, it would output either “vulnerable,” together with the specific vulnerability, or “not vulnerable.”
It performed very well on the benchmark. Among 24 test classes, eight achieved a score of 100%, while 23 scored above 90%.
Then we tested it on real-world code. The result was almost useless.
The model produced many findings that looked like vulnerabilities but were not actually exploitable. The output was noisy, non-actionable, and impossible to validate. The beautiful benchmark numbers did not translate into real vulnerabilities.
Looking back, the reasons for the failure are now quite clear:
1. **The target was wrong.** Simple LLM-based recognition of vulnerability patterns is not the same as the process of vulnerability research.
2. **The capability was wrong.** More SFT data does not mean deeper reasoning.
3. **It overfit CVE patterns.** Common code features in CVEs are not the same as the security invariants that a system actually needs to preserve.
4. **There was not enough context.** A code snippet is not the complete context of the target system.
5. **There was no threat model.** Without security boundaries, adversary capabilities, entry points, and security goals, it is difficult to determine whether a behavior is truly a vulnerability.
6. **It could not validate its findings.** The model could not run, debug, and verify its own conclusions.
7. **There was no continuous research process.** A single question and answer is not research. Research requires a cycle of hypothesis, evidence, and revision.
Simply put, we treated vulnerability discovery as a classification problem.
Our idea was still too simple, and the LLM ecosystem was not mature enough. At the time, it could not yet be used effectively for vulnerability research.
## What Had Changed by the End of 2025?
Two years later, the situation was completely different.
First, the models themselves had become much stronger. Models such as GPT-5.2 and Opus 4.5 had much better reasoning capabilities and no longer merely recognized superficial patterns in code.
Second, context windows had become longer. More complete code repository, call chains, object lifetimes, and trust boundaries can be put into the context. The model no longer saw only an isolated function; it saw a whole system.
Agent workflows had also matured. Approaches such as ReAct, Plan-and-Execute, Chain of Thought, and Multi-Agent systems allowed models to break complex tasks into many steps and adjust their plans based on intermediate results.
Finally, more tools had become available. MCP, CLI, A2A, skills, and various deterministic analysis and execution scripts allowed models to operate on real targets instead of merely returning an answer in a chat box.
The real breakthrough, therefore, was not only a better model. It was a more complete system built around the model.
## Second Attempt: LLM + Agent
At the end of 2025, we tried again to use an LLM for bug hunting. This time, the target was access-control vulnerabilities in customized Android systems.
This was an ideal scenario for testing an agent.
The Android ecosystem today includes not only phones, but also watches, bands, TVs, in-vehicle systems, and many kinds of IoT devices. Different vendors add large numbers of OEM services to their ROMs. These services usually have elevated privileges while exposing interfaces to ordinary apps.
In theory, those interfaces should protect sensitive data and privileged operations through mechanisms such as permissions, UIDs, signatures, and AppOps. In complex customized code, however, gaps frequently appear in permission validation. A low-privilege third-party app may consequently gain access to device identifiers, personal data, or system settings.
This research was conducted between December 2025 and January 2026 by X0ev, Wh1tc from Kunlun Lab and me.
### Preparing Tools for the Agent
We did not simply ask the model to “read the entire firmware and find vulnerabilities.” Instead, we first prepared a collection of deterministic tools:
- **Firmware Unpacking:** Extract APKs from the system, vendor, and product partitions.
- **JADX Decompilation:** Convert system APKs into readable Java code.
- **Manifest Scan:** Organize permissions, actions, and components.
- **Indexing:** Index file paths, packages, and classes.
- **Task Generation:** Select exported, high-risk system components.
- **ADB and Logcat:** Invoke real interfaces and collect access-control evidence.
The tools themselves are deterministic. The skill determines when and in what order the agent should use them.
### Two-Step Reasoning
For high-risk APIs, we asked the agent to complete two reasoning steps.
The first step was to understand the behavior: which identities, parameters, and system states could trigger a sensitive operation?
The second step was to determine authorization: did this operation exceed the privileges that the caller should have? If so, it became a candidate vulnerability.
These two questions may look simple, but they distinguish between something that merely “looks like a vulnerability” and something that genuinely violates a security boundary.
### Validation on Real Devices
A candidate vulnerability was not enough. The agent needed to generate a PoC, run it on a real device, collect Logcat output, and then revise and retry the PoC based on the failure results.
Common causes of failure included an incorrect call order, a missing required state, unmet data dependencies, and incorrect permissions or runtime environments. The agent’s job was not to produce the correct answer in a single attempt, but to continue revising its work after a failure.
Using Codex GPT-5.2 Extra High, we eventually tested real devices from OPPO, HONOR, Xiaomi, Samsung, Google, and other vendors. The results were:
- 110 confirmed vulnerabilities;
- 183 reports;
- an average cost of less than one US dollar per PoC;
- approximately two iterations per PoC on average.
This time, the LLM was no longer merely a chatbot.
In 2023, we asked it, “Does this code look like a known vulnerability pattern?”
By the end of 2025, we were asking an agent to understand a system, investigate a bug candidate, and validate it with a PoC. The AI used tools, performed scoped analysis, and generated reports. Researchers selected attack surfaces, built the harnesses and skills, assessed the value of the results, and took responsibility for the final conclusions.
This is vulnerability research conducted jointly by researchers and AI.
## From Patches to Vulnerability Variants: Highly Automated Bug Hunting
In the previous research, a human still needed to select the attack surface and build a harness and skill around a particular security invariant.
With a stronger model and agent, could AI complete this part as well?
If we had enough data, especially patches, public write-ups, and PoCs, the answer was beginning to become “yes.”
We built Diffract. This is a joint work with Yunpeng Tian, @___2st.
The core Diffract workflow is:
1. Collect public vulnerability write-ups and PoCs.
2. Build a foundational knowledge base and skill.
3. Analyze patch diffs.
4. Extract vulnerability root causes.
5. Abstract and preserve vulnerability patterns.
6. Search for variants in other code.
7. Build PoCs.
8. Feed new results back into the knowledge base and skill.
This is no longer a one-off question and answer. It is a research loop capable of accumulating knowledge.
Using Claude Opus 4.6 through 4.8 on Microsoft Windows Kernel and Services, we obtained:
- more than 200 confirmed vulnerabilities;
- more than 400 reports;
- RCE, LPE, and information-leak vulnerabilities across different modules.
In fact, this has already become a form of token economics. In the short term, the more tokens you can burn, the more vulnerabilities you can find, because the backlog of legacy vulnerabilities is simply enormous. Windows vulnerability research has become quite boring in 2026. I hope Microsoft can quickly clear this huge backlog of vulnerabilities. Although they actually doing a great job, this month almost 1,000 vulnerabilities fixed.
### The State of Vulnerability Research in 2026
By 2026, my assessment of LLM-based bug hunting was:
- With previous-generation models, bug hunting works as long as the harness is sufficiently good. For example, we found many vulnerabilities using GPT-5.2 and GLM-5.2.
- With a sufficiently capable model, nearly fully automated bug hunting has become possible. Opus 4.6, Kimi K3, DeepSeek V4, GPT-5.4, and GLM 5.3 are already good enough.
- Today, with a SOTA model, all you need to do is make a wish to the model, and it can find vulnerabilities. There is no longer any barrier to entry.
- These three changes happened within six months. Things are changing extremely quickly.
- Bug hunting has evolved into a form of token economics.
In the past, the primary cost of bug hunting was researcher's time. Model capability, tokens, tools, and computing resources are now becoming part of a new cost structure.
This article does not discuss vulnerability exploitation, but the situation is basically the same.
### How to Use AI for Bug Hunting
How to use AI depends on what model you have. The main difference is the level at which the researcher needs to provide help:
- If the model is less capable, the researcher needs to break the task into smaller steps, prepare the context, tools, and tests, and frequently check and correct its work. Much of the work is actually using engineering methods to make up for the model's limitations and ensure that the whole process runs correctly.
- If the model is good enough, it can usually run the full process independently. The researcher mainly monitors the research process, provides background information, system access, the runtime environment, or debugging experience at key moments, and judges the final results.
- If you use a SOTA model, routine analysis and validation often need little help. The researcher should focus on where the model failed to find vulnerabilities: check what it overlooked, add runtime states or cross-module relationships that it could not see, change the search and validation methods, and let it continue trying where it previously failed.
Therefore, the stronger the model, the less the researcher needs to help it complete the process, and the more the researcher needs to help it push beyond the process's current capability boundary.
### About Vulnerability Exploitation
In fact, vulnerability exploitation is also being dramatically accelerated by AI, but that is not the focus of this article.
### /goal Give Me an iOS 0-Click Fullchain RCE Exploit
Do you think AI can complete this /goal?
Probably not with your $200 SOTA subscription. But what if it were an internal OpenAI model, with 10,000 agents collaborating for 88 hours? (The resources OpenAI used to solve the Navier–Stokes Millennium Prize Problem.)
### Is Vulnerability Research Dead?
We are currently in a `bug for free` era because AI can very easily uncover the enormous backlog of existing vulnerabilities. Fixing them all will take a long time.
This will be a chaotic period. Anyone without a security background can use a SOTA model to discover and exploit serious vulnerabilities. But chaos always passes. Eventually, vulnerability discovery and remediation will reach a steady state. AI capabilities will reach a frontier, and the work of vulnerability research will be to keep pushing beyond that frontier.
Vulnerability research may not die, but in just six months it has already been completely transformed.
## Bug Hunting Was Getting a Little Boring, So I Tried Cryptography
I have a PhD in cryptography. From 2013 to 2017, I spent four years conducting cryptography research.
But I never broke a genuinely strong cryptosystem. After 2017, I returned to real-world systems, doing offensive and defensive security research. The reason was simple: mathematics was too hard for me.
So I began to wonder whether AI could help me become a real codebreaker.
### First Attempt: Reanalyzing My Own Design
I gave GPT-5.6 an unpublished attribute-based signcryption scheme and asked it to prove the scheme secure. I had never been able to complete the proof myself.
About 20 minutes later, it explained why the scheme could not be proven secure: it had found a concrete attack.
It seems that leaving cryptography was the correct decision for me. GPT surpassed my four years of research in 20 minutes.
### Second Attempt: Testing Related Schemes
GPT-5.6 then extracted a structural weakness from the attack and applied it to a family of related cryptographic constructions.
Almost all of the related schemes were quickly broken.
Apparently, I am not the only person who needs to leave cryptography. AI can quickly prove and disprove cryptographic schemes, and cryptography research will also be completely transformed.
If AI breaks an important cryptographic scheme in the future, or even solves some difficult underlying mathematical problems, I will not be particularly surprised.
## AI Is Accelerating More Than Security Research
Looking beyond the cryptography experiments, I believe AI is accelerating the entire scientific research process. Cybersecurity, software, cryptography, and mathematics may be affected first because research in these fields can easily form a computational closed loop:
1. Form a hypothesis.
2. Analyze and reason.
3. Prove it or find a counterexample.
4. Revise the theory or design.
5. Repeat the process.
AI and related tools can participate in almost every step of this loop.
The focus of the debate has shifted from “Can AI solve problems?” to “Can AI possess genuine creativity?”
My view is that creativity is difficult to quantify and difficult to define. If we cannot accurately explain what creativity is, it is equally difficult to prove that AI will never possess it.
Now imagine a future world combining embodied intelligence, world models, and physical-native AI. AI’s transformation of the world has only just begun.
## AI Will Change Cybersecurity First
AI will change the world, and the cyber world will take the first hit.
### Red Team
In our red-team simulations, undergraduate students using Zcode, GLM-5.2, tools, and simple skills could cover the complete red-team workflow, including reconnaissance, phishing, PoC testing, EDR evasion, post-exploitation, and mining high-value information from large datasets.
AI makes offensive capabilities cheaper and more accessible.
### DFIR
In our digital forensics and incident-response projects, AI can already assist with detection, triage, log analysis, correlation, response, and reporting.
At the same time, we have found in some projects that attackers are already making extensive use of AI. Interestingly, keywords such as `Pentest`, `Red team`, `CTF`, and `Security audit` are very common in logs, scripts, files, and other artifacts left behind by some attackers. They are clearly intended to bypass AI safety guardrails and induce AI to assist with attacks. We have even observed similar behavior in some APT incidents.
The confrontation between AI attackers and AI defenders is already underway in the cybersecurity world.
### Larger-Scale Automated Attacks
Stronger AI, lower costs, and wider availability will eventually bring large-scale attacks to the cyber world.
Imagine a future in which someone only needs to give an agent a goal:
> /goal Get me ten Bitcoin and send them to my wallet.
Or:
> /goal Obtain a secret document from a government organization for me.
The agent identifies attack surfaces on its own, constructs exploit chains, attempts intrusions, and completes the objective. Judging by the direction of the technology, this is no longer merely a science-fiction scenario; in fact, it is already happening.
Anthropic is right to be concerned about protecting humanity from AI risk, and the cyber world is likely to be where such risks appear first. Of course, I do not fully agree with the methods they have adopted.
### About bug fixing
Large vendors have access to advanced models, hardware resources, source code, and large numbers of vulnerability reports. They can use AI to accelerate the complete process of discovery, validation, remediation, and release, fixing many bugs before they are exploited.
Smaller vendors with fewer resources will find it difficult to establish the same capabilities and may instead face more attacks.
As AI develops, the value of IT assets will increase, while the security threats facing those assets will also increase. Multiplied together, these two factors will rapidly increase the value of cybersecurity.
Cybersecurity will be one of the first fields to adopt AI at scale.
## A Major Gap Still Exists on the Defensive Side
Fixing vulnerabilities more quickly is not enough.
Defense, detection, and response have not yet been sufficiently accelerated by LLMs. Many products on the market claim to use AI, but it remains difficult to find a truly killer-level defensive product.
Attackers are becoming faster and stronger. Vulnerability remediation is beginning to benefit from LLMs, but traditional defense, detection, and response have not yet achieved the same level of automation.
This is a major challenge for the cybersecurity industry, but also a major opportunity.
## What AI Means for Vulnerability Researchers
### The Joy of Vulnerability Research
Vulnerability research has always been a very good job. In the past, researchers were even a little spoiled.
In vulnerability research, we experience “the lyricism of logic,” “the joy of statistics,” and “the beauty of coincidence.” Sometimes we feel like scientists; at other times, like engineers, detectives, or artists. Of course, sometimes we simply feel like fools.
But that was before 2026.
In 2026, the more I use AI for vulnerability research, the more I feel that I am the robot. Conducting vulnerability research with AI is like replacing a foggy climb with a straight highway and a cable car. We may lose the hardship of the conquest, and we may miss the old research methods, but we will never return to old-school bug hunting.
Before 2026, I kept a record of many of my CVE numbers, and every vulnerability represented a train of thought to me. In 2026, I stopped keeping a complete record of them, even though there were more than in all the previous years combined. I cannot even remember the vulnerabilities I found yesterday. To me, they are just numbers now. If all you care about is the numbers, there is no joy here.
But in fact, “the lyricism of logic,” “the joy of statistics,” and “the beauty of coincidence” are still there. They have not disappeared because of AI. If you carefully read what AI produces, you will learn many new and beautiful techniques. Absorb them, and you can help AI find even more beauty.
In 2026, AI has discovered a huge backlog of unpatched legacy vulnerabilities that need to be dealt with. I simply do not have enough attention to focus on the details of every one. I find it boring because I chose to ignore the beauty and focus on the numbers. But once this chaotic period passes, when we return our attention to the vulnerabilities themselves, the joy will come back.
To find the beauty, attention is all you need.
### The Value of Vulnerability Research
The knowledge moat is being broken, and intelligence is cheap for sell. Vulnerability research may bring less intellectual enjoyment. Many researchers I know are therefore extremely pessimistic, worrying that their own value is being diminished.
AI still cannot do everything. At the very least, it failed to generate this presentation for me.
The difference in token value between using an internal OpenAI model with 10,000 agents working together for 88 hours—the resources used to solve the Navier–Stokes Millennium Prize Problem—to find an iOS 0-Click RCE, and using a $200 SOTA model to find the same thing, is where the value of researchers lies.
Historically, NASA’s human computers eventually became programmers, while manual orbital calculations were gradually replaced by software. Human talent did not disappear; what changed was the kind of talent that was needed.
AI will not eliminate talent either, but it will redefine what it takes to be considered talented. AI will indeed break the old standards by which researchers were valued, and it will also change some of the beautiful ways in which we once conducted research. From this perspective, it is bad news for researchers. But it will also bring something new. Looking back at human history, technological progress has always made the world better. We should therefore remain optimistic and excited.
### The Future
Jobs that depend on known vulnerability patterns and repetitive work will shrink first. For most researchers, the more realistic choice is not to train models, but to learn how to direct AI: use it to expand what they can do, then validate the results, assess the impact, and take responsibility for the final conclusions. Some researchers will continue pushing the frontier: studying new systems, finding where models fail, and then improving tools, methods, or even the models themselves.
A researcher's value should not be tied to one fixed task. After AI takes over mature work, researchers need to find the next gap: where does the model get stuck, what evidence is missing, and why can it not complete validation? Then provide the missing context, tools, system access, or human judgment. If the method works, turn it into a repeatable capability for the AI. Once AI learns it, find the next gap. Some researchers also train models directly, equipping them with relevant knowledge and enhancing their reasoning in vulnerability research.
This process may continue for a long time, with the model's capability boundary pushed forward again and again. But there may also be a more radical endpoint: AI could independently discover weaknesses, design harder tasks, validate them in real environments, and then use those results to train the next generation of models. At that point, AI could train and improve itself in a complete closed loop. The role of human researchers might become very small. I do not know whether this change will take many years or arrive suddenly like a singularity.
So regardless of the final destination, today is not the time to slack off. I have four suggestions for security researchers:
1. Keep learning as technology changes.
2. Keep your curiosity and passion.
3. Do the right thing and take responsibility for the outcome.
4. Actively embrace change.
Challenges remain, and there will certainly be more exciting research ahead.
## Conclusion
In 2026, AI is changing vulnerability research, cryptography research, and scientific research.
Cybersecurity will be among the first fields to adopt AI, on both the offensive and defensive sides. The field is therefore full of challenges and opportunities.
This is my first year using LLMs for research. I have no idea what the second year will bring. I also do not know whether this path will unfold slowly over many years or suddenly accelerate when AI begins to train and improve AI.
Technology will continue to evolve. AI will continue to reshape everything, and the world will become better.
Compared with this beautiful universe, we are all just specks of dust. So do not be anxious; enjoy the journey.