Embedded Files
Dataset Overview
Dataset Overview
581 repositories: GitHub repositories returned by prompt-caching-related search signals.
10 subject agent applications: top open-source agent systems selected for empirical analysis.
148,947 merged PRs: pull requests collected from the 10 subject systems.
622 candidate PRs: prompt-caching-related candidates identified by keyword search.
120 prompt-caching mis-integrations: manually confirmed defects used in the empirical study.
571 evaluation repositories: real-world agent repositories used to evaluate CACHELINT, excluding the 10 projects used to derive the detection model.
Empirical Study Findings
Empirical Study Findings
Finding 1: Prompt-caching mis-integrations are dominated by request-side defects. Among 120 defects, 97 are request-side and 23 are read-side. Provider compatibility and prefix stability are the two major sources of request-side mis-integrations.
Finding 2: Most prompt-caching mis-integrations fail silently. 108 of 120 defects raise no error and do not change the model output. Instead, they surface only as increased cost, added latency, or incorrect cache-usage reporting.
Finding 3: Fixes are usually simple and patterned. The 8 root causes collapse into 6 common fixing strategies, such as capability lookup, marker guard, cache-boundary fix, prefix canonicalization, lifecycle guard, and usage-counter fix.
How CACHELINT Works
How CACHELINT Works
CACHELINT models each mis-integration as a caching invocation site paired with a precondition that must hold — then checks it in the code.
Three preconditions: STAB (prefix stays byte-identical) · CAP (provider supports caching for the model) · ACCT (cache tokens counted correctly).
Two checkers. STAB and ACCT go to an LLM guard check that flags a caching-condition violation; CAP goes to a dated capability oracle that checks whether the provider's capability is satisfied. Either check raises a warning when the precondition does not hold.
Why a dated oracle? Provider capability changes over time — e.g., Bedrock's new 1-hour TTLs can make previously-correct code silently defective — so no static rule or frozen LLM can decide it. (Remove the oracle and precision drops 0.86 → 0.05.)
CACHELINT Results
CACHELINT Results
CACHELINT detects prompt-caching mis-integrations by locating prompt-caching invocation sites and checking whether their required integration preconditions hold. It combines CodeQL-based site location, a dated provider-capability oracle, and an LLM-based guard check for prefix-stability and accounting conditions.
Figure 4. CACHELINT’s findings on 33 real-world projects, placed by code size (KLOC) and popularity (GitHub stars).
Key Results
Key Results
32 true positives
5 false positives
33 affected projects
0.86 precision
16 confirmed reports
8 upstream fixes
Ablation Study
Ablation Study
We ablate the two key components of CACHELINT: the LLM-based guard check and the dated provider-capability oracle.
CACHELINT (full)
32 TP · 5 FP · Precision 0.86
w/o guard check
18 TP · 41 FP · Precision 0.31
w/o oracle
17 TP · 349 FP · Precision 0.05
Without the guard check, precision drops from 0.86 to 0.31. Without the dated oracle, precision drops to 0.05.
Page updated
Google Sites
Report abuse