Embedded Files
Prompt caching reduces the cost and latency of LLM services by reusing previously computed prompt prefixes. However, correctly integrating prompt caching in downstream LLM agents is challenging because provider APIs and caching semantics differ and evolve over time.
Prompt-caching mis-integrations are application-level defects that break cache reuse, produce incorrect cache accounting, or trigger runtime failures. We characterize 120 such defects across 10 open-source agent systems and develop CACHELINT, a static analysis technique that detects them in real-world agent codebases.
Overview of Our Work
Overview of Our Work
Figure 1. Overview of our empirical study, CACHELINT design, and real-world evaluation.
Key Contributions
Key Contributions
Empirical Study
Empirical Study
We conduct an empirical study of 120 prompt-caching mis-integrations from 10 popular open-source agent systems, characterizing their root causes, symptoms, and fixes.
Integration Model
Integration Model
We propose a prompt-caching integration model that pairs each caching invocation site with the preconditions required for correct cache integration.
CACHELINT
CACHELINT
We develop CACHELINT, a static analysis technique that detects prompt-caching mis-integrations and reports warnings with contextual fix suggestions.
At a Glance
At a Glance
120
120
Prompt-caching defects
10
10
Open-source agent systems
8
8
Root causes
90%
90%
Silent defects
0.86
0.86
Precision
8
8
Upstream fixes
Motivating Example: Volatile Timestamp in Cached Prefix
Motivating Example: Volatile Timestamp in Cached Prefix
PrivateGPT is an open-source retrieval-augmented generation framework. In one real prompt-caching mis-integration, the application prepended a sub-second timestamp to the system prompt so that the agent could answer time-related queries.
However, prompt caching requires the cached prefix to match byte-for-byte across requests. Since the timestamp changes on every request, the prefix never matches the previously cached prefix. The model output remains correct, but the system silently loses cache reuse and pays the full recomputation cost.
The fix is to coarsen the timestamp to day-level granularity.
Figure 2. A prompt-caching mis-integration in PrivateGPT: a sub-second timestamp at the front of the system prompt breaks the cached prefix on every call; the fix coarsens it to day granularity.
Page updated
Google Sites
Report abuse