Case Studies of Prompt-Caching Mis-Integration
Case Studies of Prompt-Caching Mis-Integration
Prompt-caching mis-integrations are hard to notice: the model output often stays correct while the system silently loses cache reuse, disables provider-specific caching, or reports incorrect cache usage. We highlight three representative patterns from real-world LLM agent applications.
Before: volatile timestamp in cached prefix
ts = datetime.now().isoformat()
sys = (
f"Today is {ts}\n" +
system_guidelines + context
)
# The timestamp changes on every request,
# so the cached prefix is never byte-identical.
After: stable day-level timestamp
day = datetime.now().date().isoformat()
sys = (
f"Today is {day}\n" +
system_guidelines + context
)
# The prefix stays stable during the day,
# enabling cache hits.
Listing 1. Simplified pattern: a sub-second timestamp in the cached prefix breaks byte-identical prefix matching; coarsening it to day granularity restores cache reuse.
A per-request timestamp is embedded in the cached system prompt. Because prompt caching requires the prefix to match byte-for-byte, the timestamp changes on every request and prevents cache hits. The model output remains correct, but the cache silently misses.
Before:
if model.id.startswith("claude") or "anthropic" in model.id:
block["cache_control"] = {"type": "ephemeral"}
After:
if registry.supports_cache(model.id):
block["cache_control"] = {"type": "ephemeral"}
Listing 2. Simplified pattern: hardcoded model-name checks should be replaced with capability-based cache support checks.
Some agents enable prompt caching only when the model name matches a hardcoded string such as "claude" or "anthropic". When providers rename models or introduce new cacheable models, the gate may silently exclude models that should support caching.
Before:
total_cached = usage["cache_read"] + usage["cache_creation"]
After:
total_cached = usage.get("cache_read", 0) + usage.get("cache_creation", 0)
Listing 3. Simplified pattern: cache-token usage fields should be guarded before aggregation.
In response-side defects, prompt caching works correctly, but the agent parses, sums, prices, or displays the returned cache-token counts incorrectly. Developers may believe caching is disabled or ineffective even when the provider is returning cache usage information.
Fix: guard optional cache-token fields and aggregate cache usage correctly.