LongCache

Semantic Similarity Cache for LLM & RAG Pipelines

“What is your refund policy?” costs $0.003 the first time—it should cost $0.0000 every time after that. Unlike string-matching caches, LongCache uses local BGE embeddings and FAISS to recognize paraphrased queries, verifies tenant and corpus scope to prevent cache poisoning, and serves cached answers in milliseconds.

$pip install longcache

View on GitHub Documentation Build With Us

cache_rag.py

from longcache import LongCache, RequestContext, RequestScope

context = RequestContext(

scope=RequestScope(tenant_id="acme", user_id="u1"),

operation="support.answer", model="gpt-4o",

prompt_version="v1", corpus_version="kb-v1",

)

cache = LongCache(context=context)

@cache.cached

def ask_rag(query: str) -> str:

return expensive_llm_call(query)

ask_rag("What is your refund policy?") # MISS → LLM

ask_rag("How do I get a refund?") # HIT → 3ms

Eliminate repetitive token costs without cache poisoning

Local semantic embeddings, strict multi-tenant request scoping, and singleflight concurrency control.

Semantic Similarity Matching

Local BAAI/bge-small-en-v1.5 embeddings + FAISS Flat/HNSW search recognize paraphrased queries and serve answers in milliseconds.

Immutable Scoped Identity

Composite SHA-256 keys isolate tenant, user, provider, model, prompt version, and corpus version to eliminate cross-tenant cache poisoning.

Sync & Async Singleflight

Coalesces identical concurrent in-flight requests via `get_or_compute`, `aget_or_compute`, and `@cache.cached` to prevent LLM stampedes.

SQLite, Redis & Memory Backends

Zero-dependency SQLite WAL mode by default, durable Redis response storage, multi-process Shared Redis mode, and In-Memory for CI/CD.

Tag Invalidation & Rebuild-Safe FAISS

Invalidate by tag within scoped contexts, override per-entry TTLs, and automatically rebuild FAISS indexes after sweeps and evictions.

CLI, FastAPI & SDK Adapters

Includes `@cache.cached`, OpenAI SDK replay, FastAPI middleware, LangChain/LangGraph/LlamaIndex wrappers, and the `longcache` CLI.

Backends & Ecosystem

Storage Backends

SQLite (WAL mode, zero services)
Redis (durable response storage + TTL)
Shared Redis Mode (multi-process)
In-Memory (zero I/O for unit tests)

Interfaces & Integrations

@cache.cached sync/async decorator
OpenAI SDK & dict request replay
FastAPI chat-completion middleware
LangChain, LangGraph & LlamaIndex

Reliability & Safety

Composite SHA-256 tenant/corpus scope
Sync & async Singleflight coalescing
Tag invalidation & background sweeper
CLI: stats, clear, inspect, export

Part of the Long Suite

LongCache is the performance & caching layer of the complete RAG ecosystem

The Long Suite covers the full production AI lifecycle. LongParser prepares your documents. LongTrainer builds the chatbot. LongTracer verifies the answers. LongProbe tests the retrieval. LongGuard protects runtime agent loops. LongCache eliminates repetitive token costs.

View the Long Suite ecosystem

Common questions

Traditional string-matching caches only hit when two users type the exact same character-for-character query. LongCache embeds queries locally using `BAAI/bge-small-en-v1.5` and searches a FAISS vector index so paraphrased questions—like 'What is your refund policy?' and 'How do I get a refund?'—resolve to the same verified cached answer in milliseconds without calling your LLM again.

Every cached entry is bound to an immutable composite SHA-256 `RequestContext` that isolates trusted `tenant_id` and `user_id` scope, provider, model, prompt version, execution settings, and `corpus_version`. A candidate vector match is only served if its scope and reuse policy strictly match the caller's execution context—missing identity safely bypasses the cache.

No. By default, LongCache runs 100% locally with zero external services: it embeds queries on-device with `BAAI/bge-small-en-v1.5` (offline-friendly after first model download), indexes vectors in FAISS, and persists responses in a local SQLite WAL database. When you scale out, you can switch to Redis or Shared Redis mode via `CacheConfig`.

When a cache miss occurs and 50 concurrent requests arrive for the same scoped query at the same moment, standard caches trigger 50 duplicate LLM calls (a cache stampede). LongCache's sync and async Singleflight coordinator elects a single leader to execute the origin call while followers wait and share the verified result.

LongCache gives you three ways to prevent stale RAG answers: (1) bump `corpus_version` or `prompt_version` in `RequestContext` so new document revisions automatically partition cleanly; (2) attach tags at write time and call tag-based invalidation within the scoped context; or (3) set per-entry TTLs cleaned up by the background TTL sweeper with rebuild-safe FAISS indexing.

LongCache defaults to `similarity_threshold=0.92`, which provides a balanced trade-off for general RAG pipelines. For strict FAQ lookup, use `0.95–0.99`; for broader customer support paraphrasing, start at `0.92–0.95` and monitor accepted-hit vs. miss metrics using `longcache stats` or the built-in decision metrics.

Need a high-throughput RAG system with low latency & controlled LLM costs?

We build enterprise RAG and agent systems using the full Long Suite stack. Book a free discovery call to discuss your requirements.

Book a Discovery Call View All Open Source Tools