Long Suite: Open-Source Reliability Tools for Production RAG and AI Agents

Why RAG reliability needs more than a vector database

Architecture: five tools across one lifecycle

Ingest: how LongParser prepares documents for retrieval

The processing pipeline

1. Extract. Docling (with Marker as another supported backend) converts a document into structured Block objects instead of a flat text stream.
2. Validate. Each page gets a confidence score, and right-to-left text is detected so it can be handled correctly.
3. Chunk. HybridChunker builds chunks that are token-aware, aware of the heading hierarchy and aware of tables. An embedding-based semantic chunker (using all-MiniLM-L6-v2) is available when boundaries should follow meaning. Charts and figures are cross-referenced deterministically.
4. Protect. PII redaction combines regular expressions with spaCy named-entity recognition.
5. Review. Reviewers approve, edit or reject blocks and chunks through the API before embedding.
6. Embed. Approved chunks go into Chroma, FAISS or Qdrant.

Python SDK

from longparser import DocumentPipeline, ProcessingConfig

pipeline = DocumentPipeline(ProcessingConfig())
result = pipeline.process_file("document.pdf")

print(f"Pages: {result.document.metadata.total_pages}")
print(f"Chunks: {len(result.chunks)}")
print(result.chunks[0].text)

Framework adapters

from longparser.integrations.langchain import LongParserLoader

docs = LongParserLoader("report.pdf").load()  # list of LangChain Documents

from longparser.integrations.llamaindex import LongParserReader

docs = LongParserReader().load_data("report.pdf")

REST workflow for review-driven ingestion

Document chat and conversational retrieval

Build: LongTrainer for multi-tenant RAG assistants

Quick start

from longtrainer.trainer import LongTrainer

trainer = LongTrainer(mongo_endpoint="mongodb://localhost:27017/")

bot_id = trainer.initialize_bot_id()
trainer.add_document_from_path("path/to/your/data.pdf", bot_id)
trainer.create_bot(bot_id)

chat_id = trainer.new_chat(bot_id)
answer, sources = trainer.get_response("What is this document about?", bot_id, chat_id)
print(answer)

Constructor options that matter in production

trainer = LongTrainer(
    mongo_endpoint="mongodb://localhost:27017/",
    llm=None,               # uses the configured default language model
    embedding_model=None,   # default: OpenAIEmbeddings()
    max_token_limit=32000,
    num_k=3,                # chunks retrieved per query
    chunk_size=2048,
    chunk_overlap=200,
    ensemble=False,         # multi-query ensemble retrieval
    encrypt_chats=False,    # Fernet encryption of stored chats
    encryption_key=None,
    enable_tracer=False,    # built-in LongTracer integration
    tracer_backend="mongo", # mongo | sqlite | memory
    tracer_threshold=0.5,
)

Multi-tenancy by design

Ingestion sources and backends

Agent mode and tools

from longtrainer.tools import web_search

trainer.create_bot(bot_id, agent_mode=True)
trainer.add_tool(web_search, bot_id)

A CLI and REST server for quick starts

Cache: LongCache reduces repeated LLM calls and RAG costs

How LongCache Works

Core Features

• Semantic caching: Uses BGE-small embeddings and FAISS to identify similar questions.
• Multi-tenant isolation: Separates cached responses using trusted tenant, user, model, prompt, and knowledge-base context.
• Singleflight execution: Combines eligible concurrent requests to avoid redundant model calls.
• Flexible storage: Supports SQLite, Redis, and in-memory backends.
• Cache invalidation: Uses TTLs, tags, and document-version changes to control stale responses.
• Framework integration: Supports OpenAI SDK, FastAPI, LangChain, LangGraph, and LlamaIndex integration patterns.

Getting Started with LongCache

pip install longcache

from longcache import LongCache, RequestContext, RequestScope

context = RequestContext(
    scope=RequestScope(
        tenant_id="acme",
        user_id="user-1"
    ),
    operation="support.answer",
    provider="openai",
    provider_scope="production",
    model="gpt-4o",
    prompt_version="v1",
    corpus_version="kb-v1"
)

cache = LongCache(context=context)

@cache.cached
def ask_rag(query: str) -> str:
    return expensive_llm_call(query)

answer = ask_rag("What is your refund policy?")

Production Considerations

Guard: LongGuard stops agent loops before they burn budget

What it detects

Reflect and Pivot instead of a hard crash

Using it in a plain OpenAI loop

from longguard import CircuitBreaker, GuardConfig, AgentStep
from openai import OpenAI
client = OpenAI()

breaker = CircuitBreaker(GuardConfig(model="gpt-4o", max_cost_usd=0.50, max_steps=30))

observation = None
for i in range(1, 31):
    response = client.chat.completions.create(model="gpt-4o", messages=messages)
    step = AgentStep.from_openai_response(response, step_number=i, observation=observation)

    decision = breaker.check(step)
    if decision.action == "kill":
        break
    elif decision.action == "reflect":
        messages.append({"role": "system", "content": decision.inject_prompt})

    if step.action is None:      # the model produced a final answer
        break
    observation = run_tool(step.action, step.action_input)

print(breaker.report.summary())

Framework integrations

Additional integrations and recovery options

Telemetry

Verify: LongTracer checks every answer against its sources

How verification works

1. Claim splitting. The response is split into individual sentences, each treated as a claim.
2. Evidence matching. A bi-encoder (all-MiniLM-L6-v2) performs semantic text similarity to find the source passage most relevant to each claim.
3. NLI verification. A cross-encoder (nli-deberta-v3-xsmall) classifies each claim against its evidence as entailment, contradiction or neutral.
4. Scoring. The results roll up into a trust_score between 0.0 and 1.0, a hallucination count and a verdict.

Three lines to a verdict

from longtracer import check

result = check(
    response="The Eiffel Tower is 330 meters tall and located in Berlin.",
    sources=[
        "The Eiffel Tower is a landmark in Paris, France. It is 330 meters tall."
    ],
)

print(result.verdict, result.trust_score, result.hallucination_count)

Instrumenting an existing pipeline

from longtracer import LongTracer, instrument_langchain

LongTracer.init(verbose=True)
instrument_langchain(chain)

Where the traces go

Observability

Test: LongProbe catches retrieval regressions before users do

Set up golden questions

name: support-docs
version: 1
questions:
  - id: refund-window
    question: "How long do customers have to request a refund?"
    match_mode: text
    required_chunks:
      - "30 days"
    top_k: 5
    tags: [billing]
  - id: sso-setup
    question: "How do I configure single sign-on?"
    match_mode: semantic
    semantic_threshold: 0.80
    required_chunks:
      - "SAML single sign-on is configured under Security settings"
    top_k: 5

Point it at your retriever

Check, baseline and diff

longprobe check --goldens goldens.yaml          # run all golden questions
longprobe baseline save --label v1              # store the current results
longprobe diff --baseline v1                    # compare a later run against it
longprobe watch --interval 60                   # keep re-checking while you work

Additional testing and dataset management commands

# Generate candidate questions from documents
longprobe generate ./docs --auto-capture

# Capture retrieval examples
longprobe capture

# List saved evaluation baselines
longprobe baseline list

In pytest

from longprobe import LongProbe
from longprobe.adapters import create_adapter

def test_retrieval_quality():
    adapter = create_adapter(
        "chroma",
        collection_name="test_docs",
        persist_directory="./test_db"
    )

    probe = LongProbe(
        adapter=adapter,
        goldens_path="tests/goldens.yaml",
        config_path="longprobe.yaml"
    )

    report = probe.run()

    assert report.overall_recall >= 0.85

In CI

name: retrieval-quality
on: [push, pull_request]
jobs:
  longprobe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install "longprobe[all]"
      - run: longprobe check --goldens goldens.yaml --output github
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Putting it together: an end-to-end workflow

1. Ingest once. LongParser extracts, validates, chunks and redacts each document. A reviewer approves flagged chunks, and the approved chunks are embedded into the vector store.
2. Serve queries. A LongTrainer bot retrieves the top num_k chunks and generates an answer, returning the sources with it.
3. Protect agents. If the bot runs in agent mode, a LongGuard circuit breaker checks every step and stops loops or runaway spend.
4. Verify the answer. LongTracer compares each claim in the response with the sources and records a trust score, a verdict and a trace.
5. Test every change. LongProbe runs in CI against golden questions, so a change to chunking, embeddings or the index cannot ship if retrieval gets worse.

from longtrainer.trainer import LongTrainer
from longtracer import check

trainer = LongTrainer(mongo_endpoint="mongodb://localhost:27017/")
bot_id = trainer.initialize_bot_id()
trainer.add_document_from_path("policy.pdf", bot_id)
trainer.create_bot(bot_id)
chat_id = trainer.new_chat(bot_id)

answer, sources = trainer.get_response("What is the refund window?", bot_id, chat_id)

source_texts = [getattr(s, "page_content", str(s)) for s in sources]
result = check(answer, source_texts)
if result.hallucination_count > 0:
    answer = "I could not verify that answer against the documents."

Where each failure gets caught

Which tool should you adopt first?

• Answers contain facts that are not in your documents: start with LongTracer. It is a few lines to add and gives you a measurable signal.
• Answers got worse after a change and nobody noticed: start with LongProbe and a small set of golden questions.
• Source documents are messy, scanned or sensitive: start with LongParser.
• You are building a multi-customer assistant from scratch: start with LongTrainer.
• Agents loop or your LLM bill is unpredictable: start with LongGuard.

Frequently asked questions

Is Long Suite one package or five?

Do I have to use LongTrainer to use the other tools?

Does LongTracer replace human review?

Is it free to use?

Get started with Long Suite