Production RAG without the theater

Most RAG demos stop at a happy path query. Production starts when the corpus drifts, latency spikes, and someone asks why the model cited a stale doc.

If your only proof is a notebook that returned a clean answer once, you do not have retrieval. You have a screenshot of retrieval.

Chunking is not a one line default. It is a product decision: what unit of meaning should the model see, and what metadata has to travel with it so citations stay honest.

type Chunk = {
  id: string;
  sourceId: string;
  text: string;
  updatedAt: string;
  section?: string;
};

function shouldReindex(chunk: Chunk, sourceUpdatedAt: string) {
  return chunk.updatedAt < sourceUpdatedAt;
}

Carry source identity and freshness with every chunk. When the upstream doc changes, you need a path to reindex without pretending the old embedding still represents reality.

Eval sets are how you notice silent failure. A small, fixed set of queries with expected source ids beats vibes every time you change embeddings, prompts, or ranking.

type EvalCase = {
  query: string;
  mustCite: string[];
  mustNotInvent: boolean;
};

const cases: EvalCase[] = [
  {
    query: "What is the refund window?",
    mustCite: ["policy/refunds.md"],
    mustNotInvent: true,
  },
];

async function score(run: (q: string) => Promise<{ cites: string[] }>) {
  let pass = 0;
  for (const c of cases) {
    const result = await run(c.query);
    const hit = c.mustCite.every((id) => result.cites.includes(id));
    if (hit) pass += 1;
  }
  return pass / cases.length;
}

• Keep cases boring and specific to your corpus
• Fail closed when required sources are missing
• Run the set on every retrieval or prompt change

Models invent under pressure. Guardrails decide what happens when retrieval is thin: refuse, ask for clarification, or answer only inside cited spans.

type AnswerPolicy = {
  minCitations: number;
  allowUngrounded: false;
};

function canAnswer(cites: string[], policy: AnswerPolicy) {
  if (cites.length < policy.minCitations) return false;
  return policy.allowUngrounded === false;
}

A short refusal is better than a fluent wrong answer. Users trust systems that admit when the corpus does not support a claim.

The goal is not a clever prompt. It is a system your team can operate when the first version is already in users' hands: logs for latency, alerts for empty retrieval, and a cadence to refresh stale sources.