RAG Patterns That Actually Work in Production

Naive RAG Is Dead

Here’s the pattern most tutorials still teach: chunk your documents, embed them into a vector database, run a similarity search when a user asks a question, shove the top 5 results into a prompt, and pray the LLM generates something useful.

That’s naive RAG. And in 2026, it fails in production more often than it succeeds. The retrieval misses relevant context, the chunks are too small or too large, the ranking is wrong, and the generated answers hallucinate because the model received garbage context.

This article covers the RAG patterns that actually hold up under real workloads — hybrid search, re-ranking, semantic chunking, and agentic retrieval. If you’re building anything that connects an LLM to your own data, these are the techniques that separate a demo from a product.

Why Naive RAG Fails

Before fixing the problem, you need to understand where naive RAG breaks down. The failure modes are predictable:

The Embedding Gap

Vector embeddings capture semantic similarity, but they’re terrible at exact matches. Ask “What is error code E-4012?” and the embedding model might return chunks about error handling in general, completely missing the specific error code. This is the exact-match gap, and it’s the single most common failure in production RAG systems.

The Chunking Problem

Fixed-size chunking (500 tokens, 1000 tokens, whatever) splits documents at arbitrary points. A paragraph explaining a concept gets cut in half. The first chunk has the setup, the second has the conclusion, and neither makes sense alone. The LLM gets half-context and generates half-answers.

The Ranking Problem

Bi-encoder models (used for vector search) are fast but imprecise. They encode the query and document independently, then compare vectors. This works for broad similarity but fails for nuanced relevance. Two documents might be equally “close” in vector space, but only one actually answers the question.

Pattern 1: Hybrid Search

This is the highest-ROI improvement you can make. Combine dense vector search (semantic) with sparse lexical search (keyword-based, like BM25). Run both in parallel, then merge the results.

// Pseudocode: hybrid search pipeline
async function hybridSearch(query, topK = 20) {
  // Run both searches in parallel
  const [vectorResults, lexicalResults] = await Promise.all([
    vectorStore.similaritySearch(query, topK),
    bm25Index.search(query, topK)
  ]);

  // Reciprocal Rank Fusion to merge results
  const merged = reciprocalRankFusion(vectorResults, lexicalResults, {
    k: 60  // RRF constant — 60 is a good default
  });

  return merged.slice(0, topK);
}

function reciprocalRankFusion(listA, listB, { k }) {
  const scores = new Map();

  for (const [rank, doc] of listA.entries()) {
    const id = doc.id;
    scores.set(id, (scores.get(id) || 0) + 1 / (k + rank + 1));
  }
  for (const [rank, doc] of listB.entries()) {
    const id = doc.id;
    scores.set(id, (scores.get(id) || 0) + 1 / (k + rank + 1));
  }

  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => findDocById(id));
}

The lexical search catches exact terms the embedding model misses. The vector search catches semantic matches the keyword search misses. Together, they cover each other’s blind spots.

Most vector databases now support hybrid search natively — Weaviate, Qdrant, and even pgvector with a BM25 extension. You don’t need to build this from scratch.

Pattern 2: Cross-Encoder Re-ranking

Bi-encoders are fast because they encode queries and documents separately. But this independence means they can’t capture the fine-grained interaction between a query and a specific passage.

A cross-encoder takes the query and a candidate document together as input and produces a single relevance score. It’s dramatically more accurate — but too slow to run on your entire corpus.

The pattern: use your bi-encoder (vector search) to quickly retrieve the top 50–100 candidates, then re-rank that short list with a cross-encoder to get the actual top 5–10.

// Re-ranking with a cross-encoder
async function retrieveAndRerank(query) {
  // Step 1: Fast retrieval (broad net)
  const candidates = await hybridSearch(query, 50);

  // Step 2: Re-rank with a cross-encoder
  const reranked = await reranker.rank(query, candidates, {
    topN: 5,
    model: 'cross-encoder/ms-marco-MiniLM-L-12-v2'
  });

  return reranked;
}

I recommend Cohere Rerank or an open-source cross-encoder like the ms-marco models from HuggingFace. The accuracy improvement from adding a re-ranker is usually larger than upgrading your embedding model — and it’s a lot cheaper.

Pattern 3: Semantic Chunking

Forget fixed-size chunks. Instead, split your documents based on semantic boundaries — where the topic actually shifts.

The approach: embed each sentence or paragraph, then compare adjacent embeddings. When the cosine similarity between consecutive chunks drops below a threshold, that’s a natural breakpoint.

// Semantic chunking logic
function semanticChunk(paragraphs, threshold = 0.75) {
  const embeddings = paragraphs.map(p => embed(p));
  const chunks = [];
  let currentChunk = [paragraphs[0]];

  for (let i = 1; i < paragraphs.length; i++) {
    const similarity = cosineSimilarity(
      embeddings[i - 1],
      embeddings[i]
    );

    if (similarity < threshold) {
      // Topic shift detected — start a new chunk
      chunks.push(currentChunk.join('\n'));
      currentChunk = [paragraphs[i]];
    } else {
      currentChunk.push(paragraphs[i]);
    }
  }

  chunks.push(currentChunk.join('\n'));
  return chunks;
}

An alternative approach that works surprisingly well: the parent-document strategy. Embed small chunks (1–2 paragraphs) for precise retrieval, but when you retrieve a match, pass the entire parent section or document to the LLM. This gives you granular search with broad context.

Pattern 4: Query Engineering

Your users write terrible search queries. That’s not an insult — it’s a fact of search behavior. Someone types “how to fix the error” when they mean “TypeError: Cannot read property ‘map’ of undefined in React 19 server components.”

Query engineering rewrites the user’s query before retrieval:

Query Decomposition

Break complex questions into sub-queries. “Compare the performance and DX of React Server Components vs Astro Islands” becomes three searches: (1) React Server Components performance, (2) Astro Islands performance, (3) developer experience comparison between RSC and Astro.

Hypothetical Document Embeddings (HyDE)

Instead of embedding the raw query, ask the LLM to generate a hypothetical answer, then embed that answer and use it for retrieval. The intuition: a hypothetical answer looks more like the actual document than a short question does, so it retrieves better matches.

async function hydeSearch(userQuery) {
  // Step 1: Generate a hypothetical answer
  const hypothetical = await llm.generate(
    `Write a short paragraph that would answer this question: ${userQuery}`
  );

  // Step 2: Embed the hypothetical answer (not the question)
  const embedding = await embed(hypothetical);

  // Step 3: Search using the hypothetical embedding
  return vectorStore.searchByVector(embedding, { topK: 10 });
}

HyDE sounds hacky, but it consistently improves retrieval precision by 10–25% in benchmarks. The cost is one extra LLM call per query, which is usually acceptable.

Pattern 5: Agentic RAG

Sometimes a single retrieval pass isn’t enough. Complex questions require multiple searches, filtering, and reasoning about whether the retrieved context is sufficient.

In agentic RAG, the LLM acts as an orchestrator: it decides what to search for, evaluates the results, and decides whether to search again or answer.

// Simplified agentic RAG loop
async function agenticRAG(query, maxSteps = 5) {
  let context = [];
  let step = 0;

  while (step < maxSteps) {
    const decision = await llm.generate({
      system: `You are a research assistant. Given a question and 
               any context gathered so far, decide whether to:
               1. SEARCH for more information (provide a search query)
               2. ANSWER the question with the current context`,
      messages: [
        { role: 'user', content: query },
        { role: 'assistant', content: `Context so far: ${JSON.stringify(context)}` }
      ]
    });

    if (decision.action === 'ANSWER') {
      return decision.answer;
    }

    // Perform the search and add results to context
    const results = await retrieveAndRerank(decision.searchQuery);
    context.push(...results);
    step++;
  }

  // Fallback: answer with whatever context we have
  return generateAnswer(query, context);
}

Agentic RAG is powerful but expensive. Reserve it for queries where a single retrieval genuinely isn’t sufficient — multi-hop questions, comparative analysis, or questions that span multiple document collections.

Common Mistakes in Production RAG

1. Skipping Evaluation

The biggest mistake. Teams build a RAG pipeline, eyeball a few queries, and ship it. Without systematic evaluation — measuring retrieval precision, answer accuracy, and hallucination rate — you have no idea if your changes actually improve things. Set up an eval framework before you start optimizing.

2. Over-Chunking

Chunks that are too small lose context. Chunks that are too large dilute relevance. There’s no universal “right” size — it depends on your documents. Test different sizes and measure retrieval quality.

3. Ignoring Metadata Filtering

Vector similarity alone isn’t enough. If a user asks about “Python error handling,” you should filter by language or documentation type before the vector search, not after. Pre-filtering reduces the search space and improves precision.

4. Not Tracking Source Attribution

Every answer should be traceable back to specific source chunks. This isn’t just for user trust — it’s essential for debugging. When the system generates a wrong answer, you need to know: was it a retrieval failure (wrong chunks) or a generation failure (right chunks, wrong synthesis)?

Best Practices

• Start simple. Implement hybrid search + re-ranking before trying agentic RAG or GraphRAG. These two improvements alone fix 70–80% of retrieval failures.
• Use pgvector for most projects. You probably don’t need a dedicated vector database. PostgreSQL with pgvector handles millions of vectors well and integrates with your existing infrastructure.
• Evaluate continuously. Build a test set of 50–100 question-answer pairs. Run your pipeline against them after every change. Automate this in CI.
• Cache embedding calls. Embedding the same document twice is wasted money. Use content-hash-based caching for your embedding pipeline.
• Keep chunks retrievable by humans too. If you can’t read a chunk and understand what it’s about, the LLM won’t either. Chunks should be self-contained enough to make sense in isolation.

Wrapping Up

The RAG ecosystem in 2026 has matured significantly. The days of dumping documents into a vector store and hoping for the best are over. Production RAG requires treating retrieval as a first-class information retrieval problem: hybrid search for coverage, re-ranking for precision, semantic chunking for context quality, and evaluation to prove it all works.

Start with hybrid search and a re-ranker. That’s your biggest bang for the buck. Add query engineering and agentic patterns only when you’ve measured specific failure modes that justify the complexity.

FAQ

What’s the difference between RAG and fine-tuning?

RAG retrieves external knowledge at query time — the model’s weights don’t change. Fine-tuning modifies the model’s weights to internalize new knowledge or behavior. Use RAG when your data changes frequently (docs, knowledge bases). Use fine-tuning when you need to change the model’s style, tone, or reasoning patterns.

How many chunks should I feed into the LLM context?

Fewer than you think. 3–5 highly relevant chunks typically outperform 10–15 loosely relevant ones. The LLM gets confused by irrelevant context (the “lost in the middle” problem). Quality over quantity, always.

Should I use a dedicated vector database or pgvector?

For most applications under 10 million vectors, pgvector is sufficient and simpler to operate. Move to a dedicated vector database (Qdrant, Weaviate, Pinecone) when you need advanced filtering, multi-tenancy at scale, or sub-millisecond latency on very large datasets.

How do I handle documents that change frequently?

Implement incremental re-indexing. Track document hashes, and only re-embed chunks that have actually changed. For real-time requirements, use Change Data Capture (CDC) to trigger re-indexing on database updates.

What embedding model should I use?

In 2026, OpenAI’s text-embedding-3-large and Cohere’s embed-v4 are the most popular commercial options. For open-source, BGE-large and GTE-large perform well. The choice matters less than your chunking and retrieval pipeline — a mediocre embedding model with great retrieval beats a great embedding model with naive retrieval.