Why My RAG System Kept Giving Wrong Answers (Until I Fixed These 5 Things)

I built my first "production" RAG system thinking it was basically done. Load documents, embed them, retrieve chunks, ask the LLM. Simple.

It worked great in the demo. Then real users started asking real questions, and it fell apart.

Here's what I actually had to fix, working with RAG systems for enterprise clients at Sazag Infotech.

Chunking was the first problem

The default advice is "just split every 500 tokens." That breaks the moment your documents have real structure, like headers and sections.

Fixed size chunks cut sentences in half and lose all context.

1from langchain.text_splitter import RecursiveCharacterTextSplitter
2
3# What breaks: fixed size, no awareness of structure
4bad_splitter = RecursiveCharacterTextSplitter(
5    chunk_size=500,
6    chunk_overlap=50
7)
8
9# What actually works: split at natural document boundaries
10good_splitter = RecursiveCharacterTextSplitter(
11    chunk_size=1000,
12    chunk_overlap=200,
13    separators=["\n## ", "\n### ", "\n\n", "\n", " "]
14)

I also started keeping the section header attached to every chunk from that section:

1def chunk_with_headers(document):
2    chunks = []
3    current_header = ""
4
5    for section in document.sections:
6        if section.is_header:
7            current_header = section.text
8        else:
9            chunk_text = f"{current_header}\n\n{section.text}"
10            chunks.append(chunk_text)
11
12    return chunks

Small change. 15% better retrieval accuracy.

Vector search alone was missing obvious answers

Here's a real failure. A user searches for "error code E-4502". Pure vector similarity search happily returns chunks about error handling in general, and completely misses the actual documentation for that specific code.

Because semantically, "error handling" and "E-4502" look kind of similar to an embedding model. But they are not the same thing at all.

The fix is combining dense retrieval (vector similarity) with sparse retrieval (plain keyword matching, BM25), and merging the results.

1from langchain.retrievers import EnsembleRetriever
2from langchain.retrievers import BM25Retriever
3
4vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})
5bm25_retriever = BM25Retriever.from_documents(documents)
6bm25_retriever.k = 10
7
8ensemble_retriever = EnsembleRetriever(
9    retrievers=[vector_retriever, bm25_retriever],
10    weights=[0.6, 0.4]
11)

This one change gave us a 25% jump in query accuracy.

Picking the vector database mattered more than I expected

We tried ChromaDB, Pinecone, and Weaviate.

ChromaDB is nice for prototyping but doesn't hold up at scale. We ended up using Pinecone for managed deployments, and Weaviate when a client needed everything on their own servers.

One thing that helped a lot everywhere: don't rely on vector similarity alone, pre-filter with metadata first.

1results = vectorstore.similarity_search(
2    query,
3    k=10,
4    filter={
5        "document_type": "technical_spec",
6        "version": {"$gte": "2.0"},
7        "department": user_department
8    }
9)

Smaller search space, better accuracy, faster too.

People don't ask perfect questions

Real users type messy, half-formed questions. If your system only handles the clean version, it will disappoint people constantly.

Two things helped. First, generating a few alternate phrasings of the same query before searching:

1def expand_query(original_query: str, llm) -> list[str]:
2    prompt = f"""Given this search query, generate 3 alternative
3    phrasings that might help find relevant information:
4
5    Query: {original_query}
6
7    Return only the alternative queries, one per line."""
8
9    alternatives = llm.invoke(prompt).split("\n")
10    return [original_query] + alternatives

Second, figuring out what the user actually wants before searching (a lookup? a how-to? troubleshooting?), so you can pick the right retrieval strategy for that.

You can't improve what you don't measure

This one sounds obvious but most teams skip it. We built a small evaluation pipeline with real test cases, and ran it every time we changed anything.

1test_cases = [
2    {
3        "query": "What is the maximum file size for uploads?",
4        "expected_answer": "50MB",
5        "relevant_doc_ids": ["doc_123", "doc_456"]
6    },
7]
8
9def evaluate_rag_system(rag_chain, test_cases):
10    results = []
11    for case in test_cases:
12        response = rag_chain.invoke(case["query"])
13        results.append({
14            "retrieval_hit": check_retrieval(response, case),
15            "answer_correct": check_answer(response, case),
16            "latency": response.latency
17        })
18    return aggregate_metrics(results)

We tracked retrieval precision, answer correctness, whether the answer was actually grounded in the retrieved text, and P95 latency.

Where we ended up

After fixing all five of these: 40% better query accuracy, 60% fewer "I don't know" responses, most answers under 500ms, and 30% lower cost from better caching.

None of these fixes were fancy. Better chunking, hybrid search, smarter filtering, handling messy questions, and actually measuring results. Boring stuff, but it's what actually moves the needle.

FAQ

Why does my RAG system work in the demo but fail in production?

Usually it's the chunking. Fixed-size chunks that ignore document structure look fine on a small demo doc, then quietly break on real enterprise documents with headers and sections.

Should I only use vector search for RAG?

No. Pure vector search misses exact matches like error codes or IDs. Combine it with keyword search (BM25) and merge the results.

How do I know if my RAG system is actually getting better?

Build a small set of real test questions with known correct answers, and run them automatically every time you change something.

Have questions about building RAG systems? Feel free to reach out on LinkedIn or GitHub.