Building a RAG Pipeline, Step by Step

Retrieval-Augmented Generation (RAG) exists because large language models are confidently wrong about anything they were not trained on — your internal docs, last week's release notes, a customer's contract. RAG fixes this by fetching relevant text from your data at query time and handing it to the model as context, so answers are grounded in facts you control instead of hallucinated from a frozen training set. Done well, it turns a generic chatbot into something that can actually answer questions about your world.

This is the pattern behind most "chat with your docs" products, internal knowledge assistants, and support copilots. It is also one of the most over-hyped corners of the LLM world, so this guide keeps the marketing out and walks through what each piece actually does, where it breaks, and how to choose a vector store without falling for a leaderboard.

What RAG is, and why it beats just fine-tuning

The core idea is small: instead of asking the model to remember your data, you let it look it up. At query time you find the most relevant chunks of your own corpus, paste them into the prompt, and ask the model to answer using only that context.

People often reach for fine-tuning first, and for the wrong reasons. Fine-tuning is good at teaching a model a style, a format, or a narrow skill. It is poor at injecting facts, because facts change, and retraining on every document edit is slow and expensive. RAG is the opposite: it is excellent at facts because the knowledge lives outside the model in an index you can update in seconds, and it gives you something fine-tuning never will — citations. When the model answers, you know exactly which source chunks it drew from.

A useful mental model: fine-tuning changes how the model talks; RAG changes what it knows right now. Most real systems that need current, private, or frequently-changing information start with RAG and only add fine-tuning later for tone or structure.

There are two halves to any RAG system, and it helps to keep them separate in your head:

• Indexing (offline): turn documents into searchable vectors, done ahead of time.
• Retrieval + generation (online): for each user query, find relevant chunks and generate a grounded answer.

The indexing pipeline: chunk, embed, store

This is the part you build once and run whenever your data changes. Get it wrong and no amount of clever prompting at query time will save you — retrieval can only return what you indexed sensibly.

Chunking: the most underrated decision

Embedding models and context windows both have limits, so you cannot embed a 90-page PDF as one vector. You split documents into chunks — passages small enough to be specific, but large enough to carry meaning.

Naive fixed-size splitting (every 1000 characters) is a common starting point and a common source of pain: it slices sentences in half and separates a heading from the paragraph it describes. Better strategies respect structure — split on headings, paragraphs, or Markdown sections first, then fall back to size limits. Adding a small overlap between chunks keeps context from being lost at the seams.

def chunk_text(text, size=800, overlap=120):
    # split on paragraph boundaries first, then pack
    paras = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for p in paras:
        if len(current) + len(p) <= size:
            current += "\n\n" + p
        else:
            chunks.append(current.strip())
            current = current[-overlap:] + "\n\n" + p
    if current:
        chunks.append(current.strip())
    return chunks

Two rules of thumb that hold up in practice: chunks of roughly 300–800 tokens work for most prose, and you should always store metadata alongside each chunk — source filename, section title, URL, last-updated date. That metadata is what makes filtering and citations possible later.

Embeddings: turning text into vectors

An embedding model maps a chunk of text to a fixed-length vector (say 768 or 1536 numbers) such that semantically similar text lands close together in vector space. This is what lets "How do I reset my password?" match a passage titled "Account recovery steps" even with zero shared keywords.

# Pseudocode — exact client/method names vary by provider
def embed(texts: list[str]) -> list[list[float]]:
    response = embedding_client.create(
        model="your-embedding-model",
        input=texts,
    )
    return [item.vector for item in response.data]

vectors = embed(chunk_text(document))

A few things that matter more than picking the "best" model:

• Use the same embedding model for indexing and querying. Mixing models produces vectors that are not comparable, and retrieval quietly returns garbage.
• Match the model to your language. For Indian-context apps mixing English with Hindi, Tamil, or transliterated text, prefer a multilingual embedding model and test it on your own samples before committing.
• Batch your calls. Embedding is the slow, paid step of indexing; sending chunks in batches saves real time and cost.

Storing vectors: the vector store

Once you have vectors, you need somewhere to store them and search by similarity — typically cosine similarity or dot product — fast, even across millions of entries. That is what a vector store does. It also, in good implementations, supports metadata filtering so you can say "search only documents from this team, updated this year."

Choosing a vector store without the hype

There is no single best vector store — there is the one that fits your scale, your team, and your existing stack. The honest framing: if you already run Postgres, start with pgvector; reach for something heavier only when you can name the limit you hit.

Read that table as trade-offs, not winners. pgvector keeps your data in one place and lets you JOIN against business tables — a huge operational win — but you manage tuning yourself. FAISS is a fast, dependency-light library, not a database; you handle persistence, updates, and serving. A managed service removes ops burden and scales smoothly, at the cost of money and a vendor dependency. For most teams learning RAG, pgvector or FAISS is the right place to start, and you should only migrate when you have a concrete bottleneck to point at.

The retrieval and generation path

This is the online flow that runs on every user query. The steps are simple individually; the quality comes from how carefully you wire them together.

1. Embed the query. Same model as indexing. The user's question becomes a vector in the same space as your chunks.

2. Search. Ask the vector store for the top-k most similar chunks. Apply metadata filters here — restrict by source, date, or access permissions.

def retrieve(query, k=20, filters=None):
    qvec = embed([query])[0]
    hits = vector_store.search(
        vector=qvec,
        top_k=k,
        filter=filters,   # e.g. {"team": "support"}
    )
    return [(h.text, h.metadata, h.score) for h in hits]

3. Rerank. This is the step beginners skip and then wonder why answers are mediocre. Initial vector search is fast but approximate — it casts a wide net. A reranker (a cross-encoder model that reads the query and each chunk together) reorders those candidates by true relevance. The standard pattern: retrieve 20–50 candidates cheaply, rerank, keep the top 4–6 for the prompt.

def rerank(query, candidates, keep=5):
    scored = reranker.score(
        query=query,
        documents=[c[0] for c in candidates],
    )
    ranked = sorted(zip(scored, candidates), reverse=True)
    return [c for _, c in ranked[:keep]]

4. Augment the prompt. Build the final prompt: a clear instruction, the retrieved context, and the question. Be explicit that the model should answer from the context and say so when the context does not contain the answer — this single instruction is one of your strongest defenses against hallucination.

def build_prompt(query, chunks):
    context = "\n\n---\n\n".join(
        f"[{c[1].get('source')}]\n{c[0]}" for c in chunks
    )
    return (
        "Answer using ONLY the context below. "
        "If the answer is not present, say you do not know.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}"
    )

5. Generate. Send the prompt to the LLM and return the answer — ideally with the source filenames from your metadata so users can verify. Those citations are RAG's killer feature; surface them.

Evaluating RAG: don't ship on vibes

"It looked good in three test queries" is how RAG systems quietly fail in production. Because the pipeline has two stages, you evaluate them separately.

Retrieval quality is measured independently of the LLM. Build a small set of question → expected-source pairs (even 30–50 hand-labelled examples is a strong start) and measure:

• Recall@k — did the correct chunk appear in the top-k results at all? If not, generation cannot possibly be right.
• Precision / Hit rate — how much of what you retrieved was actually relevant.

Generation quality is measured on the final answer:

• Faithfulness / groundedness — is every claim supported by the retrieved context, or did the model invent things?
• Answer relevance — did it actually address the question?

You can grade faithfulness with a careful "LLM-as-judge" setup, but always anchor it with a human-labelled sample so you trust the judge. The discipline that matters: change one variable at a time (chunk size, k, reranker on/off) and re-run the same eval set. Without a fixed eval set you are tuning blind.

Common failure modes and their fixes

Almost every struggling RAG system is failing in one of a handful of predictable ways. Diagnose before you tweak.

The two that bite hardest are bad chunking (because it corrupts everything downstream) and no reranking (because raw vector search is just not precise enough on its own). Fix those two before reaching for anything fancier.

Build one end to end

Reading about RAG and building one are different skills — the gap is in the unglamorous details: a chunker that respects your real documents, a reranker that earns its latency, an eval set that tells you whether a change helped. The Datainteg platform includes a guided RAG project that walks you through exactly this pipeline — ingestion, indexing, retrieval, reranking, and evaluation — on a real corpus, so the moving parts above stop being abstract. It is a practical way to turn this article into a system you have actually shipped and measured.

Key takeaways

• RAG grounds LLMs in your own data at query time, beating fine-tuning for facts that are private, current, or frequently changing — and it gives you citations.
• Two pipelines: offline indexing (chunk → embed → store) and online serving (embed query → retrieve → rerank → augment → generate).
• Chunking is the highest-leverage decision. Split on structure, add overlap, store rich metadata for filtering and citations.
• Use the same embedding model everywhere, and pick one that matches your language mix.
• Choose a vector store by fit, not hype: pgvector if you already run Postgres, FAISS for control, a managed service to skip ops. Migrate only when you hit a named limit.
• Reranking is not optional for good quality — retrieve wide, rerank, keep a few.
• Evaluate retrieval and generation separately with a fixed labelled set, and change one variable at a time.
• Most failures are predictable — bad chunking, no reranking, stale indexes, lost-in-the-middle — and each has a known fix.