RAG Chunking Strategy: Fix Retrieval by Splitting on Sentences

Why chunking determines retrieval quality

RAG retrieval is only as good as the chunks it searches. If a document is sliced into 25-character windows, the index stores fragments like "re retried three t" - no embedding model can make that semantically meaningful. Cosine similarity will find something, but the returned passages won't answer the user's question.

The retrieval and ranking code is irrelevant; garbage in, garbage out.

The broken pattern: fixed-width character slices

def chunk(text):
    # BAD: cuts sentences and words apart at arbitrary positions
    size = 25
    return [text[i:i+size] for i in range(0, len(text), size)]

A document like "User authentication is handled by issuing a signed JWT on login." becomes:

"User authentication is h"
"andled by issuing a sign"
"ed JWT on login."

Each chunk is syntactically broken. TF-IDF and embedding models both suffer.

The fix: split on sentence boundaries

import re

def chunk(text):
    # Split after sentence-ending punctuation followed by whitespace
    parts = re.split(r"(?<=[.!?])\s+", text.strip())
    return [p.strip() for p in parts if p.strip()] or [text.strip()]

This keeps each chunk a complete, self-contained thought. The same document now becomes:

"User authentication is handled by issuing a signed JWT on login."
"The token expires after 30 minutes and must be refreshed."

Both chunks are retrievable on their own terms - one answers authentication questions, the other answers expiry/refresh questions.

Plugging it into the pipeline

The index() and retrieve() functions don't change - only chunk() does:

def index(corpus):
    chunks = [c for doc in corpus for c in chunk(doc)]
    vectorizer = TfidfVectorizer().fit(chunks)
    matrix = vectorizer.transform(chunks)
    return vectorizer, matrix, chunks

def retrieve(question, vectorizer, matrix, chunks, k=2):
    qv = vectorizer.transform([question])
    sims = cosine_similarity(qv, matrix)[0]
    idxs = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in idxs]

Production chunking patterns

Sentence splitting is the right starting point. Real production RAG systems go further:

• Token-aware splitting - chunk on sentences but cap at a token budget (e.g. 256 tokens) with a 10-20% overlap so context isn't lost at boundaries. LangChain's RecursiveCharacterTextSplitter does this.
• Paragraph/heading splits - for structured docs (markdown, HTML), split on \n\n or heading tags first, then sentences within each block.
• Semantic chunking - embed sentences and merge adjacent ones whose embeddings are close; more expensive but produces the most coherent chunks.
• Layout-aware splitting - PDF libraries like pdfplumber preserve column and table structure before any text splitting happens.

The failure mode - tiny character slices silently tanking retrieval - is one of the most common RAG bugs in production. It never throws an error; you just get bad answers.