Retrieval that survives production · part 1
Chunking that survives contact with real documents
Build the ingestion half of a retrieval pipeline: structure-aware splitting, contextual headers, and a schema that lets you re-chunk without re-embedding the world.
Most RAG tutorials chunk at 512 characters with 50 characters of overlap and move on. Then you point it at a real corpus — documentation with code blocks, contracts with numbered clauses, support tickets with quoted threads — and the retrieval is bad in ways that are hard to attribute.
The reason is almost always that chunking destroyed the information the chunk needed to be findable. A fragment reading "This value must not exceed 4096" is useless without knowing what value, in what API, under what conditions.
This part builds ingestion: splitting documents so chunks stay meaningful, and storing them so you can change your mind later. Part 2 does retrieval; part 3 measures whether any of it works.
Set up the schema#
The single most important schema decision: separate documents from chunks, and make chunks disposable. Chunking strategy is something you will change three or four times. If re-chunking means re-fetching every source document, you will stop experimenting.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id bigserial PRIMARY KEY,
source_uri text NOT NULL UNIQUE,
title text NOT NULL,
content text NOT NULL, -- full original text, kept forever
metadata jsonb NOT NULL DEFAULT '{}',
content_hash text NOT NULL, -- skip re-ingest when unchanged
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
document_id bigint NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
ordinal int NOT NULL, -- position within the document
content text NOT NULL, -- what gets embedded
raw_content text NOT NULL, -- what gets shown to the user
heading_path text[] NOT NULL DEFAULT '{}',
token_count int NOT NULL,
embedding vector(1024),
strategy text NOT NULL, -- which chunker produced this
UNIQUE (document_id, strategy, ordinal)
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks (document_id);Two columns earn their place immediately.
content versus raw_content — the text you embed and the text you show are
not the same. You will prepend context to the embedded version (step 3) and you
do not want that leaking into what the user reads.
strategy lets two chunkings coexist. You can ingest with a new strategy,
evaluate it against the old one on the same corpus (part 3), and cut over only
if it wins. Without this column, comparing strategies means destroying the
thing you're comparing against.
Split on structure, not character count#
Fixed-size splitting cuts through sentences, code blocks, and table rows with equal indifference. Structure-aware splitting respects the document's own boundaries and only falls back to counting when a section is genuinely too big.
The rule: split at the largest boundary that gets you under budget. Headings first, then paragraphs, then sentences. Never mid-code-block.
import re
from dataclasses import dataclass, field
MAX_TOKENS = 512
MIN_TOKENS = 64 # below this, merge forward
OVERLAP_TOKENS = 64
@dataclass
class Section:
heading_path: list[str]
text: str
def split_by_headings(markdown: str) -> list[Section]:
"""Split on ATX headings, tracking the full heading path to each section."""
lines = markdown.split("\n")
sections: list[Section] = []
path: list[str] = []
buffer: list[str] = []
in_fence = False
def flush() -> None:
text = "\n".join(buffer).strip()
if text:
sections.append(Section(heading_path=list(path), text=text))
buffer.clear()
for line in lines:
# Never treat a '#' inside a fenced code block as a heading.
if line.lstrip().startswith("```"):
in_fence = not in_fence
match = re.match(r"^(#{1,6})\s+(.*)$", line) if not in_fence else None
if match:
flush()
level = len(match.group(1))
path = path[: level - 1] + [match.group(2).strip()]
else:
buffer.append(line)
flush()
return sectionsThe in_fence tracking is not a nicety. Markdown documentation is full of
shell examples containing # comments, and a naive heading regex will shred
every one of them into fragments.
Add the context the chunk lost#
This is the step that moves retrieval quality the most, and it is three lines of string concatenation.
A chunk pulled from the middle of a document has lost everything the reader knew from the surrounding structure. Put it back — in the text you embed, not the text you display.
Embedded without context
The maximum is 4096. Values above this are
rejected with a 400.Embedded with context
Anthropic API > Messages > Parameters > max_tokens
The maximum is 4096. Values above this are
rejected with a 400.The first chunk is a near-miss for hundreds of queries about limits in unrelated systems. The second is a strong match for "what is the max_tokens limit" and a weak match for everything else — which is exactly what you want.
def contextualize(section: Section, doc_title: str, chunk_text: str) -> str:
"""Build the embedded form: breadcrumb header + the chunk body."""
trail = " > ".join([doc_title, *section.heading_path])
return f"{trail}\n\n{chunk_text}"Some teams go further and use an LLM to write a one-sentence situating summary per chunk. It works and it measurably helps, but it costs a model call per chunk and the breadcrumb captures most of the benefit for free. Start here, measure in part 3, and only reach for generated context if the numbers say you need it.
Pack sections into chunks#
Now turn sections into chunks under a token budget. Two rules that matter more than the budget itself: merge tiny sections forward, and overlap only at sentence boundaries.
def pack(sections: list[Section], count_tokens) -> list[tuple[Section, str]]:
chunks: list[tuple[Section, str]] = []
for section in sections:
tokens = count_tokens(section.text)
# Small section: try to merge into the previous chunk from the same branch.
if tokens < MIN_TOKENS and chunks:
prev_section, prev_text = chunks[-1]
if prev_section.heading_path[:1] == section.heading_path[:1]:
merged = f"{prev_text}\n\n{section.text}"
if count_tokens(merged) <= MAX_TOKENS:
chunks[-1] = (prev_section, merged)
continue
if tokens <= MAX_TOKENS:
chunks.append((section, section.text))
continue
# Too big: split on paragraphs, then sentences, with sentence overlap.
for piece in split_with_overlap(section.text, count_tokens):
chunks.append((section, piece))
return chunks
def split_with_overlap(text: str, count_tokens) -> list[str]:
paragraphs = [p for p in re.split(r"\n\s*\n", text) if p.strip()]
out: list[str] = []
current: list[str] = []
for para in paragraphs:
candidate = current + [para]
if count_tokens("\n\n".join(candidate)) > MAX_TOKENS and current:
out.append("\n\n".join(current))
# Overlap: carry the last sentence forward, not a fixed char count.
tail = re.split(r"(?<=[.!?])\s+", current[-1])[-1]
current = [tail, para]
else:
current = candidate
if current:
out.append("\n\n".join(current))
return outCharacter-count overlap is the default in most libraries and it's wrong. It starts chunks mid-word, which pollutes the embedding with a fragment that means nothing. Overlapping by a complete sentence costs the same tokens and produces a chunk that reads as text.
Embed in batches, idempotently#
The ingest job will fail partway through. Plan for that: hash the source, skip unchanged documents, and delete-then-insert chunks per document inside a transaction so a crash never leaves a half-chunked document.
import hashlib
import psycopg
from psycopg.types.json import Jsonb
BATCH = 96
def ingest(conn: psycopg.Connection, doc: dict, strategy: str = "v1") -> int:
content_hash = hashlib.sha256(doc["content"].encode()).hexdigest()
with conn.transaction():
row = conn.execute(
"""
INSERT INTO documents (source_uri, title, content, metadata, content_hash)
VALUES (%s, %s, %s, %s, %s)
ON CONFLICT (source_uri) DO UPDATE
SET title = EXCLUDED.title,
content = EXCLUDED.content,
metadata = EXCLUDED.metadata,
content_hash = EXCLUDED.content_hash,
updated_at = now()
RETURNING id, (documents.content_hash = %s) AS unchanged
""",
(doc["source_uri"], doc["title"], doc["content"],
Jsonb(doc.get("metadata", {})), content_hash, content_hash),
).fetchone()
document_id, unchanged = row
if unchanged:
return 0
# Chunks for this document + strategy are disposable. Rebuild them.
conn.execute(
"DELETE FROM chunks WHERE document_id = %s AND strategy = %s",
(document_id, strategy),
)
sections = split_by_headings(doc["content"])
packed = pack(sections, count_tokens)
rows = []
for ordinal, (section, raw) in enumerate(packed):
rows.append((
document_id, ordinal,
contextualize(section, doc["title"], raw), # embedded form
raw, # displayed form
section.heading_path, count_tokens(raw), strategy,
))
for start in range(0, len(rows), BATCH):
batch = rows[start : start + BATCH]
vectors = embed([r[2] for r in batch]) # embed `content`
conn.executemany(
"""
INSERT INTO chunks (document_id, ordinal, content, raw_content,
heading_path, token_count, strategy, embedding)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
""",
[(*r, v) for r, v in zip(batch, vectors)],
)
return len(rows)The ON CONFLICT ... RETURNING trick compares the old hash to the new one in the
same statement that performs the upsert, so an unchanged document costs one round
trip and zero embedding calls. On a corpus that re-syncs nightly, this is the
difference between a five-minute job and a two-hour one.
Sanity-check the output before you trust it#
Do not move on to retrieval without looking at actual chunks. Every corpus has a document type that breaks your splitter, and you want to find it now.
-- Distribution: are you producing lots of tiny or oversized chunks?
SELECT width_bucket(token_count, 0, 600, 12) * 50 AS bucket, count(*)
FROM chunks WHERE strategy = 'v1'
GROUP BY 1 ORDER BY 1;
-- The suspicious tails.
SELECT id, token_count, left(raw_content, 120) FROM chunks
WHERE strategy = 'v1' AND (token_count < 32 OR token_count > 512)
ORDER BY token_count LIMIT 25;
-- Chunks that lost their breadcrumb entirely.
SELECT count(*) FROM chunks WHERE strategy = 'v1' AND heading_path = '{}';A healthy corpus has most chunks clustered between roughly 200 and 500 tokens,
few under 64, and a small heading_path = '{}' count coming only from documents
that genuinely have no headings. A spike at the very bottom means your splitter
is fragmenting something — go read ten of those chunks and you will see the
pattern immediately.
What you have now#
A corpus split on real boundaries, embedded with the context each chunk needs to be findable, stored so you can re-chunk without re-fetching, and a re-ingest path that skips unchanged documents.
What you do not have is any evidence that it retrieves well. Embedding similarity alone misses exact identifiers, rare terms, and anything where the user's phrasing doesn't match the document's — which is most technical search.
Part 2 adds keyword search alongside the vectors and fuses the two rankings. Part 3 builds the measurement that tells you whether any of these choices were correct, including the ones in this tutorial.
Filed under