LangChain RAG: A Practical Guide to Retrieval with Deep Agents

Retrieval-Augmented Generation in LangChain grounds an LLM's answers in your own documents by retrieving relevant chunks before generation. The current LangChain Deep Agents documentation, updated in early November 2026, extends the conventional retrieval chain with custom retrieval tools, subagents, skills-guided retrieval, and filesystem-backed storage. This walkthrough implements a working RAG system in Python, explains the four-stage retrieval flow, and identifies the common pitfalls that separate a prototype from a production-ready system.

Setting Up Your LangChain RAG Environment

You need a Python environment with LangChain core, a text splitter, an embedding provider, and a vector store. The exact package versions and import paths require validation against the current LangChain release notes before deployment, but the pattern remains consistent.

Install the core dependencies:

pip install langchain langchain-core langchain-openai langchain-community langchain-text-splitters faiss-cpu

The langchain-openai package provides the OpenAI embedding model wrapper. The langchain-community package includes FAISS, a local vector store suitable for prototyping and small-to-medium corpora. For production workloads spanning millions of chunks or requiring horizontal scaling, consider a managed vector database such as Pinecone, Weaviate, or Qdrant.

Set your OpenAI API key in the environment:

export OPENAI_API_KEY="sk-..."

The embedding model maps text into a high-dimensional vector space where semantic similarity corresponds to geometric proximity. The vector store indexes those embeddings and returns the chunks whose vectors are closest to a query vector. This semantic similarity search is the mechanism that makes retrieval context-aware rather than keyword-dependent.

Implementing Document Loading and Chunking

RAG begins with ingestion: load source documents, split them into chunks, generate embeddings, and store the vectors with metadata. The official guidance describes this as a four-stage sequence: load, split, embed, and store.

Load documents from a directory of Markdown files:

from langchain_community.document_loaders import DirectoryLoader, TextLoader

loader = DirectoryLoader(
    path="./docs",
    glob="**/*.md",
    loader_cls=TextLoader,
)

documents = loader.load()

Each Document object carries a page_content string and a metadata dictionary. Preserve source identifiers, URLs, headings, timestamps, and any access-control tags in the metadata. Without this information, you cannot generate verifiable citations or filter retrieval by tenant, version, or permission.

Split documents into chunks:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1_000,
    chunk_overlap=150,
)

chunks = splitter.split_documents(documents)

The RecursiveCharacterTextSplitter attempts to split on paragraph boundaries, then sentences, then whitespace, preserving semantic coherence within each chunk. A chunk size of 1,000 characters balances context completeness against the number of chunks you must retrieve. The 150-character overlap reduces the risk that a key passage is split across two chunks and lost during retrieval.

WHY these parameters matter: a chunk that is too small may lack sufficient context for the embedding model to capture its meaning. A chunk that is too large wastes context window space and dilutes the signal when multiple unrelated topics appear in the same chunk. The overlap ensures that a sentence near a boundary appears in at least one complete chunk.

Common chunking pitfalls include splitting code blocks, tables, API reference entries, or section headings across unrelated chunks. For structured documents, consider a splitter that respects Markdown headings or code fences. For API documentation, consider splitting by endpoint or method rather than by fixed character count.

Generating Embeddings and Storing Vectors

Generate embeddings and build the vector index:

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = FAISS.from_documents(chunks, embeddings)

The text-embedding-3-small model produces high-dimensional vectors, with the exact dimensionality documented in the current model reference. Each chunk is embedded once during ingestion. At query time, the user question is embedded with the same model, and the vector store returns the chunks whose embeddings are closest to the query embedding under cosine similarity or Euclidean distance.

Persist the index to disk:

vectorstore.save_local("faiss_index")

Load it in subsequent runs:

vectorstore = FAISS.load_local(
    "faiss_index",
    embeddings,
    allow_dangerous_deserialization=True,
)

The allow_dangerous_deserialization flag acknowledges that FAISS uses pickle, which can execute arbitrary code if the index file is tampered with. In production, store the index in a location with appropriate access controls and integrity checks.

WHY persistence matters: re-embedding the entire corpus on every application restart is expensive and slow. A persistent index also allows you to version the corpus separately from the application code, track which documents were indexed at which timestamp, and roll back to a known-good index if ingestion fails.

Configuring and Using Retrieval Tools

Convert the vector store into a retriever:

retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4},
)

The retriever returns the top 4 chunks by similarity score. A small k risks omitting a chunk that contains a critical constraint or exception. A large k increases context size, may dilute the evidence that matters, and can confuse the model with contradictory or redundant information.

Build a deterministic retrieval chain for predictable knowledge-base questions:

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    (
        "system",
        """Answer using only the supplied context.
If the context does not contain the answer, say that the information
is not available in the retrieved sources.
Cite sources using the metadata supplied with each context chunk.

Context:
{context}""",
    ),
    ("human", "{question}"),
])

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def format_docs(docs):
    return "\n\n".join([
        f"Source: {doc.metadata.get('source', 'unknown')}\n{doc.page_content}"
        for doc in docs
    ])

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

answer = chain.invoke("How do I configure a custom retrieval tool?")
print(answer)

WHY this pattern works: the retriever runs first, fetches chunks, and injects them into the prompt. The model sees the evidence and is explicitly instructed to answer only from that evidence. The format_docs function preserves source metadata so citations can reference real document identifiers rather than hallucinated URLs.

A deterministic chain retrieves on every invocation. For workflows where retrieval is optional or where the agent must choose among multiple corpora, convert the retriever into a tool.

Integrating Deep Agents for Advanced Retrieval

The LangChain Deep Agents documentation, updated in early November 2026, introduces skills-guided retrieval, custom retrieval tools, and subagents for parallel evidence analysis. A tool-based retrieval pattern allows the agent to decide when to call the retrieval tool, combine retrieval with filesystem access or external APIs, and route queries to different indexes based on skill instructions.

Convert the retriever into a tool:

from langchain.tools.retriever import create_retriever_tool

retrieval_tool = create_retriever_tool(
    retriever,
    name="search_documentation",
    description=(
        "Search the LangChain documentation for technical guidance. "
        "Use this tool when the user asks about LangChain features, "
        "API usage, or implementation patterns. "
        "Returns relevant documentation chunks with source metadata."
    ),
)

The tool name and description control when the agent invokes retrieval. A vague description allows the agent to skip retrieval or use the wrong tool. A precise description that states the corpus, the question types that require retrieval, and the expected output format reduces decision ambiguity.

Build an agent with the retrieval tool:

from langchain.agents import create_openai_functions_agent, AgentExecutor
from langchain_core.prompts import MessagesPlaceholder

agent_prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Use the search tool when answering questions about LangChain."),
    MessagesPlaceholder(variable_name="chat_history", optional=True),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

agent = create_openai_functions_agent(llm, [retrieval_tool], agent_prompt)
agent_executor = AgentExecutor(agent=agent, tools=[retrieval_tool], verbose=True)

result = agent_executor.invoke({"input": "How do I configure a custom retrieval tool?"})
print(result["output"])

WHY use a tool-based agent: the model can skip retrieval for questions that do not require corpus evidence, combine retrieval with other tools such as a calculator or database query, and adapt its retrieval strategy based on intermediate results. The trade-off is an additional decision point where the model may fail to retrieve when it should.

The official Deep Agents docs describe subagents for parallel analysis of retrieved evidence. Each subagent receives a chunk and a specific analysis task, and the parent agent synthesizes the subagent outputs into a final answer. This pattern is useful when each chunk requires independent processing—such as extracting structured data, validating constraints, or scoring relevance—but adds model calls and coordination overhead. Use subagents selectively, not as a default retrieval pattern.

Skills-guided retrieval centralizes operational instructions such as index selection, query formulation, and citation format. A skill tells the agent which corpus to use, how to construct the query, and how to cite sources. The verified documentation identifies skills as a mechanism for reducing the chance that each agent prompt implements retrieval rules differently, but does not provide a working skill implementation example. Consult the current LangChain Deep Agents API reference for skill constructor syntax and integration patterns.

Common Pitfalls and How to Avoid Them in LangChain RAG

A stale corpus is the most common retrieval failure. If the source documents change but the index is not re-built, retrieval cannot supply updated information. The model will answer from outdated chunks or fall back to its training data. Track the corpus version in logs, expose index timestamps in observability dashboards, and trigger re-indexing when source documents change.

Poor chunk boundaries reduce evidence quality. Splitting a code block, table, or API reference entry across two unrelated chunks can render both chunks useless. For structured documents, use a splitter that respects Markdown headings, code fences, or section delimiters. For API documentation, split by endpoint or method. For legal or compliance documents, split by clause or numbered section.

Missing metadata makes citations and filtering impossible. Without source identifiers, URLs, headings, or timestamps, the application cannot verify citations, filter by tenant or permission, or trace a retrieved chunk back to its origin. Preserve all relevant metadata during ingestion and pass it through retrieval and generation.

Over-retrieval increases context size and can dilute the evidence that matters. Passing 20 chunks to the model when only 2 contain the answer wastes tokens and may confuse the model with contradictory or redundant information. Start with a small k and increase it only if evaluation shows that relevant chunks are being omitted.

Under-retrieval omits chunks that contain critical constraints or exceptions. A k of 1 or 2 is appropriate for simple factual questions but may miss edge cases, warnings, or version-specific guidance. Test retrieval against a set of known difficult questions and measure whether the expected chunks appear in the top-k results.

Uncontrolled agent retrieval occurs when the tool description is vague or the agent lacks explicit instructions about when retrieval is mandatory. The agent may skip retrieval, use the wrong index, or retrieve for questions that do not require corpus evidence. Write precise tool descriptions, include retrieval rules in the agent prompt, and log tool calls to detect retrieval failures.

Unverified citations allow the model to generate citation-looking text that does not correspond to real source metadata. Do not ask the model to invent URLs or document identifiers. Render citations from the retrieved Document.metadata values in application code, and validate that every citation maps to a chunk that was actually retrieved.

No observability makes it impossible to distinguish indexing errors, retrieval errors, prompt errors, and generation errors. A missing answer can result from a failure at any stage: the document was never indexed, the chunk was split incorrectly, the embedding model failed, the vector store returned irrelevant chunks, the prompt did not instruct the model to use the context, or the model ignored the instruction. Instrument the full request with LangSmith or an equivalent tracing system. The official tutorial states that LangSmith traces retrieval, tool calls, and model responses, allowing you to inspect which chunks were retrieved, which tools were called, and what the model generated at each step.

Before changing prompts or models, add retrieval evaluation. At minimum, test whether the expected source appears in the top-k results, whether retrieved chunks contain the answer-bearing passage, whether the generated answer is supported by those chunks, whether citations map to real source metadata, and whether the system abstains when evidence is absent. A deterministic evaluation set allows you to measure the impact of chunking changes, embedding model changes, and prompt changes independently.

Azguards has experience building RAG systems on LangChain for clients across enterprise AI, web development, and eCommerce. If your team needs hands-on help implementing this—whether it's indexing strategy, retrieval evaluation, or integrating Deep Agents into an existing workflow—we build production systems on this exact stack. Let's talk about your project.

Architect Your Solution → Explore our AI & Automation Systems work →