Building a Private RAG System with Ollama: Chat with Your Documents Locally

ChatGPT reads your confidential documents and uses them for training. Ollama + a local vector store gives you the same capability — with zero data leaving your machine. Your board meeting notes, proprietary code, and draft patents stay on your hardware, answering questions at the speed of your GPU. Forget API costs and privacy waivers; this is retrieval-augmented generation (RAG) that actually respects the "private" in private AI. With 70% of self-hosted LLM users citing data privacy as their primary reason (a16z AI survey 2025), it's time to build the system they're talking about.

Let's cut through the abstraction layers and build a RAG pipeline where every component—the LLM, the embeddings, the vector database—runs locally, orchestrated by Ollama. We'll move from theory to a working system you can query from your terminal before the end of this guide.

RAG Architecture: What Actually Runs on Your Machine

A cloud RAG system is a distributed mess of API calls. A local RAG system is a carefully orchestrated symphony of processes on one box. Here’s the stack, from metal to meaning:

1. The Foundation: Ollama. This isn't just a model runner; it's your local LLM server. You ollama pull a model (like llama3.1:8b), and it sits loaded in VRAM/RAM, waiting for HTTP requests on localhost:11434. It also serves embedding models (more on that soon).
2. The Memory: Local Vector Store. This is your document brain. ChromaDB (runs in-memory or as a local server) or Qdrant (via its Docker container) stores vector embeddings of your text chunks. No external SaaS, no monthly fee per vector.
3. The Conductor: LlamaIndex or LangChain. These frameworks handle the pipeline: loading your PDF, splitting it, converting chunks to vectors via Ollama, storing them, and finally, retrieving relevant chunks to stuff into the LLM's prompt. We'll use LlamaIndex for its cleaner abstraction on top of Ollama.
4. The Source: Your Documents. Anything in your ~/Documents folder is fair game.

The magic is that steps 2-4 can happen entirely offline. The only prerequisite is having pulled the models once.

Picking Your Workhorse: Which Ollama Model for Embeddings and Generation?

Not all Ollama models are created equal. You need two: a small, fast model for creating embeddings (turning text into vectors), and a capable model for generating answers.

For Embeddings: Use nomic-embed-text. It's a 137M parameter model designed specifically for the task, supported directly by Ollama. It's small, accurate, and crucially, generates embeddings compatible with standard vector search. Don't waste your 8B parameter chat model on this.

For Generation: This is your quality/speed trade-off. Your shiny new RTX 4090 might crave the llama3.1:70b, but let's be practical.

ollama pull nomic-embed-text
ollama pull llama3.1:8b

Here’s the cold, hard data to inform your choice:

First Real Error & Fix:

Error: model 'llama3' not found

Your instinct is ollama pull llama3. Don't. The library has evolved. Fix: Run ollama pull llama3.1:8b (note the version suffix .1). Be specific.

Wiring ChromaDB as Your Local Vector Brain

We'll use ChromaDB in its persistent, local server mode. It's simple and keeps your index between sessions.

# Install the needed packages in your project environment
pip install llama-index llama-index-embeddings-ollama llama-index-vector-stores-chroma chromadb

Now, let's initialize the pipeline. This code connects Ollama (for embeddings) to ChromaDB (for storage).

# rag_setup.py
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
from llama_index.core import StorageContext

# 1. Tell LlamaIndex to use Ollama for embeddings
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")

# 2. Initialize ChromaDB client, persisting data to './chroma_db'
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("document_chunks")

# 3. Wrap it in a LlamaIndex vector store
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

print("✅ Local vector store initialized. Ollama embedding engine ready.")

Run this once. You now have a ./chroma_db directory that will hold all your document knowledge.

Document Chunking: The Secret Sauce of Retrieval Accuracy

Chunking is where most RAG systems fail silently. Too big, and you retrieve irrelevant paragraphs. Too small, and the LLM loses necessary context. Here's a strategy that works:

• Size: 1024 tokens. This is a good default for models with 4k-8k context windows.
• Overlap: 200 tokens. This ensures concepts that span a chunk boundary aren't lost.
• Metadata: Always store the source file name and the chunk's starting character. You'll need this for citations.

LlamaIndex's SentenceSplitter handles this. Let's integrate it and ingest a sample document.

# ingest.py
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from rag_setup import storage_context, Settings  # Import our previous setup

# Load documents from a 'data/' folder
documents = SimpleDirectoryReader("./data").load_data()

# Configure the chunking parser
text_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=200)
Settings.text_splitter = text_splitter

# Create the index: this embeds chunks via Ollama and stores them in ChromaDB
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    show_progress=True
)

print(f"✅ Indexed {len(documents)} documents into local vector store.")

Second Real Error & Fix:

VRAM OOM with 70B model

You got greedy. The 70B model, even quantized, needs ~40GB VRAM. Fix: Use a smaller model, or aggressively quantize: ollama run llama3.1:70b-instruct-q4_K_M. Better yet, start with llama3.1:8b.

Benchmark: How Chunk Size Dictates Answer Quality

Let's quantify the intuition. I took a 50-page technical PDF and asked a specific, detailed question. Here's how chunk size affected the system's ability to find the single relevant paragraph:

The takeaway: Small, overlapping chunks with smart retrieval beat massive chunks. 1024/200 is a robust default.

End-to-End Example: Querying Your 500-Page PDF Locally

Your data is indexed. Ollama is running (ollama serve). Time to chat. This script is your query engine.

# query.py
from llama_index.core import VectorStoreIndex
from llama_index.llms.ollama import Ollama
from rag_setup import storage_context, Settings

# 1. Point to the LLM. This makes an API call to localhost:11434
Settings.llm = Ollama(model="llama3.1:8b", request_timeout=120.0)

# 2. Load the existing index from ChromaDB (no re-embedding needed)
index = VectorStoreIndex.from_vector_store(
    storage_context.vector_store,
    embed_model=Settings.embed_model
)

# 3. Create a query engine that retrieves top 3 most relevant chunks
query_engine = index.as_query_engine(similarity_top_k=3, streaming=True)

# 4. Ask a question
print("Query Engine Ready. Type 'exit' to quit.")
while True:
    query = input("\nYour question: ")
    if query.lower() == 'exit':
        break

    print("\nAnswer: ", end="", flush=True)
    response = query_engine.query(query)
    # Print streaming response
    for text in response.response_gen:
        print(text, end="", flush=True)
    print("\n")
    # Print sources
    for i, source in enumerate(response.source_nodes, 1):
        print(f"[Source {i}] {source.metadata.get('file_name', 'N/A')}")

Run python query.py. Ask about something only in your PDF. Witness the response stream from your local machine, citing its sources. Total latency is your Ollama API first-token latency (~300ms local) plus retrieval time. It feels private and fast.

Leveling Up: Re-ranking and Context Compression

Basic "top-k similarity" retrieval sometimes fails. Two advanced tricks:

1. Re-ranking: Retrieve 10 chunks, then use a small, cross-encoder model to re-score them for true relevance to the question, not just the query keywords. This is computationally cheap and dramatically improves precision.

2. Context Compression: The LLM's context window is precious. Instead of blindly concatenating 3 retrieved chunks, use the LLM itself to summarize/extract only the parts relevant to the query from each chunk before feeding them into the final prompt. This is called a ContextChatEngine in LlamaIndex and is your best weapon against context limits.

Implementing these turns a good local RAG system into a great one, often outperforming naive cloud implementations that just do simple vector search.

Next Steps: From Prototype to Persistent Tool

You've built the core. Now, productionize it.

1. Frontend: Point Open WebUI or AnythingLLM to your local Ollama and ChromaDB. You now have a private ChatGPT UI over your docs.
2. Automation: Use cron or a folder watcher to re-index new documents dropped into a specific directory.
3. Optimization: Experiment with Modelfile to create a custom-tuned variant of your model, perhaps with a system prompt pre-set for "Answer based only on the provided context."
4. Scale: If your document library grows massive, switch the vector store to a local Qdrant instance for better performance and filtering capabilities.

The math is compelling: Running Llama 3.1 8B locally costs $0 vs ~$0.06/1K tokens on GPT-4o (as of Jan 2026). With Ollama supporting 150+ models as of Q1 2026, your private toolkit is vast. You’ve not just built a tool; you’ve established a sovereign data perimeter. Your documents answer to you, on your terms, at the speed of your hardware. Now go query something confidential.