This project is an experimental sandbox for testing out ideas related to running local Large Language Models (LLMs) with Ollama and Pydantic AI to perform agentic Retrieval-Augmented Generation (RAG) for answering questions based on your documents. The agent can decide when and how to search documents, unlike fixed RAG pipelines. We use Ollama to create embeddings with nomic-embed-text stored in LanceDB for vector search.
- Ollama version 0.13.5 or higher.
- Clone this repository to your local machine.
- Install UV using instructions from the Astral site, Installation
- Create a virtual environment and install required Python packages by running
uv sync
Run the Streamlit application:
uv run streamlit run interfaces/streamlit_app.pyThis will start a local web server and open a new tab in your default web browser. The UI allows you to select models, specify a document folder, and chat with your documents.
Note: The first time you run the project, it will download the necessary models from Ollama for the LLM and embeddings. This may take some time depending on your internet connection.
Important: The selected LLM must support tool calling (function calling). The application uses an agentic approach where the model decides when to search documents, which requires tool calling capability. Models without this feature will not work correctly—they may fail silently, output raw tool syntax, or ignore documents entirely.
Not all Ollama models support tool calling reliably. We tested several model families and sizes to find the best options for this RAG application.
Recommended Models (Benchmarked):
- qwen3.5:9b - Best overall: highest quality (4.3/5 avg on 14-question eval), ~40% faster than qwen3:8b, multimodal (vision), 256K context. Current default.
- qwen3:8b - Strong alternative (4.4/5), slightly slower. Reliable tool calling.
- qwen3:14b - Same quality tier as 8b/9b but ~50% slower. No quality advantage for this RAG task.
These models were benchmarked using an automated eval harness (bench/) with a 480B cloud model (qwen3-coder) as the LLM judge. All three handle tool calling reliably and produce accurate, well-formatted answers.
Not recommended for agentic RAG:
- lfm2.5:8b-a1b - Fast (26s/question) but unreliable: failed to call the search tool on 30% of questions. Only ~1B active parameters (despite the 8B label).
Why Model Size Matters:
Smaller models (under 8B parameters, or hybrid models with few active parameters) struggle with agentic RAG tasks:
- They may fail to call the search tool when needed
- They often hallucinate instead of searching documents
- Some get stuck in search loops, calling the tool repeatedly without converging
The app includes safeguards (thinking-mode disabled, 4-search cap per question) to keep responses responsive and prevent runaway behavior.
The application uses MarkItDown to load documents:
- PDF, Word (.docx), PowerPoint (.pptx), Excel (.xlsx)
- Markdown (.md), HTML, CSV, JSON
- Pydantic AI: Type-safe agent framework with tool calling
- Ollama: Platform for running Large Language Models locally
- LanceDB: Vector database for storing and retrieving embeddings
- MarkItDown: Microsoft's document converter for PDF, Office files, and more
- Streamlit: Web framework for interactive applications
- UV: Fast Python package installer and resolver