# How I Built a Fully Local RAG App with Ollama, FastAPI, and Qdrant

As developers, we’re often faced with the question:  
**How do we bring AI into our apps without giving up privacy, control, and budget?**

The answer: **local-first AI using Retrieval-Augmented Generation (RAG)**.

RAG allows you to feed your own data (PDFs, notes, docs) into an LLM—so it doesn't hallucinate but instead *grounds* its answers on your actual content.  
When done **locally**, this becomes a powerful, private, and fully offline assistant.

In this guide, I’ll show you how I built a **local, private ChatGPT clone** that:

* Reads PDFs or markdown files
    
* Embeds and indexes them into a vector database (Qdrant)
    
* Uses a local LLM (via Ollama) for generating responses
    
* Serves everything over a clean FastAPI backend
    

No OpenAI. No vendor lock-in. No tokens burned.

## Architecture Overview

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1743930665016/7b922e45-2482-4f7d-9424-b45d28be58fd.png align="center")

---

## 1\. The Theory Behind It

### 🔸 What is RAG (Retrieval-Augmented Generation)?

RAG bridges two worlds:

* **Information Retrieval** (search, chunking, semantic similarity)
    
* **Text Generation** (LLMs like GPT, LLaMA, Mistral)
    

Instead of making your model "know everything," you let it **look things up**. This drastically improves accuracy and interpretability.

### 🔸 Why Local?

* You control your data
    
* Costs are predictable (or free)
    
* Perfect for privacy-sensitive domains like healthcare, law, or enterprise internal tools
    

---

## 2\. Setup & Tools

### Stack:

* **Ollama** – Run Mistral or LLaMA locally with GPU or CPU.
    
* **FastAPI** – Lightning-fast Python API framework.
    
* **Qdrant** – Vector database for semantic search.
    
* **LangChain** – Orchestrates RAG logic.
    
* **Sentence Transformers** – For embedding docs.
    

---

## 3\. Installing Dependencies

### Python Packages:

```bash
pip install fastapi uvicorn langchain qdrant-client pypdf sentence-transformers
```

### Ollama:

```bash
# MacOS
brew install ollama
ollama run mistral

# Linux
curl -fsSL https://ollama.com/install.sh | sh
```

### Qdrant via Docker:

```bash
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
```

---

## 4\. Load & Chunk Your Docs

We’ll use LangChain to split PDFs into small chunks for embedding.

```python
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("example.pdf")
pages = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
docs = splitter.split_documents(pages)
```

---

## 🔎 5. Embed and Store in Qdrant

```python
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Qdrant

embedding_model = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

qdrant = Qdrant.from_documents(
    documents=docs,
    embedding=embedding_model,
    location="http://localhost:6333",
    collection_name="mydocs"
)
```

---

## 6\. FastAPI Backend

Let’s build a clean API to handle queries, retrieve docs, and pass them to the local LLM.

```python
from fastapi import FastAPI
from pydantic import BaseModel
import requests

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/ask")
def ask(query: Query):
    retriever = qdrant.as_retriever(search_kwargs={"k": 4})
    docs = retriever.get_relevant_documents(query.question)

    context = "\n\n".join([d.page_content for d in docs])
    prompt = f"""Use the following context to answer the question:\n\n{context}\n\nQuestion: {query.question}"""

    response = requests.post("http://localhost:11434/api/generate", json={
        "model": "mistral",
        "prompt": prompt,
        "stream": False
    })

    result = response.json()
    return {"answer": result["response"]}
```

Run it:

```bash
uvicorn app:app --reload
```

---

## 7\. Interacting With It

You can now hit:

```bash
curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "Summarize the document."}'
```

Or use Postman, Insomnia, or even a React/Vue frontend.

---

## 8\. Privacy, Security & Real-World Considerations

* This system never leaves your machine. Great for **air-gapped environments**.
    
* If needed, you can **Dockerize** the whole stack and deploy it on your private cloud.
    
* Upgrade Qdrant with TLS, authentication.
    
* Switch embedding model to `intfloat/e5-large-v2` for better multi-lingual/doc understanding.
    

---

## 9\. Bonus: Add File Upload + Frontend

Extend your FastAPI backend with `/upload` endpoint using `aiofiles`, and wire up a React frontend with:

* Drag-and-drop file upload
    
* Chat window with streaming responses
    
* Local memory using IndexedDB
    

---

## 10\. Final Thoughts

This stack changed how I prototype AI tools. Instead of burning tokens and stressing about data security, I now run **entire GPT-style systems locally**, with:

* Real-time responses
    
* Grounded context from my docs
    
* Full control over prompt tuning and latency
    

---

**If you enjoyed this, follow my blog or drop me a message. I love building clean, production-ready tools that put AI in the hands of indie developers and engineers.**