Tired of Cobbling Together Local RAG Stacks, I Built a 100% Offline Enterprise-Grade RAG Scaffold

From Tinkering to Packaging: The Real Pain Points of a Local RAG Developer

Running a large language model locally isn't hard anymore — Ollama makes model deployment as simple as ollama run. But if you want non-technical users or enterprise clients to actually use it, while ensuring data never leaves the internal network, that's when problems arise: backend, frontend, vector database, orchestration logic, Docker configuration... piecing these together into a production-grade full-stack RAG application often takes days or even weeks.

After months of local LLM experimentation, developer toprakkbozbeyy decided to package up all this repetitive work into an out-of-the-box enterprise-grade RAG scaffold called VaultRAG, so developers can spend their time on business logic rather than infrastructure.

Tech Stack Choices: Built for Async and Offline

VaultRAG's selection philosophy is clear — modern, asynchronous, and completely isolated from the internet.

LLM and Embedding Models run locally via Ollama. Text generation was primarily tested with Llama 3.1 and Qwen, while nomic-embed-text was chosen for embeddings. This means the entire inference pipeline requires no calls to any external API.

Backend is built on Python and FastAPI, handling async API routing and document upload processing. FastAPI's async characteristics are especially important for RAG scenarios — document parsing and vector retrieval are IO-intensive operations, and synchronous frameworks easily become bottlenecks.

RAG Orchestration is handled by LangChain, covering PDF parsing, text chunking, and vector store retrieval. Frontend uses React with Tailwind CSS, presenting an interface similar to modern chat tools to lower the onboarding barrier for enterprise users. Deployment is uniformly managed by Docker Compose, with all services starting with a single command.

The trade-offs in this combination are pragmatic: no additional message queues or complex microservices were introduced, maintaining the maintainability of a single codebase.

Ollama is an open-source tool for running large models locally. It packages model weight downloads, quantization, inference engine, and API service into a single command-line program, supporting mainstream open-source models like Llama, Qwen, and Mistral. Developers simply use ollama pull to fetch a model and ollama run to start a conversation, gaining a locally available service endpoint compatible with the OpenAI API style, so that upper-layer applications need not be aware of the underlying inference framework.

nomic-embed-text is an open-source text embedding model designed specifically for retrieval scenarios. It supports longer context inputs and performs close to or even exceeds some closed-source embedding models on vector retrieval benchmarks. Its output vectors can be directly stored in vector databases for similarity retrieval, and it can run entirely locally, making it a common choice for replacing OpenAI Embeddings in private RAG solutions.

Breaking Down the Underlying Workflow

Document Ingestion: From PDF to Vectors

After users upload a PDF through the React interface, FastAPI receives the file and LangChain handles text processing. RecursiveCharacterTextSplitter is used here to split text into chunks, aiming to preserve contextual coherence as much as possible during chunking. Embeddings are then generated via Ollama and stored in a local vector store.

No data leaves the local environment throughout this process, which is exactly the feature enterprise scenarios value most.

RecursiveCharacterTextSplitter is a text splitter provided by LangChain. It recursively attempts to split text using a set of separators (typically paragraphs, newlines, sentences, spaces, etc.) until each chunk's length falls within the configured chunk size. Compared to hard-cutting by a fixed character count, this strategy better preserves the integrity of paragraphs and sentences, reducing semantic fragmentation. It also supports setting an overlap, allowing adjacent chunks to retain partial overlapping content, preventing key information from being split exactly at chunk boundaries — this is the origin of the chunk size and overlap tuning mentioned later.

Retrieval and Generation: SSE-Driven Streaming Responses

When a user asks a question, the backend first retrieves the most relevant text chunks, assembles them into a prompt, then streams the response from the local LLM via Server-Sent Events (SSE). SSE here isn't just a UX optimization — local model inference speed is hardware-limited, and streaming output lets users see content immediately, greatly reducing perceived latency.

This architectural choice is worth noting: compared to WebSocket, SSE is more lightweight and fully sufficient for unidirectional text streaming scenarios.

Server-Sent Events (SSE) is an HTTP-based server push technology. The server maintains a long connection and continuously sends data streams in text/event-stream format to the client. Unlike WebSocket's full-duplex communication, SSE is unidirectional (server to client), with a simpler protocol, native support for reconnection, and no additional handshake upgrade process. In scenarios like LLM streaming output — where the client sends one request and the server continuously returns tokens — SSE achieves an experience comparable to WebSocket with lower implementation complexity, and is therefore adopted by a large number of chat-based AI applications.

Pitfalls Encountered: Docker Networking and Chunking Parameters

The author admits that Docker network configuration was the initial challenge. Getting the FastAPI container to communicate with an Ollama instance on the host machine (or a containerized Ollama) requires specific network configuration. Such issues are common in local development but often require additional debugging under Docker Compose orchestration.

Another key point is tuning LangChain's chunk size and overlap. Too small, and the LLM loses context; too large, and inference speed on local hardware drops significantly. This is a balance point that requires repeated experimentation based on the model and hardware — there are no universal parameters, only tuning for specific scenarios.

Docker networking: Containers are by default in an isolated bridge network, and localhost inside a container points to the container itself, not the host machine. If Ollama runs on the host, for the FastAPI container to reach it, you typically need host.docker.internal (on Docker Desktop), or on Linux, --network=host, a custom bridge network with the host gateway IP specified, to establish connectivity. If Ollama itself is also containerized, both containers need to join the same custom network and resolve each other by service name. These differences are exactly why network debugging is so frequent in local RAG deployments.

chunk size and overlap directly affect retrieval quality and inference overhead: chunks that are too small result in individual chunks lacking complete semantics and fragmented retrieved context; chunks that are too large quickly inflate the prompt, exceeding the model's effective context window and significantly slowing inference on local hardware. Overlap is used to retain overlapping text between chunks to maintain semantic continuity, but excessive overlap causes redundant storage and duplicate retrieval. In practice, finding the balance point requires repeated experimentation combining document type, model context length, and hardware compute power.

Use Cases and Open-Source Address

VaultRAG is positioned as a scaffold rather than a finished product, suitable for three types of users: freelance developers building AI tools for clients, teams participating in hackathons that need to quickly build prototypes, and individual users who want to deploy private AI for their own confidential documents.

The author has open-sourced some components and documentation on GitHub: github.com/toprakkbozbeyy/vaultrag-enterprise-starter-kit. At the end, a question was posed to the community: Is there a better way to handle local vector storage without bloating the Docker image size? This question hits a pain point for many local RAG deployments — the image size and persistence strategy of the vector database often determine the deployment cost of the entire solution.