How to build a private AI assistant on your own docs (RAG in 5 steps)

Why a company-private AI assistant

ChatGPT will answer questions about the world. But it won't tell you how your service procedure X looks in your company, who owned project Y in 2023, or what annex 3 of the contract with client Z says.

RAG (Retrieval-Augmented Generation) solves this: the AI first searches your document base, picks relevant fragments, and only then generates an answer. The result is an assistant that knows your company from the inside — without sending documents to OpenAI.

RAG vs fine-tuning

Two approaches often confused:

Fine-tuning — retrain the model on your data. Expensive (GPU, time, expertise), durable, but the model "remembers" only what it saw during training. New documents require retraining.

RAG — the model stays unchanged. Your documents are indexed into a vector database. For each question the system retrieves the 3-10 most relevant fragments and injects them into the prompt. Cheap, fast to update, ideal for knowledge bases that change.

For 95% of company deployments, RAG is the right choice. Fine-tuning only makes sense when you need a very specific writing style or a narrow domain.

On-premise stack

What we actually deploy at a client site:

LLM: Llama 3 8B or 70B (via Ollama or vLLM).
Embeddings: nomic-embed-text or bge-large-en-v1.5 (also local).
Vector database: Qdrant (open-source, on-prem).
Orchestration: LangChain or LlamaIndex.
Frontend: Open WebUI with SSO, or a custom React app.
Hardware: a server with RTX 4090 (for 8B) or A100 (for 70B).

Everything runs offline. Data never leaves the client network.

Step 1: Document audit

The worst mistake is to dump everything from company drives. RAG on chaos returns chaotic answers.

What we do first:

List sources: SharePoint, Confluence, Google Drive, mailboxes, repos, intranet.
Classify: which documents are current, outdated, confidential.
Pick a subset: start with 200-2000 documents from one area (HR procedures, product docs, support KB).
Update policy: who, how often, and by what rule adds new documents.

Without this step, three months later the client reports: "the AI lies". Because it pulls content from a 2019 doc nobody deleted.

Step 2: Indexing (chunking + embeddings)

Documents are cut into fragments (chunks) of 300-800 tokens with a 50-100 token overlap. Each chunk goes to the embedding model, which converts it into a numeric vector (typically 768 or 1024 dimensions).

Vectors are stored in Qdrant with metadata (source, date, author, section, tags). Metadata is key — it lets you filter: "HR procedures from 2025 only", "documents tagged 'current' only".

Pitfalls:

Tables and images — naive text chunking loses them. Use a parser (Unstructured.io, LlamaParse) that understands structure.
Scanned PDFs — require OCR (Tesseract, Azure Document Intelligence on-prem).
Email attachments — often contain the most knowledge but get skipped by indexers.

Step 3: Retrieval

When a user asks a question, the system does four things:

The question goes to the embedding model.
The question vector is compared against vectors in Qdrant (cosine similarity).
Top-K closest fragments are returned (typically 5-10).
Optionally a reranker (e.g. bge-reranker-large) sorts them more precisely.

For larger bases we add hybrid search — vector + classic BM25 (Elasticsearch or Qdrant BM25). This improves queries with concrete names (document numbers, codes, surnames).

Step 4: Generation (LLM answers)

The selected fragments are injected into the prompt along with the user question and system instructions. A typical prompt:

You are an assistant for company X. Answer only based on the following
fragments of company documentation. If you can't find the answer in the
fragments, say "I did not find this in the knowledge base."
Always cite the source document number.

[FRAGMENT 1] ...
[FRAGMENT 2] ...
...

User question: ...

Llama 3 70B answers and adds source numbers. The frontend renders the answer with clickable citations linking to the original documents.

Step 5: Evaluation and iteration

After deployment we test with real questions from employees:

Is the answer correct? (manual check on 100-200 questions).
Does it cite the right sources?
Are hallucinations under control? (the model doesn't make up info not in the base).
Does it abstain when it doesn't know? (rather than guessing).

Typical week-one improvements:

Adding filter rules (skip docs older than 2 years).
Increasing K (from 5 to 10 fragments).
A reranker for hard queries.
Better table chunking.
Updating embeddings for new documents (cron job overnight).

Security and access control

In a company with multiple departments you don't want HR's assistant to see Finance docs.

What we build:

Metadata filters at retrieval: SSO passes user role into RAG, the Qdrant query gets filter: { department: "HR" }.
Query logs: who, what, when — stored locally, e.g. 90-day retention.
Document access policy: ACL at the fragment level (Qdrant payload filtering).
"Explain" mode: the user can see exactly which fragments the model saw.

Deployment cost

Realistic ranges:

Pilot (one area, 500-1000 docs, RTX 4090) — €6,000–11,000.
Production (RAG on the whole company, A100, integrations) — €19,000–35,000.
Monthly operations — €500–1,200 (monitoring, updates, model upgrades).

Payback is usually 6-9 months from time saved by support, legal, consulting and HR staff who stop hunting documents manually.

Starting checklist

Before you begin a RAG deployment, check:

Pilot area selected (one department, one document type).
List of 200-2000 documents to index.
Update policy (who, how often).
Access requirements (do you need per-role ACL).
Hardware — do you have a GPU server, or do you start in a private cloud.
Evaluation plan — who tests answers in the first 2 weeks.

What's next

If you're ready, book an on-premise AI consultation — we'll show a working proof-of-concept RAG on your documents before you commit to a full deployment. You can also read the ChatGPT Enterprise vs local Llama 3 comparison before deciding on architecture.