How Do You Deploy a Private AI Knowledge Base on a Budget?

How do you deploy a self-hosted AI knowledge base on a small budget?

Deploy a self-hosted AI knowledge base by putting a RAG workspace, document parser, vector store, and model endpoint on one private Linux server. Start with a CPU host and a commercial API, then add a GPU only after real usage proves it is needed. Keep documents behind identity-based access rules.

That is the version I would ship first. A private chatbot connected to every company folder is a security project wearing a friendly UI.

What should the first version contain?

A useful knowledge base has four jobs: accept documents, split and index them, retrieve relevant passages, and generate an answer with citations. You do not need Kubernetes for the first deployment. A single Docker Compose host is easier to back up and easier to understand when something breaks on a Friday afternoon.

For the application layer, AnythingLLM is MIT-licensed and offers a Docker deployment. Open WebUI is another practical front end, with its own license and branding conditions that you should read before white-labeling a larger internal deployment. For difficult PDFs, RAGFlow uses layout-aware document processing, but its published hardware requirements are heavier.

The smallest sensible pilot is one department, one document collection, and read-only answers. Start with the sales playbook or an operations manual. Leave HR and payroll out until permissions, deletion, and audit logging have been tested.

What hardware does an SMB knowledge base need?

Separate the host from the model. A CPU server can run the web app, PostgreSQL, and vector search while sending generation to a commercial API. That is the cheapest private-interface design because the documents and access layer stay under your control while the model remains metered.

If you need local inference, Hetzner's GEX45 listing shows a 24 GB NVIDIA RTX PRO 4000 Blackwell system at €214 per month excluding VAT, plus a €209 setup fee. It includes 64 GB RAM and two 512 GB NVMe drives. That is enough to test smaller quantized models, not a guarantee of frontier-model quality.

The same page lists a GEX63 with 96 GB of GPU memory at €999 per month excluding VAT, plus a €999 setup fee. Most small businesses should not begin there. Large VRAM is useful when several people query the system at once or when the model and context are large, but buying capacity before measuring concurrency is how a pilot acquires a monthly invoice.

Which software components should I choose?

Use PostgreSQL with pgvector if your team already knows PostgreSQL and wants documents, metadata, and embeddings in one database. Use Qdrant when a separate vector service fits the team better. The choice matters less than backups and permission filters.

For local inference, Ollama is a reasonable single-user starting point. Its default API binds to localhost, which is a good default. Do not expose port 11434 to the public internet just because a tutorial told you to set a permissive host binding. Put the service on an internal Docker network.

When several employees query the system at once, vLLM is the more production-oriented option. Its documentation describes continuous batching and a Linux-based deployment model. That adds operational work, so use it when measured peak concurrency justifies the change, not because the name sounds more serious.

How do I keep private documents from leaking?

The model must never decide what a user is allowed to read. Your application should authenticate the employee, attach department or group permissions to retrieval, and fetch only permitted chunks. A system prompt cannot repair a vector search query that already returned the wrong payroll document.

Put the stack behind a VPN or a reverse proxy with TLS. Keep the model endpoint and vector database on the private network. Store secrets outside the repository, turn on multi-user authentication before inviting staff, and create a backup that you have actually restored once.

Make separate collections for general company material, engineering, and HR. This is less elegant than one universal index. It is also easier to test. If a general employee can retrieve an HR passage, you want the failure to be obvious in a staging question, not discovered in a real conversation.

Read the OWASP Top 10 for LLM Applications before launch, especially the sections on prompt injection, sensitive information disclosure, and unbounded consumption. A malicious instruction hidden in an uploaded document can be retrieved like any other text. Treat documents as untrusted input.

How should I test answer quality?

Create a test set before indexing the first production folder. Write 25 real questions: questions with one answer, questions that require two documents, and questions the system must refuse. Record the expected source, answer, citation, and permission group. Then run the same set after changing the embedding model, chunk size, parser, or local model.

I would also add five deliberately awkward PDFs. Scanned tables and badly exported spreadsheets are where the cheerful demo tends to meet reality. Check whether the parser preserves page numbers and headings. If a user cannot open the cited source, the citation is decoration.

Watch four things during the pilot: retrieval misses, unsupported answers, response time, and access-control failures. The last category is a release blocker. A slower answer can be improved. A leaked document changes the project.

What should an SMB do this week?

Choose one document owner and one department. Put a CPU-only Docker deployment behind a VPN, load 30 to 50 approved files, and connect a commercial API first. Keep the model endpoint private, log retrieval events, and ask five employees to use the test set.

If the API bill or latency is the actual problem, test a 24 GB GPU with a small open-weight model. If answer quality fails on the real questions, stop there. A private answer that is wrong is still wrong, just more expensive to debug.

My default stack for a small team is a simple RAG workspace, PostgreSQL with pgvector, a commercial API, and a backup job. Add local inference after the workflow earns it. If you want help scoping the document set or testing the deployment, contact DeuceWorks and bring the awkward PDFs.