A private LLM is a language model installed on the company’s own servers, or on dedicated servers, in France or in your country, and adapted to its business. Documents and conversations stay on that infrastructure. It is also called local AI or on-premise AI.
Key points
- A private LLM runs on your premises or on servers in France or in your country: your documents and questions do not go to an AI provider.
- The GPU memory you need depends mostly on the model size and the number of simultaneous conversations.
- It answers from your documents and cites its sources, but it can be wrong: answers can be checked.
What the platform includes
- A user interface
- An application API
- A local inference engine
- A document index
What it does
- Conversations with per-user history, answers streamed as they are written
- Analysis of PDF, Word, CSV or text documents, with answers based on the retrieved passages and the sources shown
- 16 ready-made query templates, in 4 categories
- Export of an answer or a conversation to PDF and Word
- Optional web search, enabled per query and can be disabled for the whole platform
- Administration: accounts, roles, usage statistics
- Deployment in Docker containers, health checks, operations documentation
The model
The starting point is an open-source model chosen per project: Mistral, LLaMA, Qwen or Gemma. It is adapted to the client’s business through fine-tuning, then evaluated on cases defined with them.
How much GPU memory do you need?
The graphics card memory (VRAM) has to hold two things: the model weights, and the working memory of the ongoing conversations, called the KV cache.
Weights are easy to estimate: number of parameters × bytes per parameter. An 8-billion-parameter model takes about 16 GB in 16-bit, and about 4 GB once quantised to 4-bit, plus some runtime headroom. The KV cache grows with the length of the texts processed and the number of simultaneous users.
The calculator below does the estimate for common open-source models, based on their published architecture.
Try it
GPU memory calculator
Weight precision
Text length per conversation
Simultaneous conversations
Estimated GPU memory
≈ 24 GB
- Model weights: 16 GB
- Conversations KV cache: 5.4 GB
- Runtime headroom: 2.6 GB
Example configuration: 1 × 48 GB GPU
Rough estimate: full KV cache for every conversation, in 16-bit, with no optimisation. Recent inference engines share memory across conversations: the real need is often lower. A token is a piece of a word.
Hardware at a glance
The need depends mostly on how many questions are processed at the same time. Rough guide:
| Profile | Users | Servers | Indicative GPU |
|---|---|---|---|
| Pilot | 5 to 20 | 1 machine | 1 × 16 to 24 GB (e.g. RTX 4000 Ada, L4) |
| Department or SME | 50 to 200 | 1 to 2 machines | 1 to 2 × 48 GB (e.g. L40S, A6000) |
| Large company | 500 and more | Several nodes | 4 to 8 × 80 GB and more (e.g. H100, H200) |
Current limits
- No character recognition: a scanned PDF cannot be used
- No image, audio or video input
- No connection to the company directory (SSO)
- The model can be wrong, even with sources
Frequently asked questions
Further reading
- RAG: an AI that answers from your documentsHow an AI finds the right passages in your documents, answers and cites its sources. Diagram, step-by-step demo and limits, in plain English.
- Private LLM or AI via API: how to choosePrivacy, hardware, models, time to start: the concrete criteria for choosing between an open-source private LLM and Claude or GPT through an API.
- Fine-tuning: adapting a model to your businessWhat fine-tuning an open-source model changes, how it differs from RAG and instructions, how it is evaluated and what you own at the end.
In 30 minutes, we look together at whether a pilot makes sense for you.