Private LLM deployment: open-weight models and RAG on your own servers
Private LLM deployment means running an open-weight language model, and the retrieval system around it, on servers you control, on-premises or in your own cloud account, so prompts and documents never go to an outside model provider. Sysvoy sizes the hardware, deploys the model on a production serving engine such as vLLM, connects it to your documents with permission-aware retrieval, and tests answer quality against the hosted models you'd otherwise use. A pilot on one GPU with one document source usually costs $8,000–$20,000 over 4–8 weeks; full deployments with several sources, monitoring and access control cost $20,000–$45,000 over 8–14 weeks. Hardware or GPU rental is separate.
The path a question takes
Documents are parsed and split, embedded, searched, and the answer cites the passages it came from.
- Ingest
- Chunk
- Embed
- Retrieve
- Answer
Where private LLM projects stall
And what we do differently.
Hardware bought before models are tested
GPUs sized for a model that turns out to be wrong for the job. We test candidate models on your data, often on rented GPUs, before anything is bought.
Judged against GPT by feel
A few side-by-side chats decide the project. We compare models on an evaluation set of your real questions, with scores everyone can see.
Only the model gets deployed
A model server with no sign-in, logging, update plan or capacity for busy hours. We ship the whole stack around it and size memory for peak use.
What we build
On-premise RAG
Answers from your internal documents with citations, where documents, embeddings and the model all stay inside your network.
A private model API
An OpenAI-compatible endpoint inside your network, so existing tools and apps can often switch from a hosted API with little more than a configuration change.
Air-gapped deployments
For environments that can't connect out: models, serving software and the search index delivered as offline packages, scoped case by case.
Hybrid setups
Sensitive work goes to the private model and everything else to a hosted API, behind one routing layer that applies your rules.
Model evaluation and selection
Two or three open-weight models tested on your own questions against a hosted baseline, so the choice rests on results, not benchmarks.
How we deliver
From the first workshop to production. How we work
Size it on your workload
Request volume, context length and response time decide the model size, quantisation and GPU count. We work it out before anyone buys or rents hardware.
Test models on your data
We compare candidate models on an evaluation set built from your real questions, with a hosted model as the benchmark.
Deploy and secure
The model runs on vLLM or SGLang behind your sign-in, with access rules, request logs and network controls your security team can review.
Run and upgrade
Monitoring for quality, speed and GPU use, and model upgrades that only go live after they pass the same evaluation set.
A hosted model API compared with private deployments
| Hosted model API | Private, in your cloud account | On-premises | |
|---|---|---|---|
| Where prompts and documents go | The provider's servers, under their terms | Your cloud account only | Your own data centre only |
| Upfront cost | None | Build fee; GPUs rented by the hour | Build fee plus hardware purchase |
| Running cost | Per token, rises with use | GPU hours, flat while running | Power and upkeep; hardware already paid for |
| Model choice | The provider's models, including the strongest | Any open-weight model | Any open-weight model, sized to your hardware |
| Best for | Getting started and variable volume | Data rules plus flexible capacity | Strict data rules or very high, steady volume |
A good fit when
- Your documents or prompts can't leave your network, for legal, contractual or regulatory reasons.
- You send enough requests that per-token API bills are heading past what dedicated GPUs would cost.
- Customers or auditors ask exactly where their data is processed, and you need one simple answer.
What we build with
- vLLM
- SGLang
- NVIDIA NIM
- llama.cpp
- PostgreSQL and pgvector
- Docker
- Kubernetes
- NVIDIA GPUs
- Private VPCs on AWS, Azure and Google Cloud
- Langfuse
Related work
AI commerce assistant
Self-hosted AI shopping assistant, Typesense product search and HubSpot integration for a US B2B distributor.
US distributorAI security platform
AI-driven security platform and SOC tooling for a European cybersecurity company.
Client under NDA
Planning a budget? Read our guides
- RAG development costWhat a retrieval-augmented generation system costs to build and run, from a studio that ships them. Price tiers, monthly infrastructure, build vs buy and two real examples.
- GDPR and offshore developmentYou can hire developers outside Europe and stay within GDPR. Here are the documents, the architecture patterns and the AI rules, from a team in Pakistan that builds for EU and UK clients.
Common questions
Not answered here? Email contact@sysvoy.com.
How much does a private LLM deployment cost?
Our build fee is usually $8,000–$20,000 for a pilot on one GPU with one document source (4–8 weeks), and $20,000–$45,000 for a full deployment with several sources, access control and monitoring. Hardware is separate. As of November 2026, renting one NVIDIA H100 GPU costs roughly $2–$7 an hour depending on the provider and plan, and buying a complete 8-GPU H100 server is commonly quoted at $250,000–$400,000. Prices move quickly, so we quote them fresh in discovery.
What hardware do we need?
It depends on the model size and how many people use it at once. A rule of thumb: model weights take the parameter count times the bytes per parameter, so a 70-billion-parameter model needs about 140GB at 16-bit precision (two 80GB GPUs) and about 35GB at 4-bit (one 80GB GPU), plus extra memory for conversations in progress. Several current open-weight models are designed to run on a single 80GB GPU.
Which open-weight models do you use?
We pick per project and test two or three on your data. As of November 2026, families with permissive Apache 2.0 licences include OpenAI's gpt-oss, Mistral 3 and Gemma 4. Some others, such as Llama 4 and the largest Qwen models, come with custom licences that add conditions, so we check the licence against your use before recommending one.
Are open-weight models as good as GPT or Claude?
For focused tasks with good retrieval, such as answering from your documents or extracting fields, they are often close enough, and we measure that on your own questions. On the hardest open-ended reasoning, the top hosted models usually still lead. A hybrid setup can send those requests to a hosted API and keep everything sensitive private.
Is self-hosting cheaper than using an API?
At low volume, no: an idle GPU costs money and an API doesn't. At steady, high volume it often is, and 69% of enterprises running open-weight models in production cited cost savings in a September 2026 ETR survey. We model your expected usage in discovery and show both numbers.
Can it run completely offline?
It can be done. Air-gapped setups need the model, serving software and search index delivered as offline packages, with a tested process for updates, which we scope case by case.
Do you work on our servers or in our cloud account?
Either. We deploy into your data centre or into your own AWS, Azure or Google Cloud account, so the infrastructure, keys and data stay under your control. We work only with the access you grant, and you can revoke it at any time.
Have something to build?
Tell us where things stand and what you need. A short message is enough to start.
