Private LLM deployment: open-weight models and RAG on your own servers

Private LLM deployment means running an open-weight language model, and the retrieval system around it, on servers you control, on-premises or in your own cloud account, so prompts and documents never go to an outside model provider. Sysvoy sizes the hardware, deploys the model on a production serving engine such as vLLM, connects it to your documents with permission-aware retrieval, and tests answer quality against the hosted models you'd otherwise use. A pilot on one GPU with one document source usually costs $8,000–$20,000 over 4–8 weeks; full deployments with several sources, monitoring and access control cost $20,000–$45,000 over 8–14 weeks. Hardware or GPU rental is separate.

The path a question takes

Documents are parsed and split, embedded, searched, and the answer cites the passages it came from.

  1. Ingest
  2. Chunk
  3. Embed
  4. Retrieve
  5. Answer

Where private LLM projects stall

And what we do differently.

  • Hardware bought before models are tested

    GPUs sized for a model that turns out to be wrong for the job. We test candidate models on your data, often on rented GPUs, before anything is bought.

  • Judged against GPT by feel

    A few side-by-side chats decide the project. We compare models on an evaluation set of your real questions, with scores everyone can see.

  • Only the model gets deployed

    A model server with no sign-in, logging, update plan or capacity for busy hours. We ship the whole stack around it and size memory for peak use.

What we build

  • On-premise RAG

    Answers from your internal documents with citations, where documents, embeddings and the model all stay inside your network.

  • A private model API

    An OpenAI-compatible endpoint inside your network, so existing tools and apps can often switch from a hosted API with little more than a configuration change.

  • Air-gapped deployments

    For environments that can't connect out: models, serving software and the search index delivered as offline packages, scoped case by case.

  • Hybrid setups

    Sensitive work goes to the private model and everything else to a hosted API, behind one routing layer that applies your rules.

  • Model evaluation and selection

    Two or three open-weight models tested on your own questions against a hosted baseline, so the choice rests on results, not benchmarks.

How we deliver

From the first workshop to production. How we work

  1. Size it on your workload

    Request volume, context length and response time decide the model size, quantisation and GPU count. We work it out before anyone buys or rents hardware.

  2. Test models on your data

    We compare candidate models on an evaluation set built from your real questions, with a hosted model as the benchmark.

  3. Deploy and secure

    The model runs on vLLM or SGLang behind your sign-in, with access rules, request logs and network controls your security team can review.

  4. Run and upgrade

    Monitoring for quality, speed and GPU use, and model upgrades that only go live after they pass the same evaluation set.

A hosted model API compared with private deployments

Hosted model APIPrivate, in your cloud accountOn-premises
Where prompts and documents goThe provider's servers, under their termsYour cloud account onlyYour own data centre only
Upfront costNoneBuild fee; GPUs rented by the hourBuild fee plus hardware purchase
Running costPer token, rises with useGPU hours, flat while runningPower and upkeep; hardware already paid for
Model choiceThe provider's models, including the strongestAny open-weight modelAny open-weight model, sized to your hardware
Best forGetting started and variable volumeData rules plus flexible capacityStrict data rules or very high, steady volume

A good fit when

  • Your documents or prompts can't leave your network, for legal, contractual or regulatory reasons.
  • You send enough requests that per-token API bills are heading past what dedicated GPUs would cost.
  • Customers or auditors ask exactly where their data is processed, and you need one simple answer.

What we build with

  • vLLM
  • SGLang
  • NVIDIA NIM
  • llama.cpp
  • PostgreSQL and pgvector
  • Docker
  • Kubernetes
  • NVIDIA GPUs
  • Private VPCs on AWS, Azure and Google Cloud
  • Langfuse

Related work

See all work

  • AI commerce assistant

    B2B e-commerceAI build and ongoing SEO

    Self-hosted AI shopping assistant, Typesense product search and HubSpot integration for a US B2B distributor.

    US distributor
  • AI security platform

    CybersecurityFlagship AI engagement

    AI-driven security platform and SOC tooling for a European cybersecurity company.

    Client under NDA

Common questions

Not answered here? Email contact@sysvoy.com.

How much does a private LLM deployment cost?

Our build fee is usually $8,000–$20,000 for a pilot on one GPU with one document source (4–8 weeks), and $20,000–$45,000 for a full deployment with several sources, access control and monitoring. Hardware is separate. As of November 2026, renting one NVIDIA H100 GPU costs roughly $2–$7 an hour depending on the provider and plan, and buying a complete 8-GPU H100 server is commonly quoted at $250,000–$400,000. Prices move quickly, so we quote them fresh in discovery.

What hardware do we need?

It depends on the model size and how many people use it at once. A rule of thumb: model weights take the parameter count times the bytes per parameter, so a 70-billion-parameter model needs about 140GB at 16-bit precision (two 80GB GPUs) and about 35GB at 4-bit (one 80GB GPU), plus extra memory for conversations in progress. Several current open-weight models are designed to run on a single 80GB GPU.

Which open-weight models do you use?

We pick per project and test two or three on your data. As of November 2026, families with permissive Apache 2.0 licences include OpenAI's gpt-oss, Mistral 3 and Gemma 4. Some others, such as Llama 4 and the largest Qwen models, come with custom licences that add conditions, so we check the licence against your use before recommending one.

Are open-weight models as good as GPT or Claude?

For focused tasks with good retrieval, such as answering from your documents or extracting fields, they are often close enough, and we measure that on your own questions. On the hardest open-ended reasoning, the top hosted models usually still lead. A hybrid setup can send those requests to a hosted API and keep everything sensitive private.

Is self-hosting cheaper than using an API?

At low volume, no: an idle GPU costs money and an API doesn't. At steady, high volume it often is, and 69% of enterprises running open-weight models in production cited cost savings in a September 2026 ETR survey. We model your expected usage in discovery and show both numbers.

Can it run completely offline?

It can be done. Air-gapped setups need the model, serving software and search index delivered as offline packages, with a tested process for updates, which we scope case by case.

Do you work on our servers or in our cloud account?

Either. We deploy into your data centre or into your own AWS, Azure or Google Cloud account, so the infrastructure, keys and data stay under your control. We work only with the access you grant, and you can revoke it at any time.

Have something to build?

Tell us where things stand and what you need. A short message is enough to start.