How to Deploy an LLM in a Private Cloud
How to deploy an LLM on-prem or in a private VPC with vLLM/TGI, networking, secrets, and a perimeter that keeps prompts and weights inside the network in 2026.
To deploy an LLM in a private cloud, serve pinned weights with vLLM or TGI on GPUs inside your VPC or on-prem network, expose an internal authenticated endpoint, and keep prompts, traces, and weights from leaving the perimeter. Self-hosting Llama (or any open-source or client fine-tune) is an operations problem — networking, secrets, pins, and evals — not a one-line Docker run.
What you’ll build
- An OpenAI-compatible serving endpoint (vLLM or TGI) reachable only on a private network
- Weights pulled from a private registry, not from a public download on boot
- Secrets, API keys, and TLS material in a manager — never in images or env dumps
- Autoscaling and KV-cache settings that meet a written p95 latency and tokens/sec target
- Runbooks for pin, rollback, and incident response; client owns weights, configs, and evals
Before you start
- 01A VPC or on-prem network with GPU capacity sized for the model and concurrency you named
- 02A license to serve the chosen weights (Llama, Mistral, or a client fine-tune you own)
- 03Private container registry, secret manager, and an identity provider for operators and apps
- 04A non-GPU staging path for CI of configs, plus a canary GPU pool
- 05Written data-residency rules: prompts, logs, and traces must not leave the perimeter
Key takeaways
- 01
Private LLM serving means weights, prompts, and traces stay in your VPC or datacenter; a public API with a DPA is a different product.
- 02
vLLM and TGI are the usual inference engines: continuous batching, paged KV cache, OpenAI-compatible routes.
- 03
Pin image, engine version, weights checksum, and tokenizer; cold-start from a private registry only.
- 04
Authn/z belongs on the gateway: internal LB, mTLS or OIDC, per-app quotas — the model server should not be world-reachable.
- 05
Measure p95 latency and tokens/sec on a replay of production prompts before cutover; shadow the previous pin.
When to deploy an LLM on-prem or in a VPC
Deploy in a private cloud when residency, air-gap, or cost at steady volume beats a public API. Self-host Llama (or Mistral, or a client fine-tune) when you must own the weights and the prompt never leaves the building. A provider-hosted “private” endpoint in their region is not the same as your VPC; read the data-flow diagram before you call it on-prem.
If the workload is bursty, the team cannot operate GPUs, or you need a model you are not licensed to serve, stay on a contractual API with no token markup and zero-retention options. Model-agnostic delivery means picking that honestly. Private serving is for the cases where the perimeter is the requirement.
vLLM, TGI, and the serving contract
vLLM and Text Generation Inference (TGI) are the common production engines: paged KV cache, continuous batching, and OpenAI-style `/v1/chat/completions`. Your apps should speak that contract so you can swap engines or add a second model without rewriting callers. TensorRT-LLM and other runtimes are valid if the ops team can pin and rollback them the same way.
The serving contract also includes tokenizer, chat template, and stop tokens. A mismatch with the training template destroys quality and looks like “the private Llama is worse than the API.” Store the template next to the weights. For adapters (LoRA), decide whether you merge at publish time or load adapters at runtime; both need a registry entry.
Networking, secrets, and the perimeter
Place GPU nodes in private subnets. Ingress only from the application VPC or on-prem CIDR via an internal load balancer. Use mTLS or a gateway that checks OIDC service accounts. Disable public SSH. Outbound should be deny-by-default: image pulls from the private registry, logs to the internal collector, metrics to the internal Prometheus. No training data, prompts, or weights to the public internet.
Secrets: TLS keys, registry credentials, and any optional API fallback keys live in the cloud secret manager or Vault, rotated, and not interpolated into Dockerfiles. Disk encryption on weight volumes. Operator access through the IdP with break-glass logged. If you need a vendor to debug, that is a ticketed tunnel, not a standing egress rule.
Throughput, KV cache, and quantization
Throughput is tokens/sec at a given concurrency and context. Latency is p95 time-to-first-token and time-to-complete on a replay of real prompts — synthetic 128-token prompts will lie. Continuous batching and prefix caching help multi-turn agents; size KV cache so your max context still fits with the concurrency you sell internally.
Quantization (8-bit, 4-bit) trades quality and sometimes latency for occupancy. Only ship a quantized pin if the golden set (task metric, not a public leaderboard) stays within the bound. Autoscaling needs a warm pool; a 70B cold start is not an SLO-friendly scale-from-zero.
Pins, canaries, and runbooks
Pin: container digest, engine version, weight checksum, tokenizer, chat template, quantization. Promote with a canary replica. Rollback is switching the gateway to the previous pin, not rebuilding. Health checks should fail the instance if GPU ECC or CUDA errors appear, not only if port 8000 is open.
Handover artifacts: architecture diagram, network policy, secret list, pin table, golden-set command, latency dashboard, on-call notes. The client owns weights, datasets, evals, and runbooks. ReinforcedX can run a four-week implementation inside the client cloud — discovery, environments, shadow-mode pilot, handover — without taking the prompts out of the VPC.
- Weights: private registry, checksum, license, no public boot fetch
- Network: private subnets, internal LB, mTLS/OIDC, deny-by-default egress
- Serve: vLLM/TGI, pinned template, measured p95 and tokens/sec
- Ops: canary pin, redacted in-region logs, owned runbooks
What fails after the demo
Context lengths that fit in a notebook OOM under concurrent load. Chat templates that differ from training. Log pipelines that ship prompts to a SaaS APM in another region. A Hugging Face token left in a node image. A fallback to a public API that silently turns on when GPUs throttle — and takes residency with it.
Write the fallback policy. If GPUs die, do you queue, refuse, or fail to a contractual in-region API? Each choice is valid; an implicit public fallback is not.
Step-by-step build
- 1
Choose model, license, and GPU shape
Pick a model you may serve (open weights or a client fine-tune). Size GPUs for weights + KV cache at your max context and concurrency. Write the latency/throughput target. If a frontier API already meets residency via a contractual VPC, that can be the right call — this guide is for when weights must run on your metal.
- 2
Mirror weights into a private registry
Download once in a controlled job, verify checksums, scan, and store in an internal object bucket or model registry. Record license and revision. Training data and fine-tune datasets stay in the same perimeter; they do not cross to a public trainer.
- 3
Stand up vLLM or TGI on private compute
Run the engine in Kubernetes or VMs with GPU operators, in private subnets. Mount weights from the registry. Enable tensor parallelism only as required. Expose the OpenAI-compatible HTTP API on a ClusterIP or internal NLB, not a public IP.
- 4
Lock networking, secrets, and logs
Security groups to app subnets only. Gateway auth (mTLS or OIDC). TLS certs and any Hugging Face tokens (if still needed for gated mirrors) in a secret manager. Prompt logs sampled and redacted; traces stay in-region. No egress to public model APIs unless a documented fallback exists.
- 5
Tune serving and autoscaling
Set max context, batch size, GPU memory utilization, and prefix/prompt cache as the engine allows. Autoscale on queue depth or GPU utilization with a cold-start budget. Quantize (AWQ/GPTQ/FP8) only after golden-set evals match the unquantized pin within the bound you wrote.
- 6
Eval, canary, and handover
Replay a golden set and a latency trace against the private endpoint. Canary a fraction of internal traffic. Keep the previous image and weights warm. Hand over runbooks, pins, and evals; the client owns the weights. A four-week implementation typically lands env in week 2, shadow in week 3, handover in week 4, then 30 days on-call.
Common pitfalls
The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.
Downloading weights from the public internet at boot
That is a supply-chain and residency failure. Mirror the approved revision into a private registry, verify checksums, and pin. Nodes should start without Hugging Face tokens.
Putting the inference endpoint on a public load balancer
Private cloud means private. Use internal load balancers, security groups, and mTLS or gateway auth. “VPC but 0.0.0.0/0 on 8000” is not a private LLM.
Logging full prompts by default
Prompt logs are training-data-shaped PII. Redact, sample, and keep traces in-region. If a vendor support tunnel needs logs, that is a residency exception you must write down.
One giant GPU node, no canary
A bad pin takes the only box down. Keep a canary replica on the previous image and weights so you can fail back without a two-hour cold start.
Ignoring KV cache and batching until prod
vLLM throughput comes from continuous batching and KV cache. Default settings on a 70B will miss latency SLOs that a spreadsheet assumed. Measure tokens/sec and p95 on a replay of real prompts.