Open-Source LLMs FAQ
Open-weight models are a control and residency choice — not automatically cheaper, and not automatically worse.
Open-source and open-weight LLMs are models you can host yourself, inspect under their licence, and often fine-tune without sending data to a frontier API. They win on data residency, control, and steady high utilisation; frontier APIs still win on peak quality, burst capacity, and ops you do not want to own.
What “open-source LLM” actually means
If you cannot run the weights in your VPC under a licence your counsel accepts, you do not have an open option — you have a brochure.
What is an open-source LLM?
In practice people mean a model whose weights you can download and run, often with code to train or serve it, under a published licence. Some are OSI-open; many are “open weights” with use restrictions. The operational fact is: you can host it. Quality, context length, and tool-use still vary widely. Treat the name as a family, then measure the specific checkpoint on your suite.
Are open weights the same as open source?
No. Open source (OSI) means the licence meets those criteria for code and, sometimes, model artefacts. Open weights means you can download parameters, possibly under a custom acceptable-use licence that limits competitors, volume, or domains. Training data is often not released either way. We will name the licence on the checkpoint we propose; we will not call a restricted weight file “fully open.”
Llama vs GPT — when does open-source win?
Open-weight Llama-class models win when data residency, pin-able checkpoints, or fine-tunes you can export matter more than the last point of quality, or when utilisation makes GPUs cheaper than APIs. GPT-class frontier APIs win on many hard reasoning and tool-use evals, and on burst. Hybrid is common. “Llama vs GPT” without a task and a constraint is a fan argument. We run both against your golden set.
What is the best open-source LLM for enterprise?
There isn’t a single best. The right checkpoint is the one that passes your evals inside your latency and hardware budget, under a licence you can accept. Rankings rotate every month. We shortlist two or three current candidates, freeze versions, and pick with the suite. A blog’s “best of 2026” is not a procurement artefact. If a vendor-only API wins the suite and residency allows it, we will say so.
Can we self-host an LLM?
Yes. Production self-hosting is a serving stack (typically vLLM or equivalent), GPUs or a managed inference tenancy you control, networking, secrets, autoscaling, and evals — inside your VPC or on-prem. A laptop Ollama demo is not that stack. The how-to on deploying an LLM in a private cloud is the playbook. We implement it; we do not operate a public model API.
Are open-source models worse at following instructions?
Often, on the frontier of tool use and long-horizon reasoning; not always on narrow, well-specified tasks, especially after a LoRA. Instruction quality is a property of the checkpoint and the prompt overlay, not of the word “open.” If the suite says the open model misses tool schemas, we keep an API on that slice. Blanket “open is worse” is as sloppy as blanket “open is enough.”
When do frontier APIs still win?
When the golden set is still out of reach for hostable checkpoints, when load is spiky and GPUs would idle, when you do not want to staff inference, or when multimodal or reasoning features you need are not in the open candidate. Zero-retention and regional APIs close some residency gaps — not all. We mix: API where it wins, open-weight where control wins. You pay each meter directly.
How open-weight models get into production
The download is an afternoon. The serving contract is the four weeks.
How do you deploy an open-source LLM in a private cloud?
Package a serving image (vLLM/TGI-class), pin the checkpoint, put it on GPUs in your VPC, front it with your identity and a queue, and wrap traces plus the eval suite. Networking and secrets follow your standards; training data and prompts stay in-tenancy. Quantization is a measured trade on the same suite. The four-week implementation includes this path when residency requires it.
What hardware do we need to self-host?
Enough GPU memory for the chosen checkpoint and context at the quantization you eval’d, plus headroom for KV cache and batching. Exact cards and gigabytes go stale every generation, so we size from the model card and a load test on your traffic, not from a blog’s shopping list. CPU-only is for tiny models or batch that can wait. If you cannot get accelerators, an API or a managed tenancy is the honest path.
Are we allowed to use Llama (or similar) in a commercial product?
Often yes, under the licence that ships with that checkpoint — which may include acceptable-use rules and volume or competitor clauses. That is not legal advice. Your counsel reads the licence; we will not hide a restriction we know about. Apache-licensed models are simpler. If a licence would block your distribution model, we pick another checkpoint rather than shrug.
Is fine-tuning easier on open-source models than on APIs?
You usually get an artefact you can export (a LoRA or merged weights) and train inside the perimeter. Vendor fine-tune APIs can be easier operationally and may not let you take weights home. For regulated data, open-weight plus PEFT in your VPC is the default we propose. The fine-tuning FAQ covers data and forgetting; here the point is ownership of the adapter.
How do you evaluate open-source vs closed models?
The same golden set, the same rubrics, frozen versions, plus latency and unit cost on that hardware or API. Public leaderboards are a shortlist, not a winner. Include tool-use and refusal, not only MMLU-like trivia. Re-run when either side ships a new checkpoint. If we cannot define success, we will not run a bake-off to decorate a slide.
Can we mix open-source models and frontier APIs?
Yes, and we often do. A router sends bulk or sensitive traffic to the VPC model and hard cases to an API, with pins and evals on both paths. Data-flow policy says what may leave. You pay both providers or clouds directly; no token markup. Mixing without a router is two unmanaged stubs. Mixing with a router is an architecture.
What do you need from our team to self-host?
A platform engineer who can provision GPUs or a managed inference service in your account, the residency constraint in writing, and a process owner for the eval set. Weekly review. If GPU quota does not exist, week one is procurement or a managed tenancy, not serving. We will not pretend a CPU VM will host a 70B model for interactive chat.
Operating open-weight models
Checkpoints drift, cards change, and idle GPUs cost money. Treat this as a production service.
Does self-hosting mean our data is never used for training?
Serving in your VPC means prompts do not go to a frontier trainer by default. You still need a policy: no logging raw PII, no using prod traffic to train a shared model, and a written rule if you later fine-tune on logs. API vendors offer zero-retention tiers; read them. Self-hosting is the strongest technical control; it is not automatic compliance. We implement the control; we do not issue a certificate.
Who owns the served model after handover?
You do. Checkpoints, adapters, serving configs, evals, and runbooks are in your registry. Thirty days of on-call is included. We do not keep a shadow copy required to run inference. If a base model’s licence requires attribution or a use report, that stays on your compliance list, not ours.
How do you keep open-source models updated?
You don’t auto-pull latest. Pin a digest, subscribe to upstream security and quality notes, and promote a new checkpoint through the eval suite and a canary. Open-weight releases can change behaviour as hard as a vendor silent update — you just get to choose when. A weekly “apt upgrade” mentality on weights is how you surprise production.
What actually drives self-host cost and speed?
GPU time (busy and idle), memory for weights plus KV cache, batching, context length, and quantization. Continuous batching and a sensible max context do more than a slightly smaller model with a huge window. We tune against p95 and the suite, not against a theoretical tokens-per-second tweet. You pay the cloud bill; we do not markup it.
How long to a self-hosted production model?
A serving path for one workflow can sit inside the four-week implementation if GPUs and the golden set exist. Hardening — autoscaling, multi-AZ, runbooks, quantization bake-off — is what fills the calendar. A container that answers in staging on day four is not production. If quota is the blocker, we say so in week one instead of burning the month on YAML.
Do we still need observability if the model is in our VPC?
Yes. Traces, token/step counts, quality scores, and GPU utilisation. Inside-the-perimeter is not “we will notice.” Silent quality drift happens on your checkpoint too. PII redaction in logs still applies. The observability how-to is the same discipline as for APIs; the exporter points at your cluster.
When not to self-host
Control is expensive if nobody will operate the GPUs. Pay an API when that is the constraint.
When should we not self-host an open-source LLM?
When volume is low or spiky, when you cannot staff GPUs, when the suite still needs a frontier model, or when procurement cannot provide accelerators in time. Also when the licence of the only good checkpoint conflicts with your use. An unused GPU cluster is a cost decision you can reverse by using an API; a failed residency review is not. We will recommend the API path in writing if that is the fit.
Are open-source models better for RAG, agents, or classification?
Classification and extraction with a schema often land well on smaller open models, especially after a LoRA. RAG generators can be mid-size if retrieval is strong. Agents that need reliable tool calling still often keep a frontier model on that slice. Measure per workload. Moving everything to one 70B in the VPC because “we standardised” is how you overpay and under-quality at once.
Can open-source LLMs run air-gapped?
Yes, that is a primary reason to choose them. You still need a way to patch serving images and to import new checkpoints through your software-supply process. Evals run inside. If a workflow secretly needs web-scale knowledge the weights lack, air-gap plus RAG on your corpus is the design — not hoping the model memorised your plant manual. We have implemented VPC and on-prem; we do not invent a disconnected magic model.
If the model is free, why do we need an implementation partner?
Weights are not a system. Serving, retrieval, evals, permissions, and a runbook are the system, and they are what fail. We build those in four weeks inside your cloud and leave you with the artefact. We are not a model vendor and not a GPU reseller. “Free model” with no evals is how a demo becomes an incident.
What usually fails in the first 90 days of self-hosting?
No pin (someone pulled latest), no eval gate, GPUs idle or oversubscribed with no autoscaling policy, and a prompt copied from a closed model that this checkpoint does not obey. Quantizing after go-live without a suite is the other classic. Rescue starts by freezing a digest and measuring. The private-cloud how-to is the mechanical checklist.
Open-weight models (Llama-class and others) let you serve inside a VPC, fine-tune with LoRA, and keep prompts off a vendor’s trainer. “Open source” and “open weights” are not the same; licences differ and are not legal advice from us. Frontier APIs still lead many evals and absorb burst. Production self-hosting is vLLM-class serving, quantization, autoscaling, evals, and pins — not a desktop download. ReinforcedX is model-agnostic: we implement mixed stacks in your cloud, you pay the GPU or API meter, and you own the checkpoints. Choose with a golden set, not a brand war.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.