A mid-size SaaS company gets 50,000 internal assistant requests a day. Employees ask about runbooks, failed file uploads, access rules, and escalation paths. The AI team wants a model to draft first responses, but legal has one hard rule: private records and incident history can't leave the controlled environment.
That's the real open-weight versus closed API decision. It isn't a culture war and it isn't a leaderboard race. It's a deployment choice: do you need control over weights, network boundary, customization, and rollout timing, or do you need managed frontier capability with the least infrastructure burden?
Decision frame: Set the data boundary and operating model before comparing model quality. A lane that violates either constraint isn't a candidate.
Open-weight families such as GLM-5.2, DeepSeek V4 Flash 0731, Qwen 3.6, and Llama 4 are credible production candidates, not side-project curiosities.[1][2][3][4] Closed APIs still matter because the best hosted models usually ship the newest reasoning, multimodal, structured-output, and long-context features first.[5][6][7][8]
The useful question is narrower: which lane fits this workload?
Use the right vocabulary
People often say "open source LLM" when they mean open-weight LLM.
| Term | Meaning |
|---|---|
| Open-weight | You can download and run the model weights |
| Closed API | You call a provider-hosted model and never receive the weights |
| Fully open source AI | This is the stronger claim. The Open Source AI Definition expects enough information to study, use, modify, and share the system, including details beyond a checkpoint alone[9] |
That distinction matters. "We can self-host it" and "we can reproduce how it was made" aren't the same claim.
A closed API gives you a managed runtime, model upgrades, hosted scaling, and vendor-owned constraints. An open-weight model gives you deployment control, checkpoint control, and operational responsibility. The trade is speed and convenience versus ownership and control.
What changed
Two shifts made the decision more serious.
First, open-weight quality rose. Two current releases deserve their own eval lanes:
| Property | GLM-5.2 | DeepSeek V4 Flash 0731 |
|---|---|---|
| Release and license | June 2026, MIT weights[10][1] | July 31, 2026 post-training release, MIT weights[11][2] |
| Model size | Z.AI labels it 744B total and about 40B active; Hugging Face checkpoint metadata counts about 753B[12][1] | 284B core and 13B active; Hugging Face reports about 304B for released artifact, whose card says DSpark module is attached[13][2] |
| Context and modality | 1M-token total text context; 128K output cap within it[14] | 1M-token total text context; 384K API output cap within it[15][16] |
| Technical idea | Latent KV compression and sparse attention reduce long-context work; IndexShare reuses sparse indices across four layers; MTP helps speculative decoding[17][10] | CSA and HCA mix sparse and compressed attention; attached DSpark module proposes tokens for speculative decoding[13][18] |
| Strong eval lane | Long-horizon coding, repository-scale reasoning, tools, and MCP workflows[1][14] | Fast agentic coding, tool use, repository work, and long outputs[2] |
| Main constraint | Roughly 1.51 TB of raw BF16 weights before runtime overhead makes self-hosting a cluster project[1] | Text-only, complex serving stack, and lower knowledge and hardest-agent ceiling than V4-Pro[19][13] |
Parameter totals use different counting boundaries. Treat 753B and 304B as measurements of different released artifacts, not a direct architecture comparison. Both are giant MoE systems. Active parameters help explain per-token compute, but every expert weight still needs storage and a serving plan.[1][14][2][11][13]
Official benchmark tables are useful screening evidence, not a routing decision. GLM-5.2 reports 62.1 on SWE-bench Pro and 81.0 on Terminal-Bench 2.1. DeepSeek V4 Flash 0731 reports 54.2 on NL2Repo, 54.4 on DeepSWE, and 82.7 on Terminal-Bench 2.1. Harnesses, effort settings, context limits, and some datasets differ. Run both on the same task pack before choosing one.[1][2]
Qwen 3.6 and Llama 4 remain useful candidates across local, edge, multimodal, and data-center deployments.[3][4] Neither GLM-5.2 nor Flash 0731 always wins, but a 2026 open-weight shortlist that omits both starts from an outdated field.
The runtime shift
Second, the serving stack matured. PagedAttention-style key-value (KV) cache management, vLLM, TensorRT-LLM, and serving quantization methods such as AWQ made self-hosting less exotic.[20][21][22][23] QLoRA belongs on the customization side: it trains LoRA adapters through a frozen 4-bit base model, reducing fine-tuning memory rather than defining an inference-serving format.[24] Starting a container is no longer the main problem. Keeping time to first token, throughput, utilization, eval quality, and rollback safety in range at the same time is the work.
Closed APIs still keep important advantages. Anthropic has documented 1M context availability for Claude Opus and Sonnet tiers, Google packages Gemini 3 with managed multimodal and platform features, and OpenAI exposes structured outputs as a first-party API primitive.[6][7][25][8] Those managed features can matter more than raw model ownership.
Where open-weight wins
Open-weight is the default lane when the workload has a hard data boundary. If prompts, outputs, logs, or retrieved documents can't leave your environment, self-hosting or a private managed deployment keeps the model inside the allowed network. That matters for regulated data, internal incident history, customer support transcripts, source code, and security investigations.
Privacy boundary: If private records can't leave your controlled environment, a public closed API fits only after contract terms, redaction, logging policy, and audit paths satisfy that boundary.
It also wins when deep customization is part of the product. With weights, you can fine-tune with LoRA or QLoRA, distill behavior into smaller task models, pin exact checkpoints, and run specialized evals against your own failures.[24] A broad frontier model may be stronger in general, but a tuned open model can be better for a narrow enterprise task: invoice extraction, security alert triage, policy lookup, or runbook classification.
Open-weight can also win on steady economics. Closed APIs are excellent for low or spiky traffic because usage billing avoids idle GPUs. Self-hosting becomes interesting when traffic is predictable enough to keep a serving fleet busy. The break-even point isn't universal, but the shape is stable:
| Cost side | What to count |
|---|---|
| Closed API | input tokens, output tokens, cache discounts, batch tiers, provider minimums |
| Open-weight | GPU rental or reservations, storage, networking, utilization, deployment work, monitoring, incident response |
The common mistake is comparing token billing to GPU rental alone. That makes self-hosting look cheaper than the full hosted bill.
Where closed APIs win
Closed APIs are still the fastest path to production. You can pick a model, call an endpoint, and avoid GPU capacity planning, model loading, memory tuning, batching, autoscaling, and inference on-call. For a small team with uncertain traffic, that simplicity is often the correct choice.
They also win when managed frontier features are the product. If you need the newest model immediately, hosted long context, strong multimodal behavior, managed tool calling, context caching, or structured-output guarantees, a provider API is usually faster than assembling and operating the same surface yourself.[26][6][25][8]
Closed APIs don't remove responsibility. You still need privacy review, redaction policy, evals, retries, observability, cost controls, and fallbacks. But they move the inference platform burden to the provider.
Worked example: private knowledge assistant
Return to the SaaS company. The assistant handles 50,000 requests per day. Each request averages 400 input tokens and 300 output tokens. Private data must stay inside the company's virtual private cloud.
Using GPT-5.6 Sol pricing as one current hosted example, at $5.00 per million input tokens and $30.00 per million output tokens for requests up to 272K input tokens, the math is:[5]
- input: 50,000 requests x 400 tokens = 20M input tokens/day = $100/day
- output: 50,000 requests x 300 tokens = 15M output tokens/day = $450/day
- monthly API cost: about $16,500 before batch discounts, retries, and compliance overhead
The arithmetic assumes no cache-eligible shared prefix. Prompt caching starts at 1,024 input tokens, so the 400-token request body qualifies only if reusable system instructions, tools, or other prefix content pushes the full prompt past that threshold. GPT-5.6 Sol applies higher rates to the full request above 272K input tokens and bills cache writes separately. A production calculator must price complete request shape, cache behavior, and monthly volume together.[5][27]
Keep the self-hosted side as variables until you price a specific provider, region, reservation term, and operations model:
| Variable | Meaning |
|---|---|
gpu_count * hourly_rate * 730 | monthly raw compute at full reservation |
served_tokens / theoretical_tokens | utilization discount or penalty |
storage + networking | model files, logs, artifacts, ingress, egress |
ops_hours * loaded_hourly_cost | deploys, monitoring, incident response, upgrades |
eval_and_fallback_cost | quality checks, retries, hosted exception calls |
Cost model caveat: GPU rental is one row, not the bill. Self-hosting becomes attractive when utilization is high enough to pay for compute, storage, monitoring, upgrades, and on-call ownership.
Those numbers aren't a universal price table. Replace them with your provider, region, discounts, cache hit rate, and staffing cost. Use the shape:
| Factor | Closed API path | Private managed closed | Open-weight self-host |
|---|---|---|---|
| Data boundary | Traffic goes to the provider's public endpoint unless contract, redaction, and region controls say otherwise | VPC / private-link / region-locked managed closed lanes keep data on a private path without you running weights | Data stays inside the environment you operate |
| Setup time | Days | Days to a few weeks | Weeks |
| Customization | Prompting, tools, hosted tuning where available | Same closed-model surface, with enterprise network controls | Weight-level adaptation, distillation, checkpoint pinning |
| Cost shape | Usage based, good for uncertain traffic | Usage based plus private networking / commitment terms | Fixed fleet, better when utilization is high |
| Operational burden | Lowest | Medium (network + contract, not GPU fleet) | Highest |
| Upgrade control | Provider controls model rollout | Provider controls model rollout | Team pins and rolls forward deliberately |
In this scenario, open-weight is plausible because privacy is hard and traffic is steady. If traffic were spiky, if the team lacked inference ownership, or if provider caching cut the bill materially, the closed API lane could still win.
Decide by constraint order
Start with the constraint, not model rank.
| Constraint | Default lane | Reason |
|---|---|---|
| Strict data residency or private cloud only | Open-weight or private managed closed | Deployment boundary is the hard requirement |
| Deep fine-tuning or distillation | Open-weight | Weight-level control matters |
| Small team, fast launch, unknown traffic | Closed API | Lowest platform burden |
| Need latest frontier model immediately | Closed API | Provider ships first |
| Need managed multimodal, structured-output, or turnkey ops surface | Closed API | Hosted product surface is mature |
| Need raw 1M text context alone | Open-weight or closed | GLM-5.2 and DeepSeek V4 Flash 0731 both advertise 1M windows; long context is not closed-only, but neither model is multimodal[14][19] |
| High steady traffic and infra ownership | Open-weight or hybrid | Fleet utilization can beat token billing |
| Retrieval-heavy stack | Often hybrid | Open embeddings/rerankers can pair with closed generation |
This is also the strongest interview framing. Don't start with "model X is best." Start with data boundary, quality target, latency target, traffic shape, and who owns inference on-call.
Hybrid usually wins
Most serious systems don't stay purely open or purely closed. A strong 2026 design often looks like this:
- open embeddings and rerankers for retrieval
- open-weight task models for high-volume extraction, classification, or narrow generation
- closed frontier models for hard reasoning, difficult coding, multimodal flows, or high-value exceptions
The MTEB work helped make multi-task embedding evaluation mainstream, and it's one reason teams treat retrieval separately from generation.[28] You can run local embeddings and reranking while keeping final synthesis on a managed model when the task justifies it.
Hybrid only works when escalation is explicit. If every request silently falls through to the hosted frontier model, the open lane isn't doing architectural work. Track fallback rate, quality lift, latency, cost per request, and redaction failures.
Mistakes to avoid
| Mistake | Symptom | Fix |
|---|---|---|
| Treating the decision as all open or all closed | Team argues ideology instead of workload fit | Route each workload by boundary, quality need, traffic shape, and staffing |
| Treating download as deployment | Someone calls an open-weight model "free" | Count serving, monitoring, security review, rollback, and on-call |
| Picking from public benchmarks only | Model wins leaderboard but fails internal requests | Build a golden set of 50 to 100 real examples and score quality, latency, and cost |
| Ignoring KV cache memory | Model fits at load time but fails under concurrent long context | Size context, concurrency, and parameter count together |
| Forgetting safety ownership | Self-hosted path lacks filtering, abuse monitoring, or policy checks | Add guardrails, evals, and audit logs before broad rollout.[29][30] |
Practical rule
Closed APIs fit speed, managed capability, and low operational burden. Open-weight models fit boundary control, customization, checkpoint ownership, or steady utilization. Hybrid routing is the answer when either extreme is too blunt.
For the SaaS assistant, the first architecture would likely keep retrieval and routine answers inside the controlled environment, then send only allowlisted, redacted, high-value exceptions to a closed frontier API. The team keeps privacy where it's non-negotiable and managed quality where it changes the answer.