← All insights

Route by workload: Enterprise choices for open and closed LLMs

Route by workload: Enterprise choices for open and closed LLMs

Route by workload: Enterprise choices for open and closed LLMs

Decorative open and closed LLM title card

Choose open weights and self-hosting when you need full control, data residency, or cost efficiency at high volume. Choose closed APIs when frontier capability, speed to production, and minimal operational overhead matter more than control. Most enterprises end up running both, routing routine tasks to self-hosted open models and reserving closed frontier models for complex or high-stakes work.


TL;DR:

  • Open-weight models offer greater control over data residency and fine-tuning, but require significant operational effort and accurate licensing compliance.
  • Closed APIs provide easier deployment, scalability, and access to advanced capabilities, making them ideal for low-volume use cases or teams lacking MLOps resources.
  • Cost-effectiveness shifts around 10 to 30 million tokens per day, with self-hosting becoming more economical at higher volumes while offering 85 to 95% of closed-model quality.
  • Deployments often use a hybrid approach, routing routine tasks to open models and complex or sensitive work to closed APIs to optimize cost, control, and performance.
  • Regulatory compliance and data residency needs can make self-hosting the only viable option when contracts or legal standards restrict third-party data handling.

Real-studios
Plan Your AI Deployment Strategy
Build tailored AI agents, automations, and integrations that help your business modernize customer interactions and workflows.
Explore AI consultancy

Table of Contents

Open vs closed LLMs: what actually separates them

The distinction that matters for procurement isn’t “open” versus “closed” as marketing labels. It’s what you’re actually allowed to do with the model once you have it.

An open-weight model ships its trained parameters for anyone to download, run, and modify. Meta’s Llama family, Mistral’s models, and DeepSeek’s releases all fall into this bracket. You get the weights, often a technical report, and usually permission to fine-tune and self-host. What you frequently don’t get is the training data, the full training code, or an unrestricted commercial licence. That’s why the term “open-weight” is more accurate than “open-source” for most of these releases: Meta’s Llama community licence, for instance, carries usage caps and conditions that a true open-source licence like Apache 2.0 does not impose. Mistral, by contrast, releases several models under genuinely permissive terms, while other Mistral variants carry custom commercial restrictions. The licence text is not boilerplate here. It decides whether you can deploy the model behind a paid product without a separate agreement.

A closed or proprietary model gives you none of that. OpenAI’s GPT family, Anthropic’s Claude, and Google’s Gemini are accessed exclusively through an API or managed console. You never see the weights, the training data, or the underlying architecture in detail. What you’re buying is an outcome: a response to a prompt, delivered at a service level, with the vendor responsible for everything underneath it.

This access gradient has three practical dimensions worth separating out:

  • Weights: open-weight models let you inspect, quantise, and modify the parameters directly; closed models give you none of this.
  • Training code and data: fully open-source projects (a smaller category than open-weight releases) publish enough to reproduce training; most open-weight vendors do not.
  • Modification rights: self-hosted open models can be fine-tuned on proprietary data without it leaving your infrastructure; closed models generally only support fine-tuning through the vendor’s own managed process, if at all.

An academic review of open-source LLM initiatives frames this well: openness increases reproducibility and lets the wider community iterate on a model’s weaknesses, while closed labs retain centralised control and, for now, a lead on some of the hardest reasoning and multimodal tasks. Neither side is simply “better.” They optimise for different things, and the right choice depends on what your workload actually needs.

Why open-weight models win on control and cost

Self-hosting an open-weight model hands you three things a closed API cannot: data residency, fine-tuning depth, and long-term stability against a vendor’s roadmap decisions.

Data residency is the clearest case. If a prompt never leaves your VPC, you sidestep a large category of data-transfer and retention questions before they start. Fine-tuning goes deeper too, because you can adjust weights directly on proprietary data rather than relying on a vendor’s fine-tuning API and its constraints. And because you hold the weights, a vendor deprecating a model or changing its pricing overnight can’t force an unplanned migration. Microsoft’s own guidance on enterprise model selection makes the same point: open-weight deployment gives you control over hosting and customisation, but it shifts real operational weight onto your team.

That operational weight is the trade-off nobody advertises. Running an open-weight model in production means:

  • GPU provisioning and orchestration, whether on-premises or in a cloud GPU tier
  • Ongoing MLOps: monitoring drift, managing versions, patching inference servers
  • Security ownership: you’re responsible for access controls, model isolation, and vulnerability patching that a closed vendor previously handled for you
  • Capacity planning for traffic spikes, since there’s no managed autoscaling unless you build it

Pro Tip: Before committing to self-hosting, price the fully-loaded cost including one or two dedicated MLOps engineers, not just GPU rental. Teams routinely underestimate the maintenance and versioning burden and end up with a break-even point far higher than their initial spreadsheet suggested.

Licensing needs a second look here, too. “Open-weight” is not synonymous with “unrestricted.” Read the licence attached to any model before deploying it commercially: some carry usage caps tied to your company’s size or user count, others restrict use in competing products outright. The savings from self-hosting evaporate quickly if a licence violation forces a late migration.

Why closed models remain the pragmatic default

Closed APIs solve the problem most teams actually have, which is shipping something reliable without building an inference platform first.

The managed scaling is the headline benefit: you send a request, the vendor handles the GPU allocation, load balancing, and failover, and you get a response with a contractual service level attached. That SLA matters more than it sounds. When a customer-facing agent goes down, “the vendor is investigating” is a very different conversation with your leadership than “our GPU cluster fell over and nobody who understands it is on call.”

Closed vendors also carry frontier capability further than most open-weight releases manage. Multimodal reasoning, long-context handling, and complex tool-use chains tend to appear first in GPT, Claude, and Gemini releases before open-weight equivalents catch up. That gap has narrowed considerably, but it hasn’t closed.

The practical benefits stack up:

  • Zero infrastructure to provision or patch, since the vendor’s platform absorbs that entirely
  • Built-in safety tuning and moderation layers that would take a dedicated team months to replicate
  • Enterprise agreements covering data handling, uptime, and support escalation
  • Predictable developer experience: one API, consistent versioning, documented rate limits

That last point matters more than it gets credit for. An engineering team of three can ship a working RAG-backed support agent against Claude’s or GPT’s API in a sprint or two. The same team attempting to self-host and fine-tune an open-weight model of comparable quality is looking at weeks of infrastructure work before the first prompt goes to a real customer. For a startup or a team without dedicated MLOps capacity, that speed differential often decides the question before cost enters the conversation at all.

What does it cost to run each option at scale?

Cost is where the open-versus-closed debate actually gets decided for most enterprises, and the answer depends heavily on volume.

At low token volume, closed APIs almost always win on total cost of ownership. You’re paying per token with no infrastructure to maintain, and at low traffic the fixed cost of running your own GPU cluster simply isn’t justified. As volume climbs, the economics flip. Analysis of self-hosting break-even points places the crossover where self-hosting starts winning at roughly 10 to 30 million tokens per day, though the exact figure shifts with model size, GPU pricing, and how efficiently you’re utilising your hardware.

A rough way to think about the three bands:

  1. Under 5 million tokens/day: stick with a closed API. The GPU and engineering overhead of self-hosting will exceed anything you’d spend on per-token API pricing at this volume.
  2. 5 to 30 million tokens/day: this is the genuine decision zone. Model your fully-loaded self-hosting cost (GPU instances, one to two MLOps FTEs, monitoring tooling) against your actual API spend at current usage, and re-run the model with realistic growth assumptions.
  3. Above 30 million tokens/day: self-hosting an open-weight model, particularly a quantised version served through an efficient inference engine, typically becomes materially cheaper, assuming you already have or can justify the MLOps capacity.

Statistic: Top open-weight models now deliver roughly 85 to 95% of closed-model quality on many production tasks, at a fraction of the per-token cost. That gap is small enough that, once volume crosses the crossover point, sticking with a closed API purely for a marginal quality edge is a hard sell to a finance team.

The hidden risk in this maths is vendor change. A closed provider can raise prices, deprecate a model version, or tighten rate limits with little warning, and your TCO model needs a contingency line for that. Self-hosting isn’t immune either. GPU pricing fluctuates, and a model you’ve fine-tuned heavily today may need retraining against a better base model in twelve months. Build both risks into whichever side of the line you land on.

API, self-hosted, or hybrid: which deployment pattern fits?

Three deployment patterns cover almost every enterprise LLM architecture in production today, and the right one usually depends on latency, compliance, and how predictable your traffic is.

Pure API is the simplest: every request goes to a closed vendor’s endpoint. It’s the fastest path to production and requires no inference infrastructure, but you’re fully dependent on that vendor’s uptime, pricing, and rate limits.

Self-hosted means running an open-weight model on your own infrastructure, whether that’s on-premises GPUs or a rented cloud GPU tier. This is where tooling has matured fastest. Serving frameworks like vLLM handle high-throughput batched inference efficiently, while Ollama has become the standard for lightweight local and small-team deployment of quantised models. Both meaningfully lower the engineering lift that used to make self-hosting impractical for smaller teams. The Allen Institute’s work on inference tooling shows how far quantised open-weight models paired with efficient serving frameworks have come on cost and latency in the last two years.

Hybrid routing splits traffic across both.

Deployment decisions usually come down to a handful of hard constraints:

  • Latency: edge or on-device inference requirements often force self-hosting, since a round trip to an external API adds latency you can’t recover.
  • Offline or air-gapped environments: some regulated or industrial settings simply can’t call an external API, full stop.
  • Traffic predictability: highly variable traffic favours managed API scaling; steady, high-volume traffic favours self-hosting where the fixed cost gets amortised.
  • Team capacity: self-hosting without dedicated MLOps support tends to accumulate technical debt fast, regardless of how good the model is.

Most mature enterprise deployments aren’t choosing one pattern outright. They’re building a router and letting the workload decide.

What compliance requirements actually force your hand?

Compliance is the one factor that can override a clean cost or capability argument entirely, because some regulatory positions simply don’t leave you a choice.

Data residency is the clearest trigger. If your regulator or contract terms require that customer data never leaves a specific jurisdiction or your own infrastructure perimeter, a closed API sending prompts to a third-party data centre may be a non-starter regardless of how good the model is. Analysis aimed at UK engineering leaders notes that keeping inference inside your own perimeter is often the decisive factor for regulated industries under GDPR and similar frameworks, not model quality or cost.

Before you sign with any closed vendor, get clear answers on these points:

  • Data retention policy: does the vendor retain prompts and outputs, for how long, and can you contractually mandate deletion?
  • Contractual mitigations: does the vendor offer a Business Associate Agreement (BAA) for healthcare data, or SOC 2 Type II attestation for general enterprise assurance?
  • Sub-processor transparency: does the vendor disclose which third parties (cloud infrastructure, logging tools) touch your data downstream?
  • Regional processing guarantees: is there a contractual, not just marketing, guarantee that inference happens in a specific region?

Self-hosting doesn’t remove governance work, it just relocates it. You inherit responsibility for prompt injection defences, tightly scoped access controls on the model endpoint, and audit logging of every inference call. None of that comes bundled with an open-weight download.

Pro Tip: Treat your compliance review as a gating step before the cost or performance comparison, not after. A model that scores brilliantly on your benchmark is worthless if legal can’t sign off on where the data goes.

When a regulator’s residency requirement is genuinely non-negotiable, and no closed vendor can contractually guarantee in-perimeter processing, self-hosting stops being a preference and becomes the only compliant path available.

How should you actually benchmark models before deploying?

Public leaderboards tell you almost nothing about how a model will perform on your specific workload, and treating them as a procurement decision is one of the most common mistakes enterprises make.

Published benchmarks are optimised for general capability and often for the exact tasks the model’s own team trained toward. A model that tops a public reasoning leaderboard can still underperform badly on your company’s support tickets, your industry’s jargon, or your particular document formats. The gap between a benchmark score and production usefulness is where most disappointing rollouts come from.

Building a representative test harness on your own data is not optional if the decision carries real budget behind it. A workable approach:

  1. Pull a representative sample of real production tasks, not synthetic examples, ideally 100 to 300 cases spanning your typical difficulty range.
  2. Define metrics that map to business impact: accuracy against a human-graded answer, latency at the percentile that matters (usually p95, not average), and cost per successful resolution rather than raw per-token cost.
  3. Run the candidate models side by side under identical prompts and identical grading criteria, including at least one open-weight and one closed model in the comparison.
  4. Score safety and failure modes explicitly, not just correctness. A model that’s 3% more accurate but occasionally hallucinates a confident wrong answer in a regulated context is not the better choice.

A one or two point gap on a public benchmark rarely predicts anything about your production error rate.

The hybrid architecture pattern enterprises are converging on

The pattern showing up repeatedly across enterprise deployments is a router sitting between the application and two or more model backends, deciding per request which one handles it.

A typical split sends high-volume, low-complexity tasks, think classification, extraction, or routine summarisation, to a self-hosted open-weight model. Complex reasoning, nuanced customer-facing conversations, or anything with real reputational risk goes to a closed frontier model. Guidance on 2026 model decision-making describes this as the dominant enterprise pattern precisely because it captures the cost benefit of open weights on bulk traffic while keeping frontier capability available for the cases that genuinely need it.

The router itself, not the model choice, becomes the highest-leverage engineering investment once you commit to this pattern. That means building fallback logic (what happens when the self-hosted model times out or returns a low-confidence result), monitoring that tracks accuracy and latency per route, and a clear policy for when to switch a workload from one backend to another as models improve or degrade.

Hybrid LLM workload routing architecture

Cost control inside a hybrid setup comes down to three levers: caching repeated or near-duplicate queries so they never hit either model twice, batching self-hosted inference requests to maximise GPU throughput, and choosing the smallest model that clears your accuracy bar for each route rather than defaulting to the largest available. None of this is exotic engineering. It’s disciplined operations applied to a new kind of infrastructure.

A decision checklist before you commit to either path

Run through this before signing anything or provisioning a single GPU. Five inputs decide most of the answer:

  • Token volume estimate: your current and 12-month projected daily volume, since this single number drives the TCO conversation more than anything else.
  • Compliance posture: whether any regulator or contract mandates in-perimeter processing.
  • MLOps capacity: whether you have, or can hire, the engineering depth to run inference infrastructure reliably.
  • Latency tolerance: whether your use case can absorb a network round trip or needs sub-100ms local inference.
  • Budget horizon: whether you’re optimising for lowest cost this quarter or lowest cost over three years.

For a pilot, set pass/fail thresholds before you start testing, not after you see the results. A reasonable bar: the candidate model must match or beat your incumbent on accuracy against your own graded dataset, stay within an agreed latency ceiling at p95, and cost less per successful resolution than the alternative once infrastructure is included.

  1. Confirm the exact licence text for any open-weight model, not just the marketing summary.
  2. Negotiate data retention terms with any closed vendor in writing, not as a verbal assurance.
  3. Ask explicitly whether fine-tuned artefacts (weights or adapters) are portable if you switch providers later.
  4. Get the vendor’s model deprecation and upgrade policy in writing before you build critical infrastructure on top of it.

Red flags worth walking away from: an SLA with no defined uptime percentage, a licence clause that caps your usage below your realistic growth, or a vendor that won’t commit in writing to a data retention window.

Pro Tip: Ask every closed vendor one question directly: “If you deprecate this model version, how much notice do we get, and can we export fine-tuned artefacts?” The quality of that answer tells you more about long-term risk than any benchmark score.

How a UK consultancy applies this in practice

A consultancy builds hybrid AI deployments for businesses that need this decision made practically, not theoretically. That means selecting the right model for each workload, whether that’s a self-hosted open-weight model for a high-volume RAG retrieval layer or a closed frontier model for a customer-facing voice or chat agent handling nuanced enquiries.

Core services span RAG systems, voice agents, chat agents, and email agents, each built to handle customer interaction naturally rather than through rigid scripted flows. Model selection sits underneath all of it: a support ticket triage system might run entirely on a self-hosted open model for cost reasons, while a customer-facing conversational agent might route to a closed API where reliability and nuance matter more than marginal cost savings.

Generative Engine Optimisation (GEO) work adds another dimension specific to this moment: as AI-driven answer engines increasingly summarise and cite businesses directly, being recognised correctly in those responses depends partly on how a business’s content and structured data are prepared, and partly on how the underlying models interpret and rank that information.

The balance between open and closed isn’t philosophical for a consultancy delivering these systems. It’s a workload-by-workload calculation, repeated for every client and every agent built.

Why doesn’t integration ever go as smoothly as the demo?

Every model demo looks clean because it’s disconnected from anything real. The gap appears the moment you connect a model to your actual CRM, ticketing system, or legacy database.

Closed APIs are usually the easier integration on paper: a single REST endpoint, well-documented SDKs, and predictable response formats. The friction shows up around your existing systems, not the API itself. Feeding a closed model accurate, current context from your CRM or knowledge base means building a reliable retrieval layer, and that layer has to handle your data’s actual mess: inconsistent formatting, outdated records, duplicate entries across systems that were never designed to talk to each other.

Self-hosted open models add a second layer of integration work on top of that: the inference server itself needs to sit inside your network architecture, behind your existing authentication and logging, without becoming a bottleneck for other services sharing the same infrastructure.

In both cases, the model is rarely the hard part. The hard part is the plumbing around it: authentication handoffs between systems, data transformation between formats, and monitoring that spans both the model layer and whatever legacy system it’s now talking to. Teams that budget engineering time only for “connecting to the API” and not for this surrounding integration work are consistently the ones whose timelines slip.

Why the open-versus-closed debate misses the point

The framing of open versus closed as a single, one-time decision is largely a distraction. Most production systems that work well have already stopped treating it as binary, and the ones still arguing about it in the abstract are usually the ones without a working pilot yet.

What deserves real investment isn’t a philosophical stance on openness. It’s the benchmarking discipline to test both approaches against your own data, and the observability to catch when a routing decision made in the last quarter is quietly costing you more than it’s saving. The model landscape will keep shifting: open-weight quality will keep closing the gap on frontier closed models, and closed vendors will keep responding with new capability and pricing pressure. Build your architecture to absorb that churn rather than betting the whole system on today’s leaderboard.

— Oded

Get help building your hybrid LLM strategy

Real-studios is the practical alternative to guessing your way through model selection: instead of committing to one vendor and hoping it holds up at scale, you get a workload-by-workload assessment of where a self-hosted open model earns its keep and where a closed API is worth the premium. That’s the difference between a system built on assumptions and one built on your actual traffic patterns and compliance requirements.

Real-studios’s service integrations connect whichever model architecture fits into your existing CRM, ticketing, or data pipeline without the trial-and-error most in-house teams go through first. The automation services cover the ongoing MLOps side, monitoring, routing logic, and version management, so the hybrid setup keeps working after launch, not just on demo day. For teams still validating which model fits before committing budget, the MVP service builds a working pilot on your own data fast enough to inform a real procurement decision.

If you’re weighing open weights against a closed API for your next deployment, get in touch through Real-studios’s services page and start with a scoped assessment rather than a guess.

Sources

FAQ

Is ChatGPT open or closed source?

ChatGPT runs on OpenAI’s GPT models, which are closed and proprietary. You access them only through OpenAI’s API or interface, with no access to the weights or training data.

Are there any fully open source LLMs?

Some releases go beyond open-weight and publish training code and data alongside the weights, meeting a stricter open-source standard, though most widely used models like Llama and Mistral are open-weight with licence restrictions rather than fully open-source.

What are the key differences between closed-source and open-source AI?

Closed models give you an API and a managed service with no visibility into weights or training data, while open and open-weight models give you the parameters to inspect, fine-tune, and self-host, shifting operational responsibility onto your own team.

Which LLM is better, open or closed?

Neither is universally better. Closed models like GPT, Claude, and Gemini generally lead on frontier capability and speed to production, while open-weight models like Llama, Mistral, and DeepSeek win on cost at scale, data control, and long-term flexibility.

How do I decide between self-hosting and a closed API?

Estimate your daily token volume first, since self-hosting typically becomes cheaper above roughly 10 to 30 million tokens per day, then weigh that against your compliance requirements and whether your team has the MLOps capacity to run inference infrastructure reliably.

Want this working in your business?

We use cookies to make this site better

We use cookies to understand how you use the site and improve your experience. You can accept or reject — your choice. Privacy Policy.