Self-Hosting an LLM Won't Protect Your Company Data by Itself

What self-hosting an LLM actually protects

Self-hosting removes one party from your data path: the model vendor. Prompts, retrieved documents and generated outputs stay inside a perimeter you own. No third-party processor, no cross-border transfer, no contractual promise you have to take on faith.

That is a real win, and it is a narrow one. It fixes exactly the class of risk that lives outside your network. Every other way company data leaks out of an LLM system is untouched by where the GPU sits.

Here is the honest scorecard.

Six rows improve. Three are unchanged. Two get actively worse. If the reason you are self-hosting is a compliance line about third-party processing, the top block is your answer. If the reason is a vague sense that data is safer on your own metal, read on.

Self-hosted LLM vs enterprise API: what the data terms actually say in 2026

Most comparisons of self-hosting against APIs are still arguing against 2023 consumer chat terms. The API side has moved.

Anthropic deletes API inputs and outputs after seven days by default and does not train on them, with a 30-day option for customers who want the audit trail, and Zero Data Retention available to qualifying enterprise accounts. OpenAI publishes a comparable position, and on 19 August 2026 previewed Private Safety Processing, its approach to keeping Zero Data Retention on frontier models while still detecting abuse patterns across related interactions. Under that arrangement, customer content stays on infrastructure the customer controls.

The wrinkle worth knowing before you build a compliance argument on ZDR: the guarantee is model-specific, not vendor-specific. Anthropic has stated that for its covered frontier models, commercial ZDR customers as of 9 June 2026 are subject to limited retention and review as part of safety work. The lesson is not that one vendor is careless — it is that ZDR is a per-model, per-endpoint property that can change when a new model ships. If your DPIA names a guarantee rather than a model, it will go stale without anyone telling you.

That is the strongest argument for self-hosting that has nothing to do with hackers: a model you host does not renegotiate its terms with you.

Read the last row twice. It is the whole trade.

Does GDPR or the EU AI Act require you to self-host an LLM?

No. Neither regulation mandates local inference. What self-hosting does is delete two of the most expensive sections of your paperwork.

• Article 28 processor obligations. A hosted API makes the provider a processor, which needs a written contract, sub-processor control, audit rights and breach notification. Run the model on hardware you operate and there is no processor in the relationship at all.
• Chapter V transfer analysis. Routing EU personal data to a US-hosted endpoint is a cross-border transfer needing SCCs or the Data Privacy Framework, with the residual Schrems II risk that a DPO has to write down every year. Self-hosting inside your own region removes the transfer rather than justifying it.

What it does not remove: your record of processing activities, your DPIA, your lawful basis, and — since 2 August 2026 — the EU AI Act's Article 50 transparency duties, which require disclosing that a user is interacting with an AI system and marking generated content in machine-readable form. Obligations for high-risk systems were pushed to December 2027 by the Digital Omnibus package, which buys documentation time but changes nothing about who is accountable.

The practical version: self-hosting is a way to make a compliance argument shorter, not a way to skip having one. Teams that treat the GPU purchase as the compliance deliverable are the ones that fail the audit on logging and access control instead.

The risk you inherit the moment you self-host

Here is the part the privacy pitch leaves out. The most popular way to run a model locally ships with no authentication, and a large share of the installed base is sitting on the public internet.

SentinelLABS and Censys spent roughly 300 days mapping the internet and found about 175,000 unique exposed Ollama hosts across 130 countries, with nearly half configured for tool calling — that is, able to execute code and reach other systems. By May 2026, in the context of a critical vulnerability, Cyera put the count of exposed servers at roughly 300,000. Take the exact figure with appropriate caution; the order of magnitude is not in dispute.

Those instances are not exposed because the operators are careless people. They are exposed because the default is wrong for the way the tool gets used: no authentication out of the box, and a very common reconfiguration to listen on all interfaces the moment someone needs to reach the model from another machine.

Then a bug lands on top of that. CVE-2026-7482, nicknamed Bleeding Llama, is an out-of-bounds heap read rated CVSS 9.1 that lets an unauthenticated attacker pull process memory using three API calls. The researchers at Cyera who disclosed it listed what comes back: prompts, system instructions, environment variables and API keys. Every one of those is the exact data class you self-hosted to protect.

The gateway layer has the same problem. A March 2026 supply chain compromise in a widely used AI gateway library (CVE-2026-33634, CVSS 9.4) demonstrated that one poisoned dependency can expose an organisation's entire portfolio of AI provider credentials at once. On the API side of the fence, a stolen key costs you one vendor's blast radius. In a self-hosted stack, the gateway is usually holding all of them.

The uncomfortable summary: self-hosting an LLM to protect company data works only if your team runs inference infrastructure to the standard it runs databases. Most teams that make this decision for privacy reasons have never had to.

What it costs to self-host an LLM in 2026

Capability is no longer the blocker. Open-weight models now cover the range from a laptop to a rack, and the sizing rule is straightforward: weights must fit in fast memory, at roughly 2 bytes per parameter in FP16, half that at INT8, a quarter at 4-bit — plus headroom for the KV cache, which grows with context length and concurrency.

The bill is where intuition fails. Rented H100 capacity ran between roughly $1.49 and $6.98 per GPU-hour depending on provider across published rate surveys in August 2026, and the sticker rate is the least interesting number in that range: at 40% utilisation, a $3.99 card costs nearly $10 per hour of useful work. Idle GPUs are the single largest source of self-hosting regret.

Three cost lines that consistently get left out of the business case:

• Engineering. Budget half to one full-time engineer for the first year. For most teams this exceeds the GPU line, and it does not disappear after launch — models, runtimes and CVEs all keep moving.
• Utilisation. Per-token APIs charge zero when nobody is asking. A reserved GPU charges the same at 3am.
• Redundancy. One GPU is a demo. Production means a second node, or an honest plan for what happens when the first one dies.

For cheap, high-volume open models served by commodity inference providers, there is often no crossover point at all — the per-token rate is below what a single rented card can produce. Self-hosting wins on economics for flagship-tier usage, custom fine-tuned checkpoints, and steady high-volume traffic. It wins on control at any volume, which is a different argument and should be made on its own terms.

The leak your architecture creates no matter where the model runs

In practice, company data does not escape through the model vendor. It escapes through three things you build yourself.

1. The retrieval index that dropped its permissions

The most common serious incident in an internal AI rollout is not exfiltration. It is an employee asking a chatbot a question and receiving content from a document they were never allowed to open. That happens when a RAG pipeline ingests a shared drive without carrying per-document ACLs into the vector store and re-checking them at query time.

Self-hosting makes this failure mode more likely, not less. The sentence "we can't put that in the cloud" was quietly acting as a data classification gate. Remove it and the ingestion scope expands to everything, usually in the same sprint.

2. Your own prompt logs

Self-hosted stacks log generously by default. Gateways, observability tooling and chat frontends all capture full prompts and completions, and those land in a database that frequently has weaker encryption, retention and access controls than the vendor policy you replaced. You did not eliminate retention. You moved it somewhere with no DPA and no deletion schedule.

3. Agent tool scope

The moment the model can call tools, the security question stops being about text and starts being about capability. Nearly half of those exposed hosts had tool calling enabled. Scope every tool to the identity of the requesting user, not the identity of the service — an agent that queries your database with a service account is a privilege escalation waiting for a well-phrased prompt.

These are architecture decisions, not infrastructure decisions, and they are the same whether inference runs in your rack or someone else's. If you want them designed properly the first time, that is the work behind our AI automation and workflow engineering service — agents, retrieval pipelines and the permission model underneath them.

How to secure a self-hosted LLM deployment

If you self-host, treat the inference server as production infrastructure holding sensitive data — because that is exactly what it is. In rough order of how often each one is missed:

1. Never bind the inference server to 0.0.0.0. Bind to localhost or a private interface, and put a reverse proxy in front. Verify from outside, not from the box.
2. Require authentication on every endpoint. API keys or OAuth tied to role-based access control, not a shared token in a wiki page.
3. Scan yourself the way attackers do. Confirm your inference ports are not reachable from the public internet, then re-check after every infrastructure change.
4. Use a production runtime for production. vLLM, SGLang or TensorRT-LLM for served traffic; keep the developer-friendly local tools on developer machines.
5. Patch the serving stack on a security cadence. Inference runtimes are now a CVE surface with active exploitation, not a hobby dependency.
6. Validate model artifacts. Pull weights from known publishers with checksums. A malformed model file has been a working attack vector.
7. Isolate the gateway's secrets. Provider keys in a secret manager with per-service scoping, so one compromised dependency does not surrender the whole keychain.
8. Carry ACLs through retrieval. Filter at query time by the requesting user's actual permissions, and re-index when permissions change.
9. Set a retention policy on prompt logs and encrypt them. Then delete on schedule, and prove it.
10. Log inference requests for audit — user, timestamp, model, tools invoked. You need this for incident response, and increasingly for regulators.
11. Rate-limit and monitor for abuse. Unusual token volume from one identity is the earliest signal of both misuse and compromise.

When to self-host, when to use an API, and how to run both

The strongest pattern is not a choice at all. Classify the data, then route the traffic — most organisations have three tiers and only one of them justifies dedicated hardware.

Self-host when at least one of these is true: a regulator or a customer contract names data locality; your DPIA cannot absorb a third-party processor; you need a fine-tuned checkpoint nobody hosts; or your steady-state volume is high enough that reserved GPUs beat per-token pricing at realistic utilisation.

Stay on an API when your volume is bursty, when your team has no on-call rotation for infrastructure, or when the honest answer to "who patches the inference server at 2am" is nobody. A hosted model with enterprise terms and a tight architecture protects company data better than a self-hosted model with an open port. That is not a close call.

If you are weighing this decision and want the architecture reviewed before you buy hardware — data classification, routing, retrieval permissions and the deployment itself — tell us what you are building and we will scope it.