For most regulated teams, the honest answer in 2026 is neither pure on-premise nor naive cloud: it is a cloud LLM on a route you can contract for, behind a per-request PHI gate, with residency pinned to a region you chose. Go fully on-premise only when a contract, a regulator, or a threat model actually forbids the alternative — because the operational cost is real and permanent.
The tell that this stopped being an engineering question: as of October 2026, TrueFoundry is handing out complimentary copies of the Gartner Hype Cycle for AI Governance 2026 — governance, not GPU count, is what buyers are being sold. That matches what I see in practice. The deployment question arrives disguised as a performance question and turns out to be a contractual one.
Key takeaways
- On-premise buys you control over the payload and the network path. It does not buy you a BAA, and it does not make you compliant by itself.
- Cloud LLMs can carry PHI, but only on routes where a BAA is actually offered — and only if your prompt routing enforces that per request, not per environment.
- Residency is a routing decision, not a marketing claim. Pin the region, log which region served each call, and be able to prove it.
- The CLOUD Act question does not disappear when you self-host if a US-headquartered vendor touches the stack.
- Cost is dominated by people, not GPUs. A single-node on-prem deployment is cheap to run and expensive to own.
Why does 'where your LLM runs' matter for PHI and regulated data?
Because the location determines who is a business associate, who can be compelled to disclose, and what you can prove in an audit. A prompt containing PHI is a disclosure the moment it leaves your boundary. Everything else — encryption, logging, retention — is a control on top of that first fact.
This is the part people skip. They ask "is the model good enough" and "is it fast enough," then discover months later that the vendor's terms permit training on submitted data, or that logs retain payloads for a retention window nobody agreed to. Ask three questions before anything else: does the payload get stored, does it get trained on, and which legal entity receives it.
On-premise LLMs: what does full control actually buy you?
Self-hosted LLM compliance is about custody, not certification. On-premise means the prompt never crosses a third-party boundary, so there is no business associate to contract with and no vendor retention policy to audit. You own the disk, the network, and the deletion. That is a genuinely strong position — and it is the only architecture where "no third party ever sees this" is literally true.
The trade-offs are equally real. Model weights are frozen at the version you pulled; you own every CVE, every driver mismatch, and every capacity spike. If your peak is three times your average, you buy for the peak and idle the rest. And you still need the surrounding controls: access logs, key management, and a documented answer to "who can read the inference server's memory."
Two things on-premise does not give you for free. It does not give you a HIPAA BAA, because there is no counterparty to sign one — you are the covered entity and the custodian. And it does not automatically satisfy GDPR or the Swiss nFADP, which care about lawful basis and data subject rights regardless of where the box sits. Treat it as posture, and take the specifics to your counsel.
Cloud LLMs: how does shared responsibility and a vendor BAA actually work in 2026?
A cloud LLM can process PHI lawfully when the provider offers a BAA covering that specific service, and when your architecture keeps PHI on covered routes only. The mechanism matters more than the promise: BAAs attach to named services and regions, not to a vendor's whole catalogue. A model that is covered in one region may not be covered in another.
This is where most implementations quietly fail. Teams get a BAA, then route traffic through a fallback model or a logging proxy that is not covered. The fix is a per-request gate rather than an environment-level assumption — I wrote up the mechanics in implementing per-request PHI gates for LLM APIs, and the short version is that every call carries its data class and residency, and the gateway refuses to forward PHI to a route that cannot accept it.
If you are weighing a specific provider's healthcare posture, our breakdown of OpenAI's HIPAA-eligible offerings in 2026 covers what is and is not covered. And if you are comparing gateways to enforce this, the 2026 LLM router and AI gateway comparison is the one I would start with.
The genuine advantage of cloud: you get frontier models on the day they ship, elastic capacity, and a bill that scales down to zero. That is not a small thing. It is usually the reason the on-premise plan loses.
Data residency and the CLOUD Act: what changes between the two architectures?
Residency is about where processing happens; the CLOUD Act is about whose law can reach the provider. They are separate problems, and self-hosting only solves the first one cleanly. If your on-premise stack includes a US-headquartered management plane, telemetry agent, or support contract with remote access, you have not escaped the second question — you have narrowed it.
For data sovereignty work, name the region explicitly and log it per call. "EU processing" is not an answer; "served from a Frankfurt region, recorded in the audit trail for this request" is. For GDPR and nFADP, the residency argument supports your transfer analysis but does not replace it. Ask counsel to confirm the transfer mechanism, not the marketing page.
What does on-prem LLM vs cloud LLM actually cost, and how do they scale?
The cost of on-prem LLM vs cloud LLM is usually misread as a hardware comparison. It is not. Hardware is the cheap, predictable part; the expensive part is the engineer who owns patching, capacity, and uptime, forever. Cloud converts that into a variable bill you can forecast and cancel.
Scaling differs in shape, not just size. Cloud absorbs a tenfold spike in an afternoon. On-premise absorbs it after a procurement cycle. If your workload is spiky or experimental, that asymmetry dominates every other line item.
Hybrid LLM deployment strategy: when is it the right answer?
Hybrid is the default for regulated workloads, not a compromise. Keep PHI on the controlled path and send everything else to the cheapest capable model. Classify at the request, route on the classification, and log the decision.
A hybrid LLM deployment strategy works when the classification is enforced by infrastructure rather than by developer discipline. If a developer can accidentally send PHI to the open route by forgetting a flag, you have a policy, not a control. Default-deny is the only version of this that survives contact with a real sprint.
Technical deep dive: what do secure data flows look like on each side?
Both architectures need the same three artifacts: a data class on the request, a residency pin, and an audit record on the response. The difference is who enforces them. Illustratively, the shape is:
curl \
-H "Authorization: Bearer $TOKEN" \
-H "X-Data-Class: phi" \
-H "X-Residency: eu-central" \
-d '{"model":"...","messages":[...]}'
The gateway either finds a route that accepts phi in eu-central, or it returns an error. It does not silently downgrade. On the response side you want an audit identifier and the region that actually served the call, so the log answers "where did this go" without storing the prompt itself.
That last point is the one I would defend hardest: log the routing decision, not the payload. An audit trail that contains PHI is a second copy of the problem. Our guide to HIPAA-compliant LLM APIs goes deeper on what belongs in that record.
Comparison: on-premise vs cloud vs hybrid for regulated data
| Dimension | On-premise | Cloud (BAA-covered route) | Hybrid |
|---|---|---|---|
| Where PHI is processed | Your hardware only | Provider region you select | Split by request classification |
| BAA counterparty | None — you are the custodian | Provider, for named services and regions only | Both, scoped per route |
| Residency control | Absolute | Contractual and regional | Per-request pin |
| Payload logging | Whatever you build | Provider policy plus your gateway | Gateway-controlled |
| Model freshness | Frozen at pull | Same-day | Mixed |
| Scaling a spike | Procurement cycle | Minutes | Minutes on the open route |
| Cost shape | Fixed, people-heavy | Variable | Variable with a fixed floor |
| Ops burden | Fully yours | Provider's, minus your routing layer | Shared |
| CLOUD Act exposure | Reduced, not eliminated | Present via the provider | Depends on the route |
| GDPR / nFADP | Posture only, not automatic | Needs transfer mechanism | Per-route analysis |
Decision framework: which should you pick?
Pick on-premise when a contract, regulator, or threat model forbids third-party processing, and you can fund an owner for it permanently. Pick cloud when you need frontier capability and can contract for the route. Pick hybrid when most of your traffic is not PHI — which, in my experience, it almost never is.
If you are a small team with a handful of PHI-adjacent use cases, you probably do not need on-premise yet. Build the routing gate first. The gate is the durable asset; the deployment target is a configuration behind it. That is the posture CloudSitus is built around: stateless handling, no payload logging, no training on your data, per-request PHI routing, region residency, and an audit trail on every call. BAA coverage applies to the Azure routes — not to every route, and not as a blanket certification. Worth checking against your own requirements before you commit either way.
FAQ
Is an on-premise LLM automatically HIPAA compliant?
No. HIPAA compliance is about administrative, physical, and technical safeguards plus your agreements and policies. Self-hosting removes a third party from the picture, which simplifies the BAA question, but it does not substitute for the safeguards themselves.
Can I use a cloud LLM with PHI if I have a BAA?
Only on the services and regions the BAA actually covers, and only if your routing enforces that per request. A BAA that covers one service does not cover a fallback model, a logging proxy, or a different region.
Does GDPR require an on-premise LLM?
No. GDPR governs lawful basis, transfers, and data subject rights, not the physical location of the server. On-premise can strengthen your transfer position, but it does not remove the need for a legal basis or a transfer mechanism. Ask your counsel.
What is the cheapest option for a small regulated team?
Usually a cloud route with a BAA and a per-request gate. On-premise has a low hardware floor and a high people floor, and the people cost does not scale down when usage does.
Does the CLOUD Act apply to my on-premise deployment?
It can, if a US-headquartered vendor has remote access, telemetry, or a support contract touching the stack. Self-hosting narrows the exposure rather than eliminating it, so map every third party that can reach the system.