The moment your prompt lands in GPU memory, it is plaintext. TLS protects it in transit, disk encryption protects it at rest, and neither touches the milliseconds it spends in RAM. Confidential computing for LLMs in regulated industries closes that data-in-use gap with hardware-enforced trusted execution environments (TEEs) — and it is the only hardware-level answer that does.
Key takeaways
- TEEs encrypt memory in use: CPU (Intel TDX, AMD SEV-SNP) and GPU (NVIDIA confidential computing, AMD MI300) — and for LLMs, the GPU TEE is the one that actually protects the model.
- Confidential computing shields prompts, model weights, and outputs from the infrastructure layer — not from your own application bugs, not from prompt injection.
- Performance overhead is real but workload-dependent; CPU TEEs run single-digit percentages, GPU confidential computing costs more. Benchmark your own path.
- Attestation is non-negotiable: verify the TEE on every call, and log the result to an audit trail, or you cannot prove anything to a regulator.
- Confidential computing is a control, not a certification. HIPAA BAA coverage exists only on specific Azure routes; GDPR/nFADP demands region residency. Ask your counsel.
What Problem Does Confidential Computing Solve for LLMs?
The encryption gap. Data at rest is encrypted on disk; data in transit is encrypted by TLS. Data in use — the moment a prompt is loaded into memory for inference — is plaintext. Confidential computing encrypts that memory and isolates the workload inside a hardware TEE that the host OS, hypervisor, and even the cloud provider's administrators cannot read.
For regulated industries, this is the difference between a contractual promise and a technical boundary. A HIPAA BAA obligates the provider; a TEE constrains the hardware. Red Hat's work on private data inference with proprietary LLMs frames it precisely: confidential computing is the path to running sensitive data through models without either side trusting the other's infrastructure. That is the whole game for healthcare and finance.
How Confidential Computing Works for LLM Inference (CPU and GPU TEEs)
A TEE is a hardware-enforced region of memory that the host OS, hypervisor, and cloud admin cannot read. CPU TEEs — Intel TDX, AMD SEV-SNP — encrypt host memory and provide remote attestation. But LLM inference runs on GPUs, so GPU confidential computing is where the real protection lives. NVIDIA's confidential computing mode on H100/H200 encrypts GPU memory and provides GPU attestation; AMD's MI300 series supports SEV-SNP.
| Attribute | CPU TEE (Intel TDX / AMD SEV-SNP) | GPU TEE (NVIDIA CC / AMD MI300) |
|---|---|---|
| What it protects | Host memory, CPU execution | GPU memory, weights, activations |
| Attestation | Mature, well-documented | Available, still evolving |
| Typical overhead | Low, single-digit % | Higher, workload-dependent |
| Right for | Routing, pre/post-processing, small models | LLM inference itself |
If you are running a 70B model, the CPU TEE is table stakes and the GPU TEE is the one that matters. A trusted execution environment for LLM workloads that stops at the CPU leaves the model and your PHI exposed where the actual computation happens.
What Confidential Computing Protects: Data in Use, Model Weights, and Prompts
Three things: your prompts and their intermediate activations, the model weights, and the outputs. All stay encrypted in memory. For regulated workloads, the prompt is the PHI — the patient note, the claim detail, the financial record. For model providers, the weights are the intellectual property.
The distinction that matters: confidential computing protects against the infrastructure layer — a compromised hypervisor, a rogue cloud admin, a host OS exploit. It does not protect against the application layer. If your own service logs prompts to a database, the TEE is irrelevant. That is why LLM data encryption in use is a necessary layer, not a complete strategy.
What Confidential Computing Does NOT Protect: Beyond Hardware Isolation
Prompt injection, application-layer leaks, model provider code bugs, key management failures, and side channels. A TEE cannot stop a malicious prompt from steering the model, and it cannot stop your own code from exfiltrating data. The zero-trust framing from Petronella's enterprise private AI analysis is the right mental model: trust nothing, verify everything — and remember the TEE is only one trust boundary.
- Prompt injection — the model can still be manipulated by malicious input; TEEs do not filter content.
- Application-layer leaks — if your app logs, caches, or forwards prompts, the TEE never sees the leak.
- Model provider bugs — a vulnerability in the inference stack inside the TEE is still exploitable.
- Key management — if attestation keys or memory encryption keys are compromised, the boundary falls.
- Side channels — mitigated by design, not eliminated; researchers keep finding new ones.
- The human layer — admins with control-plane access can still misconfigure attestation policies.
Performance Implications: Benchmarks and Real-World Overhead
Honest answer: overhead varies, and anyone who gives you a single number is selling something. CPU TEEs add single-digit percentage overhead in most workloads I have seen. GPU confidential computing is heavier — memory encryption on the GPU competes with the memory bandwidth the model needs. The practical guide to confidential computing for privacy-preserving LLM inference walks through the trade-offs; the short version is that you should expect measurable but tolerable overhead for most regulated workloads, and you should measure it before you commit.
Benchmark your own inference path. Batch size, model size, KV cache behavior, and whether you offload to CPU all change the picture. A rule of thumb I have settled on: if the overhead breaks your latency budget, the fix is usually batching or a smaller model — not abandoning confidential computing.
Deployment Considerations for Regulated Workloads: Integrating with Existing Security Postures
Confidential computing is one layer, not the whole stack. For healthcare, pair it with a per-request PHI gate that inspects every prompt before it reaches the model — we wrote a how-to guide on implementing per-request PHI gates for LLM APIs that covers the mechanics. For Europe, region residency and GDPR/nFADP posture matter as much as the TEE; see our data residency deep dive for LLMs in Europe.
Deployment checklist:
- Confirm the route's legal posture: HIPAA BAA coverage exists on Azure confidential computing routes; GDPR/nFADP requires region residency. Ask your counsel — do not assume.
- Require attestation on every call, not just at setup.
- Log attestation results and inference metadata to an audit trail.
- Gate PHI before the model, not after.
- Decide who holds the keys — you or the provider — and document it.
Confidential Computing vs. Other LLM Security Approaches
| Approach | Protects against | Gaps | Best for |
|---|---|---|---|
| Private VPC / network isolation | Network-level access | Cloud admins, host OS, memory | Baseline hygiene |
| On-premise | Physical access | Your own ops team, cost, scale | Small, static workloads |
| Confidential computing cloud | Infrastructure layer, memory in use | App layer, prompt injection | Regulated data at scale |
On-premise is not automatically more secure — it just moves the trust boundary to your own team. Our on-premise vs. cloud comparison for regulated data digs into that trade-off. Confidential computing in a cloud gives you the hardware boundary without the operational burden, which is why it is the pragmatic choice for most healthcare and financial workloads.
Verifying Confidentiality: Attestation and Audit Trails
Attestation is how you know the TEE is real. The hardware produces a signed quote — a cryptographic proof of what code is running, in what enclave, on what hardware. You verify that quote before sending a single prompt. Audit trails record every call: which route, which region, which attestation result.
For regulated workloads, this is the difference between "we use confidential computing" and "we can prove it." The HIPAA compliant LLM APIs guide covers how attestation and audit trails fit into a broader compliance posture. If you cannot produce an attestation record for a given inference, a regulator will treat it as if the TEE was never there.
If you are evaluating confidential computing for PHI-bearing LLM workloads, start with a proof of concept that measures overhead on your actual model and verifies attestation end to end. That is the only way to know whether the trade-off is worth it for your use case — and for most regulated workloads, it is.
FAQ
What is confidential computing for LLMs?
Confidential computing runs LLM inference inside a hardware-enforced TEE that encrypts memory in use, protecting prompts, model weights, and outputs from the infrastructure layer — cloud admins, hypervisors, and host OS exploits included.
Does confidential computing protect against prompt injection?
No. A TEE protects the hardware boundary, not the model's behavior. Prompt injection is an application-layer problem and needs separate controls like input filtering and per-request gating.
What is the performance overhead of confidential AI inference?
CPU TEEs add single-digit percentage overhead; GPU confidential computing is heavier and workload-dependent. Benchmark your own inference path — batch size, model size, and KV cache behavior all change the result.
Is confidential computing HIPAA compliant?
Confidential computing is a technical control, not a certification. HIPAA BAA coverage exists on Azure confidential computing routes; GDPR/nFADP requires region residency. Ask your counsel about your specific posture.
GPU confidential computing vs CPU TEE — which do I need?
For LLM inference, the GPU TEE is the one that protects the model and activations. CPU TEEs matter for routing, pre-processing, and smaller models — but if your PHI touches a GPU, the GPU TEE is non-negotiable.