Implementing Per-Request PHI Gates for LLM APIs: A How-To Guide

The PHI boundary belongs at the API edge, not buried inside your application — that is the entire argument in the medical-necessity issue titled "[phi] Keep the PHI boundary at the API edge", and it is the right one. A per-request PHI gate is a check that runs on every single LLM call, before the prompt leaves your trust boundary, and decides allow, redact, or block based on what is actually in that payload. It is not a firewall rule and not a one-time data-flow diagram. It is code on the hot path, and it has to be boring, fast, and logged.

Key takeaways

• Gate the request, not the tenant. A tenant that is "allowed to use PHI" still should not send it to a route with no BAA.
• Detect with layered signals: deterministic validators first, NER second, and never trust a single regex.
• Log the decision and the audit ID, never the payload. If your gate logs prompts, you have built a second PHI store.
• BAA coverage is route-specific. On Azure OpenAI routes it is available; on non-Azure routes it generally is not. Ask your counsel what your posture actually covers.
• Test with a corpus of real-shaped fake PHI, and re-run it every time you touch the gate.

1. What Is a Per-Request PHI Gate and Why Does It Matter for Healthcare LLMs?

A per-request PHI gate is a synchronous policy check that inspects each outbound LLM request, classifies any protected health information in it, and then routes, masks, or rejects the call before it reaches a model. It matters because LLM privacy controls in healthcare fail at the payload level, not the network level — the leak is a name in a symptom description, not an open port.

The distinction that trips teams up: a gate is not a redaction pipeline that runs once at ingest. Prompts are assembled at runtime from retrieved documents, user free text, and tool output. PHI appears in the last hundred milliseconds before the call. That is where the check has to live.

Two design rules I would hold to. First, the gate is fail-closed: if detection errors out, the request does not go. Second, the gate is stateless — it holds no prompt content after the response returns, or you have just moved the problem.

2. Where Should You Intercept and Filter? Architectural Patterns

Intercept at the last hop you control before the provider, because that is the only place where the final assembled prompt is visible. In practice that means a gateway or a sidecar, not scattered SDK calls, unless you have exactly one caller and expect it to stay that way.

The gateway pattern is the one I would pick for anything touching PHI, because it gives you one place to enforce routing, one place to emit audit events, and one place to patch. The tradeoff is real: you now own an availability dependency. The on-premises LLM gateway write-up on DEV is a decent starting point for the SRE side of that tradeoff — capacity, failover, and what happens when the gateway itself is the incident.

3. How Do You Detect and Redact PHI in Real Time?

Layer deterministic detectors first, then a statistical model, and treat every layer as producing findings rather than a verdict. Deterministic checks are cheap, explainable, and auditable — which is exactly what a compliance lead will ask you to justify six months from now.

For PHI masking for LLMs, replace rather than delete. [PATIENT_NAME_1] keeps the prompt coherent and gives you a stable token to map back if the workflow genuinely needs it — and if it does not, do not keep the map. Deletion changes sentence structure and degrades model output, which is how you end up with clinicians pasting raw text into a consumer chatbot because "the internal one is useless."

On latency: run deterministic checks inline and always. Run NER inline only if you have measured it. If your NER pass costs more than your token generation, move it to a pre-processing step and cache the result per document.

4. How Do You Wire the Gate Into OpenAI and Azure OpenAI Calls?

Put the gate in front of the client, not inside it. The gate returns a decision object; the caller sends the scrubbed prompt and stamps the decision onto the request as headers so the audit trail is reconstructable from the provider logs you already keep.

# gate.py — runs on every request, before any provider call
from dataclasses import dataclass

@dataclass
class GateDecision:
 allow: bool
 route: str # e.g. "azure-openai-eastus2" or "blocked"
 findings: list[str]
 audit_id: str

def gate(prompt: str, tenant: dict) -> GateDecision:
 findings = detect_phi(prompt) # Step 3: regex + validators + NER
 if findings and not tenant["phi_route_enabled"]:
 return GateDecision(False, "blocked", findings, new_audit_id())
 return GateDecision(True, tenant["route"], findings, new_audit_id())

decision = gate(prompt, tenant)
if not decision.allow:
 raise PHIBlocked(decision.audit_id) # fail closed

scrubbed = mask(prompt, decision.findings)

resp = client.chat.completions.create(
 model=decision.route,
 messages=[{"role": "user", "content": scrubbed}],
 extra_headers={
 "X-PHI-Gate-Decision": "allow",
 "X-PHI-Redactions": str(len(decision.findings)),
 "X-Audit-Id": decision.audit_id,
 },
)

The header name for extra headers differs between SDKs and versions — check yours rather than copying this blindly. For secure PHI routing LLM work on Azure, the shape is the same but the auth and URL differ:

curl -sS "" \
 -H "api-key: $AZURE_OPENAI_KEY" \
 -H "Content-Type: application/json" \
 -H "X-PHI-Gate-Decision: allow" \
 -H "X-Audit-Id: $AUDIT_ID" \
 -d '{"messages":[{"role":"user","content":"<redacted prompt>"}],"max_tokens":256}'

Pin the api-version you have actually tested; Azure requires it and the available versions move. One honest caveat: upstream providers do not guarantee they echo your custom headers back. If you need the audit ID on the response side, your gateway must attach it, not the provider. That is a good reason to terminate the call at your own edge and stream the response back yourself.

For the wider provider picture, our 2026 guide to HIPAA-compliant LLM APIs and secure PHI handling walks through which routes carry what.

5. What Should You Log, and How Do You Build an Audit Trail?

Log the decision, the detector that fired, the count of redactions, the route, the tenant, and the audit ID. Never log the prompt, the completion, or the matched substring — a redaction log that contains the redacted value is a PHI store with extra steps.

Structure each event so it answers the three questions an auditor will ask: what was sent, where did it go, and who decided. A minimal event looks like this:

• audit_id — unique per request, echoed on the response
• tenant_id, route, region
• decision — allow / redact / block
• detectors_fired — names only, no values
• redaction_count
• gate_version — so you can prove which logic was live on that date

Retention is a legal question, not an engineering one. Set it with your counsel, not with a default in your logging library. And keep the gate's own access logs separate from the application's request logs, because the gate log is the one you will have to produce.

6. How Do You Test That Your PHI Gate Is Airtight?

Test with a fixed corpus of synthetic PHI that looks like the real thing, and run it as a blocking CI check. The failure mode you are hunting is not "detector missed a name" — it is "a new code path stopped calling the gate at all."

Checklist:

• ☐ Every LLM call site goes through the gate. Grep for direct client construction in CI and fail the build.
• ☐ Synthetic corpus covers names, MRNs, dates, addresses, phone numbers, and PHI embedded in prose, not just in fields.
• ☐ Adversarial cases: PHI split across two messages, base64-encoded, inside a code block, in a retrieved document.
• ☐ Fail-closed test: kill the detector and confirm the request is rejected, not passed.
• ☐ Header propagation test: confirm the audit ID arrives at the provider and comes back.
• ☐ Log inspection test: assert that no test payload string appears anywhere in your logs.

Run it on every gate change and on a schedule, because detection quality drifts as your prompt shapes change.

7. When Should You Build vs. Buy?

Build if you have one or two call sites, a small number of PHI categories, and engineers who will own the detector corpus for years. Buy or use a gateway if you need region residency, per-request routing, and an audit trail you can hand to an auditor without writing it yourself.

A self-hosted proxy like GATE — Gateway for AI Traffic & Enforcement, which does virtual keys, quotas, routing and guardrails, is a reasonable middle path if you want the gateway mechanics without building them. You still own the PHI detection quality and the audit story.

Our own posture, stated plainly: stateless requests, no payload logging, no training on your data, a per-request PHI gate, region residency, and an audit trail on every call. BAA coverage applies to our Azure routes — not to every route, and we will not call anything "HIPAA-certified." Whether that posture satisfies your obligations is a question for your counsel.

If you want to see how the routing and residency pieces fit together before you commit to a design, start with our 2026 comparison of LLM routers and AI gateways for regulated industries, and if your workload is mostly OpenAI, the guide to OpenAI's HIPAA-eligible offerings covers which routes actually carry a BAA.

FAQ

What is a per-request PHI gate in an LLM API?

It is a synchronous check that inspects each outbound LLM request for protected health information and decides allow, redact, or block before the prompt leaves your trust boundary. It runs on every call, not once at ingest, because prompts are assembled at runtime.

Does a HIPAA BAA cover every LLM route?

No. BAA coverage is route- and region-specific. In practice it is available on Azure OpenAI routes and generally not on consumer or third-party routes, so your gate has to enforce routing, not just redaction. Confirm your specific posture with your counsel.

Is regex enough for real-time PHI detection?

Regex reliably catches structured identifiers like MRNs, SSNs, and phone numbers, but it misses names and PHI embedded in prose. Layer a NER model on top, keep deterministic checks inline, and treat the model pass as a second signal rather than the verdict.

Should the gate log prompts so you can investigate incidents?

No. Log the decision, the detector names, the redaction count, the route, and the audit ID. A log containing the redacted value is a second PHI store with weaker controls than the first one.

How often should you re-test a PHI gate?

On every change to the gate, and on a schedule. Detection quality drifts as prompt shapes and retrieved document formats change, and the most common real failure is a new call site that bypasses the gate entirely.