Confidential Computing Summit 2026 -- San Francisco, CA
Authors: Ivan Font (@font) & Donald Hunter (@donaldh), Red Hat
When an AI model runs in the cloud, the model's weights and the user's prompts sit in plain memory. Anyone with access to the host machine -- a cloud administrator, a compromised hypervisor -- could read that data. This is the "memory-in-use" problem.
It creates a stalemate:
- Model providers need to keep their model weights secret (it's their intellectual property).
- Customers need to keep their prompts and data private.
Standard encryption protects data when it is stored on disk or sent over a network, but not while it is actively being used in memory.
This poster presents a solution that protects both sides at the same time using hardware-level isolation. The model stays encrypted until it runs inside a verified, locked-down environment. The customer's data never leaves that same protected space. Neither party has to trust the other -- or the cloud provider.
The solution uses Confidential Containers (CoCo) and Trusted Execution Environments (TEEs) -- special hardware features in modern CPUs and GPUs that encrypt memory and prevent anyone (including the machine's own administrators) from reading it.
Here's what happens step by step:
- Deploy: KServe schedules the AI workload inside a confidential virtual machine with hardware-encrypted memory.
- Measure: A guest agent inside the VM generates a cryptographic proof (a "Quote") that the hardware and software are genuine and untampered.
- Verify: A remote attestation service (Trustee) checks that proof against the model owner's security policy.
- Decrypt: Only after verification passes are the model's decryption keys released -- directly into the protected memory. The keys never exist in the clear outside of it.
The model weights are decrypted in protected CPU memory, then transferred through a secure channel to the GPU's own protected memory for inference. When the workload stops, the hardware wipes both CPU and GPU memory automatically. Nothing is left behind.
| Component | Options |
|---|---|
| CPU | Intel TDX, AMD SEV-SNP |
| GPU | NVIDIA H100, H200, B200 series |
KServe is the standard way to serve AI models on Kubernetes. By integrating confidential computing directly into KServe, developers can deploy protected AI workloads using the same tools and workflows they already know -- no need to become experts in attestation or key management.
The key upstream contribution that makes this possible:
- KServe PR #5382 -- This merged change solves a blocking issue that occurred when starting large AI models inside confidential VMs. It allows KServe to coordinate the initialization of models in hardware-isolated environments without breaking attestation guarantees or scaling.
The design avoids tying users to any single cloud provider or hardware vendor:
| What | Standard used |
|---|---|
| Encrypted container images | OCI ocicrypt |
| Encrypted model files in object storage | JSON Web Encryption (JWE / RFC 7516) |
| Attestation evidence | IETF RATS / Entity Attestation Token |
| Key release protocol | Generic KBS Protocol (works with Trustee, Intel Trust Authority, and others) |
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
spec:
predictor:
runtimeClassName: kata-cc
model:
modelFormat: { name: vllm }
storageUri: s3://my-bucket/encrypted-model/
confidential:
enabled: true
resourceId: "kbs:///default/modelkeys/llama3-70b-prod"A developer writes the same kind of manifest they would for any other model -- the only additions are runtimeClassName: kata-cc and the confidential block. The platform handles the rest.
When organizations deploy valuable AI models to infrastructure they don't fully control -- public clouds, customer sites, or third-party environments -- they need a guarantee that no one with infrastructure access can extract or steal the model weights.
This solution provides that guarantee through a full lifecycle of protection:
Model owner side:
- Encrypts the AI model weights with their own encryption keys
- Stores those keys in their own key management system (KMS)
- Defines policies for when and where keys can be released
- Distributes the encrypted model to customers
Customer/platform side:
- Deploys the model on a Kubernetes cluster with confidential computing hardware (CPU TEE + GPU TEE)
- The platform creates a confidential VM with hardware-encrypted memory for both CPU and GPU
- A guest attestation agent inside the VM generates cryptographic evidence proving the environment is genuine
Attestation and key release:
- The attestation evidence (covering both CPU and GPU) is submitted to Trustee
- Trustee verifies the evidence and issues a signed attestation token
- The workload presents this token to the model owner's KMS
- The KMS validates the token, checks its policies, and only then releases the decryption key
- The key is encrypted in transit and only decrypted inside the CPU's protected memory
Runtime protection:
- Model weights are decrypted in CPU protected memory, then transferred to GPU protected memory through a secure channel
- Both CPU and GPU memory are hardware-encrypted and isolated -- no one can dump memory, attach debuggers, intercept data transfers, or modify GPU firmware to bypass protection
- The model owner retains full control: they can revoke keys at any time and maintain a complete audit trail of which environments accessed their model
Termination:
- When the workload stops, both CPU and GPU VMs are destroyed
- CPU and GPU memory are wiped by hardware -- no remnants of model weights or keys remain
- Model providers can distribute encrypted models to customers without risking exposure of their weights -- even on infrastructure they don't control.
- Customers can run inference on sensitive data knowing that neither the cloud provider nor the model owner can see their inputs.
- Regulated industries (finance, healthcare, government) can move AI workloads from proof-of-concept to production in the cloud while meeting compliance requirements for data protection.
The design behind this work started with a formal enhancement proposal shared with the KServe community. KServe PR #5382 was based on this proposal:
The proposal covers how KServe's storage initializer can be extended to decrypt models inside a verified TEE, supporting all major storage backends (OCI registries, S3, Hugging Face, GCS, Azure Blob, HTTPS) using open standards.
| Project | What it does | Link |
|---|---|---|
| KServe | Serves AI models on Kubernetes | kserve.github.io |
| Confidential Containers (CoCo) | Connects confidential VMs to Kubernetes pods | github.com/confidential-containers |
| Trustee | Verifies hardware attestation and releases keys | github.com/confidential-containers/trustee |
| Kata Containers | Runs containers inside lightweight VMs for isolation | github.com/kata-containers |
| vLLM | High-throughput inference engine for large language models | github.com/vllm-project/vllm |
- Introducing Confidential Containers Trustee: Attestation Services Solution Overview and Use Cases
- The power of confidential containers on Red Hat OpenShift with NVIDIA GPUs
- AI meets security: POC to run workloads in confidential containers using NVIDIA accelerated computing
- Confidential Computing Consortium
- Event: Confidential Computing Summit 2026, San Francisco, CA
- Format: Poster session
- Track: Artificial Intelligence / Secure AI Workflows
- Level: Advanced
- Audience: Security architects, AI/ML platform engineers, IT strategy leaders
This project is licensed under the terms of the Apache License 2.0.