Post Syndicated from Alex Graf original https://aws.amazon.com/blogs/compute/nitro-sandboxes-hardware-isolation-for-every-eks-pod-ecs-task-and-ec2-workload/
Security has been the foundation of AWS cloud computing since day one. When we built Amazon Elastic Compute Cloud (Amazon EC2), we designed it with strong isolation that keeps customer workloads and data separated from every other tenant on the same hardware. The AWS Nitro System took that further by moving virtualization, storage, and networking onto dedicated hardware, making zero operator access to customer data the default. Customers then asked us for more. They wanted tightly scoped, attestable environments for their most sensitive processing: the code and data that required protection even from their own administrators. That’s why we built Nitro Enclaves. Today, more of those sensitive workloads run in containers on Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), and self-managed Kubernetes and container runtimes on Amazon EC2. Customers are asking for the same level of isolation and confidentiality for container environments.
Container environments today host many independent workloads on a single EC2 instance by using Linux container technology. For a lot of customers, Linux containers have become the prevalent workload delivery vehicle because of their ease of deployment and the ability to have them share resources in one host through a shared kernel. This sharing has some downsides. Operators of a shared cluster have access to all the tasks running in it. A weakness in the shared kernel reachable by one container also exposes every other container on the host. With the current rate of Linux Common Vulnerabilities and Exposures (CVEs), containerized workloads need an architecture that isolates each workload from other workloads and from cluster operators.
Introducing AWS Nitro Sandboxes
This week we introduced AWS Nitro Sandboxes, a new Amazon EC2 building block that gives each container the boundary of a hardware-isolated virtual machine (VM). Each sandbox gets the security properties of the Nitro System (strong isolation, zero operator access to customer data, and attestation) while keeping the familiar container experience. And isolation is designed so that no one outside the Nitro Sandbox can read its contents: Neither the cluster administrator, the platform team, nor AWS can read EKS pod or ECS task memory.
A Nitro Sandbox is a lightweight virtual machine that you launch and control from your EC2 instance. To your instance, it behaves like a child VM. Under the hood, it runs as a separate VM alongside your instance. The Nitro System enforces separation between the sandbox and its parent instance, between sandboxes, and between the sandbox and AWS operators. The parent instance contributes vCPU, memory, and I/O. Nitro keeps the workload isolated and, when you ask, proves what code it is running.
Because a Nitro Sandbox is a first-level guest with no nested hypervisor, and the Nitro Hypervisor enforces its CPU and memory boundary in hardware, compute and memory run at near-native EC2 performance. Instead of needing a separate instance for every workload or customer that needs a hard boundary, you can consolidate many isolated workloads onto instances you already run. You can then improve utilization and lower cost without giving up the boundary that made separate instances attractive in the first place.
Nitro Sandboxes share the same Nitro System isolation technology that we have operated for years as Nitro Enclaves. With Nitro Enclaves, we designed everything around the smallest possible surface: You dedicate full CPU cores and memory to a separate VM that you can only reach over a vsock channel. With Nitro Sandboxes, we built on the same foundation, but adjusted the experience to the needs of container workloads in three ways. First, a sandbox’s vCPUs can share cores with the parent instance through time slicing. Second, the parent can attach standard virtio network and block devices, not only a vsock channel, so the sandbox gets real networking and storage. Third, you can run the sandbox through a standard virtual machine monitor (VMM) such as Firecracker. The result is a general-purpose, hardware-isolated environment that behaves like a normal VM and runs almost anywhere EC2 runs.
We then took Nitro Sandboxes and natively integrated them into Amazon EKS and Amazon ECS, providing a turnkey solution to Nitro-backed, confidential virtual-machine-based container isolation at scale. Nitro Sandboxes are broadly available with the following features:
- Instances: Every EC2 instance type that supports Nitro Enclaves, across general purpose, compute optimized, memory optimized, storage optimized, and accelerated families, on AWS Graviton, Intel, and AMD processors, in commercial Regions worldwide, at no additional charge.
- Storage: Sandboxes reach storage the same way an EC2 instance does: local instance store for scratch space, Amazon Elastic Block Store (Amazon EBS) for persistent volumes, and Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS) over the network. To keep data confidential from the parent, you can use attestation-gated AWS Key Management Service (AWS KMS) keys, the same model as Nitro Enclaves, now integrated into Nitro Sandboxes, EKS, and ECS.
- Networking: Each sandbox runs its own kernel, and therefore its own network stack. The parent connects that stack to your Amazon Virtual Private Cloud (Amazon VPC) the same way it does for containers today. On EKS, pods keep their own VPC IP address through the Amazon VPC CNI. On ECS, each task gets its own elastic network interface in
awsvpcmode. Network policies and security groups are enforced on the parent instance, outside the sandbox, so a compromised workload cannot turn them off.
With Nitro Sandboxes, you can give every workload that needs a hard boundary its own VM, without giving it its own instance. AI agents that write and run their own code can run each session in a separate sandbox. Code that a prompt injection gets the agent to run stays inside one VM, which is deleted when the session ends. Software as a service (SaaS) providers can run many tenants on shared EKS nodes or ECS Managed Instances, each in its own sandbox, instead of giving each tenant a dedicated instance. Continuous integration and continuous delivery (CI/CD) pipelines can run privileged builds and untrusted pull request code inside a per-job VM rather than on dedicated build machines. And with confidential sandboxes, model providers can deploy closed-weight models in a customer’s account, knowing the weights are decrypted only inside an attested sandbox. Regulated industries can give auditors cryptographic proof of exactly what ran.
Isolation by design
Containers on a shared node all trust one Linux kernel. A single kernel issue can lead to data exposure across pods or tasks. With Nitro Sandboxes, each pod, task, or agent runs in its own VM with its own kernel. A kernel issue in one sandbox doesn’t affect its neighbors.
The Nitro Hypervisor establishes strong, virtual-machine-based security isolation through its leading security posture and built-in confidential computing which we continuously extend. The hypervisor removes the parent’s access to the sandbox’s memory and CPU register state. Only I/O buffers that the sandbox explicitly shares stay visible to the parent.
Nitro Sandboxes can therefore treat the parent instance as untrusted and attest to what runs inside. They use the Nitro-based Nitro Secure Module (NSM) to gain an attestation document in the same style as Nitro Enclaves. A workload can be configured to get its secrets only after it proves it is running the code you approved. The system is designed so that no one outside the sandbox can read its contents: Neither the cluster administrator, the platform team, nor AWS can read EKS pod or ECS task memory. You write AWS KMS key policies that require specific attestation measurements before AWS KMS releases a decryption key. This creates a verifiable link between who can decrypt and what code is actually running.
Isolation modes in EKS and ECS
On top of Nitro Sandboxes, EKS and ECS provide two isolation modes. With these modes, you can choose between faster launch times and cryptographic attestation:
- Confidential sandbox: Sandboxes are fully attestable. Image loading and other runtime operations are gated by a Cedar policy, image contents are verified and extracted inside the sandbox, and the ephemeral layer is encrypted. AWS KMS releases your secrets only to workloads that prove they run the code you approved. Use this mode for anything that handles secrets, including when you want protection from your own operators. SaaS providers can show their customers that tenant data is protected through attestation. Regulated industries can give auditors cryptographic evidence in addition to documentation.
- Isolated sandbox: Each pod (EKS) or task (ECS) runs in its own hardware-isolated VM with its own kernel, protecting the node and neighboring workloads from anything that happens inside the sandbox. The node and its operators are trusted, just as they are for pods and tasks today, and image validation works the same as it does without sandboxes. For workloads that also need protection from the node operator, use confidential sandbox.
Getting started on Amazon EKS
To get started on EKS, use EKS Auto Mode. Add runtimeClassName to your pod spec, and EKS handles instance selection, Nitro component setup, and scheduling. Use confidential-sandbox for a confidential sandbox or isolated-sandbox if you do not need protection from node operators. For self-managed EKS, you will need a compatible Amazon Machine Image (AMI) to support Nitro Sandboxes.
To learn more about the EKS experience with Nitro Sandboxes, see the Amazon EKS blog post.
Getting started on Amazon ECS
You can use Nitro Sandboxes on ECS with ECS Managed Instances. ECS launches compatible EC2 instances in your account, sets up a fully configured Bottlerocket image, and runs each task in its own Nitro Sandbox. ECS supports the two isolation modes, CONFIDENTIAL_SANDBOX and ISOLATED_SANDBOX.
Each task reports its isolation mode in a read-only trustedExecutionSummary field when you describe it. Per-task security groups, VPC Flow Logs, Application Load Balancer, Network Load Balancer, and Amazon VPC Lattice work as before. To learn more about the ECS experience with Nitro Sandboxes, see the Amazon ECS Developer Guide.
Getting started on Amazon EC2
If you manage your own instances, or build your own platform, you can launch Nitro Sandboxes directly on nearly any instance built on the AWS Nitro System. Amazon Linux and Firecracker support Nitro Sandboxes today. We have started to upstream support for Nitro Sandboxes in Linux and other virtual machine monitors. We also worked with operating system vendor partners to add support early, so you can keep the stack your workload already depends on.
In addition to containers, Nitro Sandboxes serve as a building block for more in-instance virtualization use cases. Nitro Sandboxes support a special “Direct Mode” switch which disables attestation, but allows the parent to retain access to sandbox memory. We have seen early adopters use this mode to execute non-enlightened operating systems, including Windows or Android, in sandboxes. Other customers have used Nitro Sandboxes to create fully attested package build environments, where each built package receives a tamper-proof signature of its build inputs. See the Amazon EC2 User Guide for details.
Pricing and availability
Nitro Sandboxes, along with the EKS and ECS integrations, are available at no additional charge. You pay for the EC2 instances, plus the standard EKS Auto Mode or ECS Managed Instances management fee if you use those capabilities. Nitro Sandboxes run on nearly every EC2 instance, EKS cluster, and ECS cluster, so strong per-container, per-task, and in-instance isolation is available to everyone.
For more information, see the Nitro Sandboxes documentation in the Amazon EKS User Guide, the Amazon ECS Developer Guide, or the Amazon EC2 User Guide for direct use on EC2 instances.