ReadyOn’s Four Walls of tenant isolation on Amazon EKS

Post Syndicated from Roshan Daneshvaran original https://aws.amazon.com/blogs/architecture/readyons-four-walls-of-tenant-isolation-on-amazon-eks/

ReadyOn, an AWS Partner delivering an intelligent Labor Orchestration Platform for Fortune 100 enterprises, runs a multi-tenant platform on Amazon Elastic Kubernetes Service (Amazon EKS) that handles some of the most sensitive data in enterprise IT. Kubernetes is central to that platform, and it is also one of the most complex surfaces to secure. A default Kubernetes deployment is not secure out of the box: upstream Kubernetes historically lets anonymous requests reach the API server, where they are denied only by RBAC, pods run as root by default, and network policies do not exist until someone writes them.

For single-tenant clusters, hardening is a well-documented challenge. But when you add multi-tenancy, the threat model fundamentally changes. The concern shifts from whether an unauthorized user can reach the cluster to whether Tenant A can access Tenant B’s data. Most multi-tenant Kubernetes platforms rely on a single isolation mechanism: the namespace. Kubernetes designers never intended namespaces to be a security boundary.

ReadyOn’s Harmony platform powers workforce intelligence for Fortune 100 enterprises. It processes payroll data, organizational hierarchies, and operational analytics spanning hundreds of thousands of employees. Cross-tenant access would trigger regulatory obligations across multiple jurisdictions and damage the trust enterprise clients place in their technology partners. The stakes demanded something better than one wall.

This post describes the architecture of ReadyOn’s four independent layers of isolation on Amazon EKS. Their Four Walls model combines Kubernetes namespaces, Karpenter-managed dedicated node pools, Amazon Virtual Private Cloud (Amazon VPC) security groups, and per-tenant Amazon Aurora databases. Together they create a compound defense: to cross a tenant boundary, an unauthorized user would need to simultaneously overcome the Kubernetes API, the node scheduler, the AWS software-defined network, and the data layer.

The Four Walls model

ReadyOn’s architecture places four independent barriers between tenants, each operating at a different layer of the stack and requiring a fundamentally different technique to cross:

  • Wall 1 – Namespace isolation: Each tenant operates in a dedicated Kubernetes namespace with strict RBAC policies, resource quotas, and admission control. An Argo CD ApplicationSet generates these resources, so every tenant receives an identical security posture by construction.
  • Wall 2 – Compute isolation: Karpenter provisions dedicated, auto scaling node pools for each tenant using a dual-taint strategy. Every node carries both a tenant-identifying taint and a workload-type taint. A pod must tolerate both to be scheduled. This configuration prevents cross-tenant pod placement.
  • Wall 3 – Network isolation: Per-tenant Amazon VPC security groups restrict database access to only that tenant’s node pool. Kubernetes network policies enforce default-deny between namespaces. The cluster API server is private, accessible only through VPN with OpenID Connect (OIDC) and MFA.
  • Wall 4 – Data isolation: Each tenant has a dedicated Amazon Aurora database cluster and tenant-scoped secrets in AWS Secrets Manager. Workloads use short-lived credentials through IAM Roles for Service Accounts (IRSA), and each tenant has per-tenant observability instances. There is no shared database with row-level filtering.

The power of this model is layering. An unauthorized user would need to cross all four walls simultaneously to reach the tenant boundary. These are layered, overlapping controls: an unauthorized user must defeat them in combination, not one at a time. Because some layers share a control plane (admission control, GitOps, the EKS control plane, and IAM), ReadyOn treats them as defense in depth rather than as fully independent probabilities.

Wall 1: Namespace isolation on Amazon EKS

The namespace is the most visible tenant boundary in Kubernetes. While ReadyOn is realistic about the limitations of namespace isolation as a security mechanism, the namespace still forms the logical foundation upon which the other isolation layers build.

GitOps-enforced consistency

Every tenant namespace is provisioned through an Argo CD ApplicationSet. When ReadyOn onboards a new tenant, they add a single entry to a configuration manifest in Git. The automation generates all required resources: the namespace, network policies, RBAC bindings, resource quotas, and secrets configurations. There are no “legacy” tenants with weaker policies. Every tenant receives an identical security posture by construction.

RBAC and admission control

Each namespace carries role bindings that restrict API access to only that tenant’s resources. No tenant principal can list, get, or modify resources in another namespace. An admission control framework validates all resources against a policy set that enforces security contexts, blocks privileged configurations, and prevents creation of resource types that tenants should not own (DaemonSets, ClusterRoles, admission webhooks).

The generated RoleBinding scopes a tenant group to its own namespace only. Argo CD renders this from the tenant entry in Git, so every namespace gets the identical binding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-developers
  namespace: tenant-a
subjects:
- kind: Group
  name: tenant-a-developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-namespace-editor
  apiGroup: rbac.authorization.k8s.io

Self-healing drift correction

The GitOps controller continuously reconciles live cluster state against the declared state in Git. Any manual modification (whether from a misconfiguration or an unauthorized user who gains limited API access) is automatically reverted within seconds. A human operator never runs kubectl apply against a production cluster.

Wall 2: Compute isolation with Karpenter

Namespace isolation operates at the Kubernetes API layer. Wall 2 drops to the compute layer: the actual machines where tenant code executes. In Harmony, tenants do not share compute nodes.

The dual-taint strategy

ReadyOn’s Karpenter provisioners create nodes with two taints: a tenant-identifier taint (for example, tenant=acme-corp) and a workload-type taint (for example, workload=frontend). A pod must tolerate both taints to be scheduled. Tenant A’s frontend nodes are distinct from Tenant A’s batch nodes, and both are entirely separate from Tenant B’s infrastructure.

Admission controller validation

Even if a pod is created with forged tolerations, an admission controller validates that the toleration claims match the namespace’s tenant identity. The scheduler cannot be directed into cross-tenant placement.

Scoped node credentials

All application nodes enforce IMDSv2 with a reduced hop limit, preventing pods from reaching the instance metadata endpoint. The IAM role attached to each node is scoped to a single tenant’s resources, limiting the scope of a container escape. Platform nodes run on managed node groups with a separate taint that no tenant workload can tolerate.

Wall 3: Network isolation at the VPC layer

Walls 1 and 2 operate within the Kubernetes abstraction. Wall 3 drops below Kubernetes entirely, enforcing isolation at the Amazon VPC and security group level: infrastructure that the Kubernetes control plane does not manage and that a pod with limited access cannot manipulate.

Per-tenant security groups

Each tenant’s Amazon Aurora cluster is protected by a security group that allows inbound connections only from the security group attached to that tenant’s application nodes. Tenant A’s nodes cannot open a TCP connection to Tenant B’s database. This is an Amazon VPC security group rule enforced by the AWS software-defined network. Crossing it would require overcoming that networking layer itself.

Multi-tier VPC architecture

The VPC is divided into four tiers: a public perimeter tier (load balancers only), an application tier (EKS worker nodes in private subnets), a database tier (Aurora clusters with no internet access), and a control plane tier (private EKS API endpoint, VPN-only access with OIDC and MFA).

Default-deny network policies

Kubernetes network policies enforce default-deny for inter-namespace traffic. Tenant pods communicate only with pods in their own namespace and explicitly allowed platform services. All other traffic is denied and logged. Amazon VPC Flow Logs capture all network traffic for anomaly detection.

Wall 4: Data isolation with Amazon Aurora

The final wall addresses the ultimate target: the data itself. Walls 1 through 3 prevent an unauthorized user from reaching another tenant’s data. Wall 4 maintains data isolation even if all other layers fail.

The case against shared databases

A shared database places the entire burden of isolation on application-layer query logic, where a single missing WHERE tenant_id = ? clause can result in inadvertent disclosure. ReadyOn chose dedicated Amazon Aurora clusters per tenant. There is no row-level filter to forget. Tenant A’s code cannot construct a connection to Tenant B’s database: it lacks the endpoint, the credentials, and the network path.

Dedicated Aurora clusters also support unique AWS Key Management Service (AWS KMS) encryption keys per tenant. Each tenant’s data-at-rest is encrypted with a distinct KMS key, meaning that even if raw storage were inadvertently accessed, one tenant’s key cannot decrypt another tenant’s data. Equivalent per-tenant key separation is complex to achieve in a shared database, where every tenant’s rows share the same storage and separation depends on correctly implemented row-level security.

IRSA: No long-lived workload credentials

Every workload authenticates to AWS APIs by using short-lived credentials from AWS Security Token Service (AWS STS), issued through OIDC federation between the EKS cluster and IAM. Each tenant’s workloads assume a unique IAM role scoped to only that tenant’s resources. Credentials expire within minutes. Application workloads hold no long-lived access keys.

Per-tenant observability

Each tenant’s metrics, logs, and traces are routed to dedicated observability backends through OpenTelemetry. Even a spike in error rates (which could reveal sensitive operational intelligence) is invisible to other tenants.

Breaking the threat sequence at every stage

ReadyOn maps their defenses to their own 10-stage multi-tenant threat model, with each stage mapped to the relevant MITRE ATT&CK techniques. The critical inflection point is their lateral movement stage (Stage 8), where an unauthorized user in a Tenant A pod with limited access attempts to cross the boundary to Tenant B’s resources. Figure 1 illustrates this inflection point, showing an unauthorized process inside a Tenant A pod attempting to reach Tenant B and being stopped at each of the four walls.

Diagram of a Tenant A pod on the left and Tenant B resources on the right, separated by four numbered walls; six cross-tenant attack paths each stop at the wall that blocks them, and none reach Tenant B

Figure 1: Stage 8 lateral movement, where each of the six cross-tenant paths a compromised Tenant A workload might attempt terminates at the numbered wall that blocks it

Figure 1 depicts a Tenant A pod on the left and Tenant B’s resources on the right. Four labeled vertical barriers sit between them, numbered 1 through 4, representing the four walls in order: (1) namespace isolation, (2) compute isolation, (3) network isolation, and (4) data isolation. Each of the six cross-tenant paths an unauthorized user might attempt is drawn as an arrow from Tenant A that terminates at the numbered wall that blocks it: creating a route into another namespace stops at wall 1. Scheduling a pod on another tenant’s nodes stops at wall 2. Opening a database connection and sending cross-namespace traffic stop at wall 3. And retrieving secrets and querying monitoring data stop at wall 4. No arrow reaches Tenant B.

At Stage 8, all four walls apply simultaneously. ReadyOn systematically validates that every cross-tenant path is blocked:

  • Scheduling workloads on another tenant’s nodes is blocked by the dual-taint strategy plus admission controller validation.
  • Reaching another tenant’s database is blocked by per-tenant security groups at the Amazon VPC layer.
  • Reading another tenant’s secrets is blocked by IRSA scoping plus tenant-scoped secrets paths.
  • Querying another tenant’s monitoring data is blocked by dedicated per-tenant observability instances.
  • Communicating with pods in another namespace is blocked by default-deny network policies.
  • Creating a route that directs traffic into another tenant’s namespace is blocked by policy: tenants cannot create or modify ingress resources, and the platform ingress layer re-validates tenant context on every request.

ReadyOn validates these boundaries through regular adversarial exercises that simulate a tenant pod with full access and attempt every crossing path. In these exercises to date, no test has produced a successful cross-tenant path.

Workload hardening: The innermost defense

Every container in Harmony runs with Pod Security Standards at the “restricted” level:

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: [ALL]
  seccompProfile:
    type: RuntimeDefault

Containers cannot install tools, modify binaries, or use the kernel interfaces that container escape techniques depend on. Combined with IMDSv2 restrictions and IRSA, even a container escape does not yield useful node-level credentials.

GitOps as the security control plane

Every change to cluster state begins as a pull request requiring review and automated policy checks. The Git repository is the single source of truth. The cluster is a reflection of Git. Zero secrets are stored in Git: the External Secrets Operator resolves references to AWS Secrets Manager at deployment time. Even unauthorized access to the Git repository yields zero usable credentials.

Benefits

ReadyOn’s Four Walls architecture delivers measurable security and operational benefits:

  • 10 threat model stages defended, each mapped to MITRE ATT&CK techniques, with all six cross-tenant paths blocked at the lateral movement stage.
  • No long-lived credentials in application workloads: all workload authentication uses short-lived OIDC-federated tokens.
  • Consistent security posture across all tenants enforced by GitOps templating, with no legacy exceptions.
  • Self-healing drift correction reverts unauthorized cluster changes within seconds.
  • Multi-region active-passive architecture with automated failover through Amazon Route 53.

Conclusion

Multi-tenant Kubernetes security is not a single problem with a single solution. Namespace isolation is a strong logical boundary, and the Kubernetes documentation recommends pairing it with additional layers rather than relying on it alone as a security boundary. ReadyOn built those additional layers.

ReadyOn’s Four Walls model demonstrates that defense-in-depth is achievable on AWS by layering native services at independent abstraction layers: Kubernetes RBAC at the API layer, Karpenter at the scheduler layer, Amazon VPC security groups at the network layer, and IAM at the data access layer. Each wall independently raises the cost of a cross-tenant attempt, and together they provide defense in depth: an unauthorized user must defeat the walls in combination, not one at a time.

For organizations running sensitive multi-tenant workloads on Amazon EKS, the Four Walls model provides a reference architecture for zero-trust tenant isolation without sacrificing the economics and operational velocity that multi-tenancy delivers.

Learn more

About the authors