Post Syndicated from Ajinkya Patil original https://aws.amazon.com/blogs/security/addressing-protocol-security-gaps-in-multi-tenant-kubernetes/

Protocols like Android Debug Bridge (ADB) and Web Real-Time Communication (WebRTC), by default, use unencrypted channels for network communications and don’t support enterprise authentication mechanisms like JSON Web Tokens (JWTs). In a multi-tenant Kubernetes cluster—including Amazon Elastic Kubernetes Service (Amazon EKS)—pods can reach other tenants’ open ports by default unless you configure network policies to restrict cross-namespace traffic. Also, modifying each protocol individually doesn’t scale. This post describes an architecture built on three reusable components: namespace-scoped mutual TLS (mTLS) gateways, managed TURN relay through Amazon Kinesis Video Streams, and JWT identity chains tied to an enterprise identity provider (IdP). The post focuses on architecture decisions and learnings rather than step-by-step deployment.

This post assumes an EKS cluster with namespace isolation configured, an enterprise IdP that issues JWTs, and AWS Private Certificate Authority (AWS Private CA)—a managed service for issuing private certificates—set up in your account. cert-manager—an open-source Kubernetes add-on that automates certificate issuance and renewal—and the AWS Private CA issuer plugin should already be running in the cluster. The architecture also assumes permissions to deploy Envoy Gateway, a Kubernetes API gateway built on the Envoy proxy. The key authentication mechanism used is mTLS, where both client and server present certificates to verify each other.

ADB, a command-line tool for communicating with Android devices, requires a proxy or custom API to handle authentication; its unencrypted channel exposes data in transit. Virtual Network Computing (VNC), a protocol for remote desktop access, and similar TCP services share this problem: their credentials are local to the protocol, with no path to a central identity store.

In the Cuttlefish deployment (Cuttlefish is Google’s configurable virtual Android device that runs in containers), the first approach routed this traffic through a bastion host to the Amazon EKS control plane, as Figure 1 shows.

Figure 1: Previous architecture (superseded), shown for comparison only. Communication flow between developer workstation, bastion host, and Amazon EKS control plane

Figure 1 shows a common pattern across the industry to access a private Amazon EKS cluster. We share it to help others identify and migrate away from similar designs for low latency environments. Do not implement this design. The current architecture is described and shown later in this post.

In the now deprecated architecture, a custom API handles authentication by checking each user’s identity against the enterprise IdP and enforcing pod-level authorization. The custom API powers a command-line interface (CLI) that lets you list pods in your assigned namespace and connect to your devices.

The JWT identity chain maps each user to an AWS Security Token Service (AWS STS) RoleSessionName, so every AWS API call in the audit log traces back to a specific person, even in a multi-tenant environment.

Those credentials set up a two-hop port forward that goes from your machine to the bastion, then from the bastion to the target pod. Because all traffic was routed through the Kubernetes API server, concurrent sessions created a throughput bottleneck even though the cluster itself stayed private.

A TURN server—a media relay—is needed when communicating parties are behind a NAT Gateway and can’t set up direct peer-to-peer connections.

The initial deployment used STUNner, an open source Kubernetes Session Traversal Utilities for NAT (STUN) and Traversal Using Relays around NAT (TURN) gateway designed for WebRTC media ingestion. STUNner introduced three operational and security challenges.

Because STUNner had no native integration with AWS services or the enterprise IdP, we built a custom token-exchange service to convert IdP tokens into time-limited, namespace-scoped TURN credentials. As usage grew, patching, monitoring, and scaling the TURN servers became a recurring burden. Pod migrations between Availability Zones required manually updating routing rules, and STUNner dropped WebRTC connections during peak usage.

WebRTC’s signaling server introduced a separate security risk. WebRTC encrypts media using Datagram Transport Layer Security (DTLS) and Secure Real-Time Transport Protocol (SRTP) but doesn’t define authentication or encryption for the signaling server; you must build that yourself, which leaves control-plane signaling exposed.

The architecture doesn’t use the Kinesis Video Streams signaling channel because integrating it required substantial changes to the Cuttlefish source patch. The Cuttlefish source code was patched to support non-Google TURN servers, and further changes to support non-Google signaling servers deviated the deployment further from upstream Cuttlefish, risking compatibility issues. Instead, the architecture kept the default Cuttlefish signaling server, with JWT-based session validation and TLS termination at the signaling endpoint designed around it at the application layer.

The architecture retains namespace isolation, JWT-based authentication, the custom authorization API, and the Cuttlefish patch; the last three of which are custom to this deployment. All components except Cuttlefish patch are reusable.

The reusable components include namespace-scoped Envoy Gateway for mTLS-authenticated TCP proxying, cert-manager-driven certificate rotation through AWS Private CA, and Kinesis Video Streams as a managed TURN provider.

The mTLS gateway assumes several other controls are already in place:

• Default-deny NetworkPolicy in each tenant namespace. Without this, traffic can bypass the gateway.
• JWT-based authentication through the custom API. The API validates each user’s identity against the enterprise IdP before returning connection details.
• Namespace-scoped mTLS certificates issued by AWS Private CA through cert-manager.
• Kubernetes namespace isolation with scoped role-based access control (RBAC).

For ADB connectivity, the architecture uses mTLS authentication with Envoy Gateway. Each namespace runs its own Envoy Gateway instance, which routes traffic only to pods within that namespace. cert-manager, running in the cluster with the AWS Private CA issuer plugin, requests short-lived mTLS certificates from AWS Private CA for each namespace at creation time. In production, apply NetworkPolicies before deploying the gateway and issuing certificates.

The certificate commonName uses the convention <user-namespace>.eks.internal so that Envoy can validate the certificate against the expected namespace during the mTLS handshake. The .eks.internal suffix is a naming convention for internal cluster certificates; choose a domain that doesn’t conflict with your organization’s DNS. The usages field specifies both client authentication and server authentication because the same certificate authenticates the client to the gateway and the gateway to the client during the mutual TLS exchange. A cluster-scoped AWSPCAClusterIssuer is used instead of a namespace-scoped AWSPCAIssuer so that a single CA configuration can issue certificates across all tenant namespaces without duplicating issuer resources in each one.

Before deploying Envoy Gateway, each tenant namespace needs a NetworkPolicy that enforces a default-deny ingress posture.

Each certificate is issued with a 12-hour lifetime. A shorter lifetime means a compromised certificate expires quickly without requiring Certificate Revocation List (CRL) or Online Certificate Status Protocol (OCSP) infrastructure, while the 4-hour renewal window gives cert-manager multiple retry attempts before expiry. cert-manager tracks expiry and re-requests a new certificate from AWS Private CA before the current one lapses (by default, renewal starts at the two-thirds mark of the certificate’s lifetime). Envoy Gateway picks up the renewed certificate through Kubernetes Secrets using its Secret Discovery Service (SDS), with no pod restart required.

You first authenticate with the enterprise IdP through the custom CLI, which calls the custom API. The API validates your JWT, confirms your namespace assignment, and returns connection details for the appropriate namespace-scoped Envoy Gateway, including the client certificate for that namespace. The API delivers the certificate over HTTPS. Because certificates are namespace-scoped and expire in 12 hours, a compromised response yields access only to the holder’s own namespace for a limited window. The CLI then presents this certificate to establish a direct mTLS connection, through Envoy, to the target pod. The JWT proves who you are, and the mTLS certificate proves which namespace the connection belongs to.

Envoy proxies the TCP stream directly to the pod and pools connections to reduce latency. The hop between Envoy and the pod isn’t encrypted by mTLS. NetworkPolicy restricts ingress so that only the namespace-scoped Envoy Gateway can reach the ADB port on the target pod. For environments with compliance requirements, extend mTLS to the pod using a sidecar proxy or pod-level certificates. Evaluate this gap against your organization’s data classification and regulatory requirements.

For the REST API path, the custom API sits behind AWS WAF. The ADB data path has no WAF protection; it relies on mTLS, NetworkPolicies, and namespace isolation.

Figure 2: ADB communication flow using Envoy Gateway and custom REST API

Figure 2 shows this ADB communication flow using Envoy Gateway and the custom REST API. It logs your authenticated identity, namespace, and timestamp. The mTLS certificate doesn’t carry a user identifier in its subject field. This applies when namespaces are strictly 1:1 with users. For shared namespaces or regulated environments requiring granular attribution, issue per-user certificates from AWS Private CA.

Kinesis Video Streams provides built-in STUN and TURN infrastructure and signaling channels. It replaced the self-managed STUNner because it integrates with AWS Identity and Access Management (IAM), so the custom token-exchange service is no longer needed. This is an important factor in reducing operational costs. For implementation details, see the Amazon Kinesis Video Streams WebRTC developer guide, which covers signaling channel setup, IAM policies, and TURN credential retrieval.

Kinesis Video Streams WebRTC charges per active signaling channel, per signaling message, and per TURN streaming minute. Idle namespaces incur no TURN cost. See Amazon Kinesis Video Streams pricing for latest pricing and Amazon Kinesis Video Streams with WebRTC service quotas for service quotas.

Figure 3: Final architecture with ADB and WebRTC communication channels

Figure 3 shows both ADB and WebRTC communication channels in the completed architecture. Each path has different controls: ADB traffic flows through the namespace-scoped Envoy Gateway, where mTLS and NetworkPolicies enforce access. WebRTC media relay runs through Kinesis Video Streams, with IAM scoping credentials per session. The custom REST API is the only HTTP-based path; it runs over HTTPS with AWS WAF at the edge.

The JWT identity chain connects all three paths: the custom API validates each user’s identity before any connection opens, so the audit log records who connected to which namespace and when.

The mTLS gateway and certificate rotation pieces transfer directly to most TCP protocols. The TURN relay is specific to WebRTC and similar media-streaming protocols that need NAT traversal. The JWT identity chain requires custom work and isn’t directly transferable. You need to build the API that validates tokens and maps users to namespaces, and a client that retrieves and presents the certificate. The Cuttlefish source patch is specific to this case study and not reusable.

Two common protocol families show how the reusable parts apply:

• VNC and RDP both store credentials locally, with no path to an enterprise identity provider. Wrapping VNC in mTLS through Envoy replaces the shared password model with certificate-based identity. RDP benefits similarly: certificate authentication works without domain-joining the containers, so you don’t need Active Directory inside each one.
• MQTT, SIP, and similar TCP protocols often encrypt the payload but skip session-level authentication entirely; meaning you can verify what was sent but not who sent it. Wrapping these protocols in mTLS through Envoy closes that gap without modifying the protocol itself.

JWT identity chain secures session establishment at the application layer. For Internet of Things (IoT) device management, industrial control, or game streaming, Envoy Gateway adds mTLS to TCP protocols that lack built-in encryption in transit. Managed TURN through Kinesis Video Streams handles NAT traversal for WebRTC-based streaming components.

Application installs through the dual port-forwarding chain (local machine to bastion, bastion to pod) took 3–4 minutes, and individual ADB commands took 1–2 seconds per round trip.

After migrating to the mTLS Envoy Gateway, ADB commands added under 50 ms of gateway latency compared to local setup. The latency was measured as the difference in round-trip time between a command sent through Envoy and one sent to a local Android device. Through Envoy, individual ADB commands completed in under 100 ms and application installs finished in under 30 seconds, because Envoy proxies the TCP stream directly to the pod instead of routing through the Kubernetes API server. This was an improvement from 1–2 seconds for ADB commands and 3–4 minutes for application installs using bastion host setup.

The main concern addressed by replacing bastion-host forwarding with mTLS proxying was latency compared to local setup. Gateway overhead stayed under 50 ms and met developer expectations.

After migrating to Kinesis Video Streams, WebRTC connections stayed stable during peak usage, and pod migrations between Availability Zones no longer required manual updates to routing rules. Because Kinesis Video Streams integrates with IAM, new protocols added to the cluster are automatically covered by the same audit trail with no additional logging configuration needed.

mTLS authenticates both ends of a connection and encrypts traffic in transit, but it only works when other controls are in place. The NetworkPolicy and pod security controls described in the preceding ADB connectivity section must be in place before mTLS adds value. Without default-deny ingress rules, traffic bypasses the gateway entirely. At the application layer, the custom API validates each user’s JWT before issuing connection details and logs the identity, namespace, and timestamp for every session. Certificates expire after 12 hours and cert-manager rotates them automatically. Short-lived certificates reduce exposure, but if your threat model requires immediate revocation, you need CRL or OCSP infrastructure on top of AWS Private CA. Envoy Gateway is a trust boundary. Review its configuration, resource limits, and access controls against the Envoy security best practices.

The custom API is the primary trust boundary in this architecture. Apply the same rigor to its security review, logging, and access controls as you would to any authentication service.

With namespace-scoped mTLS gateways and managed TURN relay, you can secure unauthenticated protocols on shared EKS clusters without modifying the protocols themselves. ADB command latency drops from 1–2 seconds to under 50 ms, and WebRTC connections remain stable during pod migrations. See the Envoy Gateway documentation, cert-manager documentation, and Amazon Kinesis Video Streams WebRTC developer guide to build and deploy this architecture.

If you have questions about this post, contact AWS Support or start a new thread on AWS re:Post.