All posts by Ajinkya Patil

Addressing protocol security gaps in multi-tenant Kubernetes

Post Syndicated from Ajinkya Patil original https://aws.amazon.com/blogs/security/addressing-protocol-security-gaps-in-multi-tenant-kubernetes/

Protocols like Android Debug Bridge (ADB) and Web Real-Time Communication (WebRTC), by default, use unencrypted channels for network communications and don’t support enterprise authentication mechanisms like JSON Web Tokens (JWTs). In a multi-tenant Kubernetes cluster—including Amazon Elastic Kubernetes Service (Amazon EKS)—pods can reach other tenants’ open ports by default unless you configure network policies to restrict cross-namespace traffic. Also, modifying each protocol individually doesn’t scale. This post describes an architecture built on three reusable components: namespace-scoped mutual TLS (mTLS) gateways, managed TURN relay through Amazon Kinesis Video Streams, and JWT identity chains tied to an enterprise identity provider (IdP). The post focuses on architecture decisions and learnings rather than step-by-step deployment.

This post assumes an EKS cluster with namespace isolation configured, an enterprise IdP that issues JWTs, and AWS Private Certificate Authority (AWS Private CA)—a managed service for issuing private certificates—set up in your account. cert-manager—an open-source Kubernetes add-on that automates certificate issuance and renewal—and the AWS Private CA issuer plugin should already be running in the cluster. The architecture also assumes permissions to deploy Envoy Gateway, a Kubernetes API gateway built on the Envoy proxy. The key authentication mechanism used is mTLS, where both client and server present certificates to verify each other.

The challenge: Unauthenticated protocols in multi-tenant Kubernetes

ADB, a command-line tool for communicating with Android devices, requires a proxy or custom API to handle authentication; its unencrypted channel exposes data in transit. Virtual Network Computing (VNC), a protocol for remote desktop access, and similar TCP services share this problem: their credentials are local to the protocol, with no path to a central identity store.

In the Cuttlefish deployment (Cuttlefish is Google’s configurable virtual Android device that runs in containers), the first approach routed this traffic through a bastion host to the Amazon EKS control plane, as Figure 1 shows.

Figure 1: Previous architecture (superseded), shown for comparison only. Communication flow between developer workstation, bastion host, and Amazon EKS control plane

Figure 1: Previous architecture (superseded), shown for comparison only. Communication flow between developer workstation, bastion host, and Amazon EKS control plane

Figure 1 shows a common pattern across the industry to access a private Amazon EKS cluster. We share it to help others identify and migrate away from similar designs for low latency environments. Do not implement this design. The current architecture is described and shown later in this post.

In the now deprecated architecture, a custom API handles authentication by checking each user’s identity against the enterprise IdP and enforcing pod-level authorization. The custom API powers a command-line interface (CLI) that lets you list pods in your assigned namespace and connect to your devices.

The JWT identity chain maps each user to an AWS Security Token Service (AWS STS) RoleSessionName, so every AWS API call in the audit log traces back to a specific person, even in a multi-tenant environment.

Those credentials set up a two-hop port forward that goes from your machine to the bastion, then from the bastion to the target pod. Because all traffic was routed through the Kubernetes API server, concurrent sessions created a throughput bottleneck even though the cluster itself stayed private.

Why self-managed relay infrastructure falls short

A TURN server—a media relay—is needed when communicating parties are behind a NAT Gateway and can’t set up direct peer-to-peer connections.

The initial deployment used STUNner, an open source Kubernetes Session Traversal Utilities for NAT (STUN) and Traversal Using Relays around NAT (TURN) gateway designed for WebRTC media ingestion. STUNner introduced three operational and security challenges.

Because STUNner had no native integration with AWS services or the enterprise IdP, we built a custom token-exchange service to convert IdP tokens into time-limited, namespace-scoped TURN credentials. As usage grew, patching, monitoring, and scaling the TURN servers became a recurring burden. Pod migrations between Availability Zones required manually updating routing rules, and STUNner dropped WebRTC connections during peak usage.

WebRTC’s signaling server introduced a separate security risk. WebRTC encrypts media using Datagram Transport Layer Security (DTLS) and Secure Real-Time Transport Protocol (SRTP) but doesn’t define authentication or encryption for the signaling server; you must build that yourself, which leaves control-plane signaling exposed.

The architecture doesn’t use the Kinesis Video Streams signaling channel because integrating it required substantial changes to the Cuttlefish source patch. The Cuttlefish source code was patched to support non-Google TURN servers, and further changes to support non-Google signaling servers deviated the deployment further from upstream Cuttlefish, risking compatibility issues. Instead, the architecture kept the default Cuttlefish signaling server, with JWT-based session validation and TLS termination at the signaling endpoint designed around it at the application layer.

Namespace-scoped mTLS gateways and managed media relay

The architecture retains namespace isolation, JWT-based authentication, the custom authorization API, and the Cuttlefish patch; the last three of which are custom to this deployment. All components except Cuttlefish patch are reusable.

The reusable components include namespace-scoped Envoy Gateway for mTLS-authenticated TCP proxying, cert-manager-driven certificate rotation through AWS Private CA, and Kinesis Video Streams as a managed TURN provider.

The mTLS gateway assumes several other controls are already in place:

  • Default-deny NetworkPolicy in each tenant namespace. Without this, traffic can bypass the gateway.
  • JWT-based authentication through the custom API. The API validates each user’s identity against the enterprise IdP before returning connection details.
  • Namespace-scoped mTLS certificates issued by AWS Private CA through cert-manager.
  • Kubernetes namespace isolation with scoped role-based access control (RBAC).

ADB connectivity through mTLS

For ADB connectivity, the architecture uses mTLS authentication with Envoy Gateway. Each namespace runs its own Envoy Gateway instance, which routes traffic only to pods within that namespace. cert-manager, running in the cluster with the AWS Private CA issuer plugin, requests short-lived mTLS certificates from AWS Private CA for each namespace at creation time. In production, apply NetworkPolicies before deploying the gateway and issuing certificates.

The certificate commonName uses the convention <user-namespace>.eks.internal so that Envoy can validate the certificate against the expected namespace during the mTLS handshake. The .eks.internal suffix is a naming convention for internal cluster certificates; choose a domain that doesn’t conflict with your organization’s DNS. The usages field specifies both client authentication and server authentication because the same certificate authenticates the client to the gateway and the gateway to the client during the mutual TLS exchange. A cluster-scoped AWSPCAClusterIssuer is used instead of a namespace-scoped AWSPCAIssuer so that a single CA configuration can issue certificates across all tenant namespaces without duplicating issuer resources in each one.

Before deploying Envoy Gateway, each tenant namespace needs a NetworkPolicy that enforces a default-deny ingress posture.

Each certificate is issued with a 12-hour lifetime. A shorter lifetime means a compromised certificate expires quickly without requiring Certificate Revocation List (CRL) or Online Certificate Status Protocol (OCSP) infrastructure, while the 4-hour renewal window gives cert-manager multiple retry attempts before expiry. cert-manager tracks expiry and re-requests a new certificate from AWS Private CA before the current one lapses (by default, renewal starts at the two-thirds mark of the certificate’s lifetime). Envoy Gateway picks up the renewed certificate through Kubernetes Secrets using its Secret Discovery Service (SDS), with no pod restart required.

You first authenticate with the enterprise IdP through the custom CLI, which calls the custom API. The API validates your JWT, confirms your namespace assignment, and returns connection details for the appropriate namespace-scoped Envoy Gateway, including the client certificate for that namespace. The API delivers the certificate over HTTPS. Because certificates are namespace-scoped and expire in 12 hours, a compromised response yields access only to the holder’s own namespace for a limited window. The CLI then presents this certificate to establish a direct mTLS connection, through Envoy, to the target pod. The JWT proves who you are, and the mTLS certificate proves which namespace the connection belongs to.

Envoy proxies the TCP stream directly to the pod and pools connections to reduce latency. The hop between Envoy and the pod isn’t encrypted by mTLS. NetworkPolicy restricts ingress so that only the namespace-scoped Envoy Gateway can reach the ADB port on the target pod. For environments with compliance requirements, extend mTLS to the pod using a sidecar proxy or pod-level certificates. Evaluate this gap against your organization’s data classification and regulatory requirements.

For the REST API path, the custom API sits behind AWS WAF. The ADB data path has no WAF protection; it relies on mTLS, NetworkPolicies, and namespace isolation.

Figure 2: ADB communication flow using Envoy Gateway and custom REST API

Figure 2: ADB communication flow using Envoy Gateway and custom REST API

Figure 2 shows this ADB communication flow using Envoy Gateway and the custom REST API. It logs your authenticated identity, namespace, and timestamp. The mTLS certificate doesn’t carry a user identifier in its subject field. This applies when namespaces are strictly 1:1 with users. For shared namespaces or regulated environments requiring granular attribution, issue per-user certificates from AWS Private CA.

WebRTC relay through Kinesis Video Streams

Kinesis Video Streams provides built-in STUN and TURN infrastructure and signaling channels. It replaced the self-managed STUNner because it integrates with AWS Identity and Access Management (IAM), so the custom token-exchange service is no longer needed. This is an important factor in reducing operational costs. For implementation details, see the Amazon Kinesis Video Streams WebRTC developer guide, which covers signaling channel setup, IAM policies, and TURN credential retrieval.

Kinesis Video Streams WebRTC charges per active signaling channel, per signaling message, and per TURN streaming minute. Idle namespaces incur no TURN cost. See Amazon Kinesis Video Streams pricing for latest pricing and Amazon Kinesis Video Streams with WebRTC service quotas for service quotas.

Figure 3: Final architecture with ADB and WebRTC communication channels

Figure 3: Final architecture with ADB and WebRTC communication channels

Figure 3 shows both ADB and WebRTC communication channels in the completed architecture. Each path has different controls: ADB traffic flows through the namespace-scoped Envoy Gateway, where mTLS and NetworkPolicies enforce access. WebRTC media relay runs through Kinesis Video Streams, with IAM scoping credentials per session. The custom REST API is the only HTTP-based path; it runs over HTTPS with AWS WAF at the edge.

The JWT identity chain connects all three paths: the custom API validates each user’s identity before any connection opens, so the audit log records who connected to which namespace and when.

Applying this pattern to other unauthenticated protocols

The mTLS gateway and certificate rotation pieces transfer directly to most TCP protocols. The TURN relay is specific to WebRTC and similar media-streaming protocols that need NAT traversal. The JWT identity chain requires custom work and isn’t directly transferable. You need to build the API that validates tokens and maps users to namespaces, and a client that retrieves and presents the certificate. The Cuttlefish source patch is specific to this case study and not reusable.

Two common protocol families show how the reusable parts apply:

  • VNC and RDP both store credentials locally, with no path to an enterprise identity provider. Wrapping VNC in mTLS through Envoy replaces the shared password model with certificate-based identity. RDP benefits similarly: certificate authentication works without domain-joining the containers, so you don’t need Active Directory inside each one.
  • MQTT, SIP, and similar TCP protocols often encrypt the payload but skip session-level authentication entirely; meaning you can verify what was sent but not who sent it. Wrapping these protocols in mTLS through Envoy closes that gap without modifying the protocol itself.

JWT identity chain secures session establishment at the application layer. For Internet of Things (IoT) device management, industrial control, or game streaming, Envoy Gateway adds mTLS to TCP protocols that lack built-in encryption in transit. Managed TURN through Kinesis Video Streams handles NAT traversal for WebRTC-based streaming components.

Performance results

Application installs through the dual port-forwarding chain (local machine to bastion, bastion to pod) took 3–4 minutes, and individual ADB commands took 1–2 seconds per round trip.

After migrating to the mTLS Envoy Gateway, ADB commands added under 50 ms of gateway latency compared to local setup. The latency was measured as the difference in round-trip time between a command sent through Envoy and one sent to a local Android device. Through Envoy, individual ADB commands completed in under 100 ms and application installs finished in under 30 seconds, because Envoy proxies the TCP stream directly to the pod instead of routing through the Kubernetes API server. This was an improvement from 1–2 seconds for ADB commands and 3–4 minutes for application installs using bastion host setup.

Key implementation lessons

The main concern addressed by replacing bastion-host forwarding with mTLS proxying was latency compared to local setup. Gateway overhead stayed under 50 ms and met developer expectations.

After migrating to Kinesis Video Streams, WebRTC connections stayed stable during peak usage, and pod migrations between Availability Zones no longer required manual updates to routing rules. Because Kinesis Video Streams integrates with IAM, new protocols added to the cluster are automatically covered by the same audit trail with no additional logging configuration needed.

Security considerations

mTLS authenticates both ends of a connection and encrypts traffic in transit, but it only works when other controls are in place. The NetworkPolicy and pod security controls described in the preceding ADB connectivity section must be in place before mTLS adds value. Without default-deny ingress rules, traffic bypasses the gateway entirely. At the application layer, the custom API validates each user’s JWT before issuing connection details and logs the identity, namespace, and timestamp for every session. Certificates expire after 12 hours and cert-manager rotates them automatically. Short-lived certificates reduce exposure, but if your threat model requires immediate revocation, you need CRL or OCSP infrastructure on top of AWS Private CA. Envoy Gateway is a trust boundary. Review its configuration, resource limits, and access controls against the Envoy security best practices.

The custom API is the primary trust boundary in this architecture. Apply the same rigor to its security review, logging, and access controls as you would to any authentication service.

Conclusion

With namespace-scoped mTLS gateways and managed TURN relay, you can secure unauthenticated protocols on shared EKS clusters without modifying the protocols themselves. ADB command latency drops from 1–2 seconds to under 50 ms, and WebRTC connections remain stable during pod migrations. See the Envoy Gateway documentation, cert-manager documentation, and Amazon Kinesis Video Streams WebRTC developer guide to build and deploy this architecture.

If you have questions about this post, contact AWS Support or start a new thread on AWS re:Post.

Ajinkya Patil

Ajinkya Patil

Ajinkya is a Senior Security Consultant at Amazon Professional Services, specializing in security consulting for AWS customers across the automotive, finance, and media industries since 2019. He has presented at AWS re:inforce and contributed articles to the AWS Security Blog and AWS Prescriptive Guidance. Outside of work, he enjoys traveling and photography.

Introducing the AWS Network Firewall CloudWatch Dashboard

Post Syndicated from Ajinkya Patil original https://aws.amazon.com/blogs/security/introducing-the-aws-network-firewall-cloudwatch-dashboard/

Amazon CloudWatch dashboards are customizable pages in the CloudWatch console that you can use to monitor your resources in a single view. This post focuses on deploying a CloudWatch dashboard that you can use to create a customizable monitoring solution for your AWS Network Firewall firewall. It’s designed to provide deeper insights into your firewall’s performance and security events simplifying security monitoring.

Network Firewall is a managed service that you can use to deploy essential network protections to Amazon Virtual Private Clouds (Amazon VPCs). Network Firewall provides comprehensive logs and metrics through CloudWatch, and we’re expanding its capabilities with this CloudWatch dashboard. This enhancement makes it easier to visualize, analyze, and act on the wealth of data generated by your firewall.

This open source solution streamlines network security monitoring with a user-friendly AWS CloudFormation template that quickly deploys a dedicated monitoring dashboard. This solution incorporates a suite of CloudWatch features—basic monitoring metrics, vended logs, Logs Insights queries, Contributor Insights rules, and the dashboard itself—into a centralized view. Preconfigured widgets provide instant insights into critical areas such as top talkers, protocol distributions, and alert log trends, in addition to HTTP and TLS flow analysis. A consolidated view of key metrics and logs enables faster identification of potential security threats or performance issues. With all of this relevant network firewall data in one place, your team can respond more quickly to emerging security events.

In this blog post, we provide an overview of the dashboard and a step-by-step guide to deploy it in your environment.

Solution overview

The CloudWatch dashboard can be deployed in all AWS Regions where Network Firewall is available today, including the AWS GovCloud (US) Regions and China Regions. While the dashboard comes pre-configured, you can quickly adjust queries, time ranges, and refresh intervals to help meet your specific needs. By default, the dashboard queries firewall flow and alert log events over a 3-hour period, impacting the number of log events scanned. Logs Insights and Contributor Insights widgets showcase the top 10 data points by default, but you can enhance results by modifying queries or adjusting the Top Contributors value, though this might lead to increased costs. You can configure the auto-refresh interval of the widgets to get real-time visibility and optimize costs. See the Amazon CloudWatch Pricing guide for up-to-date free and paid tier pricing considerations.

The dashboard, shown in Figure 1, can be deployed using CloudFormation and includes data and analytics from the following sources:

  1. Native CloudWatch metrics from the AWS/NetworkFirewall and AWS/PrivateLinkEndpoints namespaces
  2. CloudWatch Logs Insights queries that analyze Network Firewall flow and alert logs
  3. CloudWatch Contributor Insights rules that aggregate data from Network Firewall flow and alert logs.

Figure 1: CloudWatch dashboard

Figure 1: CloudWatch dashboard

Walkthrough

In the dashboard, the Logs Insights and Contributor Insights widgets display the top 10 data points by default. You can edit the Insights queries or change the Top Contributors to a larger value to display more results, as shown in Figure 2.

Figure 2: Top Talkers dashboard showing a change to the Top Contributors value

Figure 2: Top Talkers dashboard showing a change to the Top Contributors value

You can also manually refresh the data within a single or multiple widgets, or you can configure the entire dashboard to automatically refresh at a configured time interval as shown in Figure 3. The dashboard won’t automatically refresh the widget data by default.

Figure 3: Configuring the dashboard to automatically refresh

Figure 3: Configuring the dashboard to automatically refresh

Prerequisites

Deploying the Network Firewall CloudWatch Dashboard is straightforward. You will need the following:

  1. A Network Firewall in your VPC.
  2. Your Network Firewall must be configured to publish firewall flow and alert logs to two different CloudWatch log groups. For example, firewall flow logs are published to /my-firewall-flow-logs and alert logs are published to /my-firewall-alert-logs.

If you haven’t deployed Network Firewall in your VPC, you can use one of the available AWS Network Firewall Deployment Architecture templates to create a firewall. After creating a firewall, configure CloudWatch log groups for the firewall flow and alert logs and configure stateful logging as described previously. Fine-tune your firewall policy and rule configuration and make sure that you’re routing traffic symmetrically through the firewall. With the firewall now in the routed path and publishing metrics and log events, you can proceed with this Network Firewall CloudWatch dashboard template.

Deployment

The Network Firewall dashboard CloudFormation template creates a monitoring dashboard for a single Network Firewall firewall. Make sure that you launch this CloudFormation stack in the same AWS Region and account as the firewall, regardless of whether the firewall is set up centrally or in a distributed manner.

To deploy the dashboard:

  1. Choose Launch Stack for the relevant AWS Region. Make sure that you’re signed in to the appropriate AWS account and Region.
    • Region: China

      Launch Stack

    • Region: Gov Cloud

      Launch Stack

    • Region: All other regions supported by AWS Network Firewall

      Launch Stack

  2. You will be redirected to the Create stack page in the AWS Management Console for CloudFormation. Make sure that you’re in the correct Region and using the correct template. Choose Next. The following are the Regions and their template names:
    1. China Region: nfw-cloudwatch-dashboard-china.yaml
    2. Gov Cloud Region: nfw-cloudwatch-dashboard-govcloud.yaml
    3. All other Regions: nfw-cloudwatch-dashboard.yaml
Figure 4: Make sure that you’re using the correct template

Figure 4: Make sure that you’re using the correct template

When launching the stack, you will need to enter the following parameters:

  1. Stack name: A descriptive name for this CloudFormation stack. For example, my-firewall-dashboard.
  2. Firewall name: The firewall name as seen in the Amazon VPC console. In the Amazon VPC console, choose Network Firewall in the navigation pane, then choose Firewalls.
  3. Firewall subnets: The firewall subnet IDs to which your firewall endpoints are attached. The firewall subnets can be found on the Firewall details tab of your firewall in the Amazon VPC
  4. Flow log group name: The name of the CloudWatch log group where your firewall flow logs are stored.
  5. Alert log group name: The name of the CloudWatch log group where your firewall alert logs are stored.
  6. Contributor Insights rule state: Enable or disable the Contributor Insights rules (the template defaults to enabled). Disabling will stop the rules from scanning log data and displaying results in the Contributor Insights widgets. After the rules are created, you can change the state of one or more Contributor Insights rules from CloudWatch console by choosing Insights from the navigation pane, and then choosing Contributor Insights.

After the stack reaches CREATE_COMPLETE status, go to the Outputs tab and choose the FirewallDashboardURI link to open the new dashboard in the CloudWatch Dashboards console. It might take a few minutes for the Logs Insights and Contributor Insights widgets to start displaying data. For more details about each widget, see the README. If you don’t have log events matching the query parameters in the widgets, some widgets might not show data points.

Troubleshooting

If you encounter issues during or after deployment, review the following:

  • Firewall logging is enabled and configured to use CloudWatch instead of Amazon Simple Storage Service (Amazon S3) or Amazon Kinesis.
  • Both firewall flow and alert logging are enabled, not just one.
  • Log group names are entered correctly; incorrect names will cause widgets to point to invalid data.
  • Correct subnets are selected. Incorrect choices can impact the PrivateLink metrics widgets.
  • Firewall name is entered correctly. An incorrect name can disrupt metrics widgets, dashboard, and Contributor Insights widget names and break the firewall link.

Cleaning up

You can delete the Network Firewall CloudWatch dashboard and all of the associated resources with a few clicks. Deleting the dashboard will not impact the routing and network traffic inspection performed by the firewall.

  1. Sign in to the CloudFormation console in the Region where you launched the stack and choose Stacks from the navigation pane.
  2. Select the Stack name you chose when launching the stack. For example, my-firewall-dashboard.
  3. Choose Delete.

Conclusion

We encourage you to see for yourself how this new dashboard can enhance your network security management. To get started with the AWS Network Firewall CloudWatch Dashboard, visit our GitHub repository for detailed instructions and the CloudFormation template. For a visual overview of the dashboard and its capabilities, check out our YouTube video.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Ajinkya Patil
Ajinkya Patil

Ajinkya is a Security Consultant at Amazon Professional Services, specializing in security consulting for AWS customers within the automotive industry since 2019. He has presented at AWS re:Inforce and contributed articles to the AWS Security blog and AWS Prescriptive Guidance. Beyond his professional commitments, he indulges in travel and photography.
Todd Pula
Todd Pula

Todd is a Sr. Cloud Support Engineer at AWS and a service experience owner for AWS Network Firewall. He is an accomplished problem solver helping customers build and troubleshoot complex cloud networking and security solutions. He has a Master’s degree in Information Technology and is a Cisco Certified Internetwork Expert (CCIE).
Amish Shah
Amish Shah

Amish is a seasoned product leader with over 15 years of experience in developing innovative and scalable solutions for networking, security, and cloud use cases. He currently leads the AWS Network Firewall service, where he helps to develop security solutions that protect AWS workloads. Outside of work, Amish enjoys playing cricket and soccer, loves to travel, and has recently picked up a hobby of collecting niche fragrances.

How to automate rule management for AWS Network Firewall

Post Syndicated from Ajinkya Patil original https://aws.amazon.com/blogs/security/how-to-automate-rule-management-for-aws-network-firewall/

AWS Network Firewall is a stateful managed network firewall and intrusion detection and prevention service designed for the Amazon Virtual Private Cloud (Amazon VPC). This post concentrates on automating rule updates in a central Network Firewall by using distributed firewall configurations. If you’re new to Network Firewall or seeking a technical background on rule management, see AWS Network Firewall – New Managed Firewall Service in VPC.

Network Firewall offers three deployment models: Distributed, centralized, and combined. Many customers opt for a centralized model to reduce costs. In this model, customers allocate the responsibility for managing the rulesets to the owners of the VPC infrastructure (spoke accounts) being protected, thereby shifting accountability and providing flexibility to the spoke accounts. Managing rulesets in a shared firewall policy generated from distributed input configurations of protected VPCs (spoke accounts) is challenging without proper input validation, state-management, and request throttling controls.

In this post, we show you how to automate firewall rule management within the central firewall using distributed firewall configurations spread across multiple AWS accounts. The anfw-automate solution provides input-validation, state-management, and throttling controls, reducing the update time for firewall rule changes from minutes to seconds. Additionally, the solution reduces operational costs, including rule management overhead while integrating seamlessly with the existing continuous integration and continuous delivery (CI/CD) processes.

Prerequisites

For this walkthrough, the following prerequisites must be met:

  • Basic knowledge of networking concepts such as routing and Classless Inter-Domain Routing (CIDR) range allocations.
  • Basic knowledge of YAML and JSON configuration formats, definitions, and schema.
  • Basic knowledge of Suricata Rule Format and Network Firewall rule management.
  • Basic knowledge of CDK deployment.
  • AWS Identity and Access Management (IAM) permissions to bootstrap the AWS accounts using AWS Cloud Development Kit (AWS CDK).
  • The firewall VPC in the central account must be reachable from a spoke account (see centralized deployment model). For this solution, you need two AWS accounts from the centralized deployment model:
    • The spoke account is the consumer account the defines firewall rules for the account and uses central firewall endpoints for traffic filtering. At least one spoke account is required to simulate the user workflow in validation phase.
    • The central account is an account that contains the firewall endpoints. This account is used by application and the Network Firewall.
  • StackSets deployment with service-managed permissions must be enabled in AWS Organizations (Activate trusted access with AWS Organizations). A delegated administrator account is required to deploy AWS CloudFormation stacks in any account in an organization. The CloudFormation StackSets in this account deploy the necessary CloudFormation stacks in the spoke accounts. If you don’t have a delegated administrator account, you must manually deploy the resources in the spoke account. Manual deployment isn’t recommended in production environments.
  • A resource account is the CI/CD account used to deploy necessary AWS CodePipeline stacks. The pipelines deploy relevant cross-account cross-AWS Region stacks to the preceding AWS accounts.
    • IAM permissions to deploy CDK stacks in the resource account.

Solution description

In Network Firewall, each firewall endpoint connects to one firewall policy, which defines network traffic monitoring and filtering behavior. The details of the behavior are defined in rule groups — a reusable set of rules — for inspecting and handling network traffic. The rules in the rule groups provide the details for packet inspection and specify the actions to take when a packet matches the inspection criteria. Network Firewall uses a Suricata rules engine to process all stateful rules. Currently, you can create Suricata compatible or basic rules (such as domain list) in Network Firewall. We use Suricata compatible rule strings within this post to maintain maximum compatibility with most use cases.

Figure 1 describes how the anfw-automate solution uses the distributed firewall rule configurations to simplify rule management for multiple teams. The rules are validated, transformed, and stored in the central AWS Network Firewall policy. This solution isolates the rule generation to the spoke AWS accounts, but still uses a shared firewall policy and a central ANFW for traffic filtering. This approach grants the AWS spoke account owners the flexibility to manage their own firewall rules while maintaining the accountability for their rules in the firewall policy. The solution enables the central security team to validate and override user defined firewall rules before pushing them to the production firewall policy. The security team operating the central firewall can also define additional rules that are applied to all spoke accounts, thereby enforcing organization-wide security policies. The firewall rules are then compiled and applied to Network Firewall in seconds, providing near real-time response in scenarios involving critical security incidents.

Figure 1: Workflow launched by uploading a configuration file to the configuration (config) bucket

Figure 1: Workflow launched by uploading a configuration file to the configuration (config) bucket

The Network Firewall firewall endpoints and anfw-automate solution are both deployed in the central account. The spoke accounts use the application for rule automation and the Network Firewall for traffic inspection.

As shown in Figure 1, each spoke account contains the following:

  1. An Amazon Simple Storage Service (Amazon S3) bucket to store multiple configuration files, one per Region. The rules defined in the configuration files are applicable to the VPC traffic in the spoke account. The configuration files must comply with the defined naming convention ($Region-config.yaml) and be validated to make sure that only one configuration file exists per Region per account. The S3 bucket has event notifications enabled that publish all changes to configuration files to a local default bus in Amazon EventBridge.
  2. EventBridge rules to monitor the default bus and forward relevant events to the custom event bus in the central account. The EventBridge rules specifically monitor VPCDelete events published by Amazon CloudTrail and S3 event notifications. When a VPC is deleted from the spoke account, the VPCDelete events lead to the removal of corresponding rules from the firewall policy. Additionally, all create, update, and delete events from Amazon S3 event notifications invoke corresponding actions on the firewall policy.
  3. Two AWS Identity and Access Manager (IAM) roles with keywords xaccount.lmb.rc and xaccount.lmb.re are assumed by RuleCollect and RuleExecute functions in the central account, respectively.
  4. A CloudWatch Logs log group to store event processing logs published by the central AWS Lambda application.

In the central account:

  1. EventBridge rules monitor the custom event bus and invoke a Lambda function called RuleCollect. A dead-letter queue is attached to the EventBridge rules to store events that failed to invoke the Lambda function.
  2. The RuleCollect function retrieves the config file from the spoke account by assuming a cross-account role. This role is deployed by the same stack that created the other spoke account resources. The Lambda function validates the request, transforms the request to the Suricata rule syntax, and publishes the rules to an Amazon Simple Queue Service (Amazon SQS) first-in-first-out (FIFO) queue. Input validation controls are paramount to make sure that users don’t abuse the functionality of the solution and bypass central governance controls. The Lambda function has input validation controls to verify the following:
    • The VPC ID in the configuration file exists in the configured Region and the same AWS account as the S3 bucket.
    • The Amazon S3 object version ID received in the event matches the latest version ID to mitigate race conditions.
    • Users don’t have only top-level domains (for example, .com, .de) in the rules.
    • The custom Suricata rules don’t have any as the destination IP address or domain.
    • The VPC identifier matches the required format, that is, a+(AWS Account ID)+(VPC ID without vpc- prefix) in custom rules. This is important to have unique rule variables in rule groups.
    • The rules don’t use security sensitive keywords such as sid, priority, or metadata. These keywords are reserved for firewall administrators and the Lambda application.
    • The configured VPC is attached to an AWS Transit Gateway.
    • Only pass rules exist in the rule configuration.
    • CIDR ranges for a VPC are mapped appropriately using IP set variables.

    The input validations make sure that rules defined by one spoke account don’t impact the rules from other spoke accounts. The validations applied to the firewall rules can be updated and managed as needed based on your requirements. The rules created must follow a strict format, and deviation from the preceding rules will lead to the rejection of the request.

  3. The Amazon SQS FIFO queue preserves the order of create, update, and delete operations run in the configuration bucket of the spoke account. These state-management controls maintain consistency between the firewall rules in the configuration file within the S3 bucket and the rules in the firewall policy. If the sequence of updates provided by the distributed configurations isn’t honored, the rules in a firewall policy might not match the expected ruleset.

    Rules not processed beyond the maxReceiveCount threshold are moved to a dead-letter SQS queue for troubleshooting.

  4. The Amazon SQS messages are subsequently consumed by another Lambda function called RuleExecute. Multiple changes to one configuration are batched together in one message. The RuleExecute function parses the messages and generates the required rule groups, IP set variables, and rules within the Network Firewall. Additionally, the Lambda function establishes a reserved rule group, which can be administered by the solution’s administrators and used to define global rules. The global rules, applicable to participating AWS accounts, can be managed in the data/defaultdeny.yaml file by the central security team.

    The RuleExecute function also implements throttling controls to make sure that rules are applied to the firewall policy without reaching the ThrottlingException from Network Firewall (see common errors). The function also implements back-off logic to handle this exception. This throttling effect can happen if there are too many requests issued to the Network Firewall API.

    The function makes cross-Region calls to Network Firewall based on the Region provided in the user configuration. There is no need to deploy the RuleExecute and RuleCollect Lambda functions in multiple Regions unless a use case warrants it.

Walkthrough

The following section guides you through the deployment of the rules management engine.

  • Deployment: Outlines the steps to deploy the solution into the target AWS accounts.
  • Validation: Describes the steps to validate the deployment and ensure the functionality of the solution.
  • Cleaning up: Provides instructions for cleaning up the deployment.

Deployment

In this phase, you deploy the application pipeline in the resource account. The pipeline is responsible for deploying multi-Region cross-account CDK stacks in both the central account and the delegated administrator account.

If you don’t have a functioning Network Firewall firewall using the centralized deployment model in the central account, see the README for instructions on deploying Amazon VPC and Network Firewall stacks before proceeding. You need to deploy the Network Firewall in centralized deployment in each Region and Availability Zone used by spoke account VPC infrastructure.

The application pipeline stack deploys three stacks in all configured Regions: LambdaStack and ServerlessStack in the central account and StacksetStack in the delegated administrator account. It’s recommended to deploy these stacks solely in the primary Region, given that the solution can effectively manage firewall policies across all supported Regions.

  • LambdaStack deploys the RuleCollect and RuleExecute Lambda functions, Amazon SQS FIFO queue, and SQS FIFO dead-letter queue.
  • ServerlessStack deploys EventBridge bus, EventBridge rules, and EventBridge Dead-letter queue.
  • StacksetStack deploys a service-managed stack set in the delegated administrator account. The stack set includes the deployment of IAM roles, EventBridge rules, an S3 Bucket, and a CloudWatch log group in the spoke account. If you’re manually deploying the CloudFormation template (templates/spoke-serverless-stack.yaml) in the spoke account, you have the option to disable this stack in the application configuration.
     
    Figure 2: CloudFormation stacks deployed by the application pipeline

    Figure 2: CloudFormation stacks deployed by the application pipeline

To prepare for bootstrapping

  1. Install and configure profiles for all AWS accounts using Amazon Command Line Interface (AWS CLI)
  2. Install the Cloud Development Kit (CDK)
  3. Install Git and clone the GitHub repo
  4. Install and enable Docker Desktop

To prepare for deployment

  1. Follow the README and cdk bootstrapping guide to bootstrap the resource account. Then, bootstrap the central account and delegated administrator account (optional if StacksetStack is deployed manually in the spoke account) to trust the resource account. The spoke accounts don’t need to be bootstrapped.
  2. Create a folder to be referred to as <STAGE>, where STAGE is the name of your deployment stage — for example, local, dev, int, and so on — in the conf folder of the cloned repository. The deployment stage is set as the STAGE parameter later and used in the AWS resource names.
  3. Create global.json in the <STAGE> folder. Follow the README to update the parameter values. A sample JSON file is provided in conf/sample folder.
  4. Run the following commands to configure the local environment:
    npm install
    export STAGE=<STAGE>
    export AWS_REGION=<AWS_Region_to_deploy_pipeline_stack>

To deploy the application pipeline stack

  1. Create a file named app.json in the <STAGE> folder and populate the parameters in accordance with the README section and defined schema.
  2. If you choose to manage the deployment of spoke account stacks using the delegated administrator account and have set the deploy_stacksets parameter to true, create a file named stackset.json in the <STAGE> folder. Follow the README section to align with the requirements of the defined schema.

    You can also deploy the spoke account stack manually for testing using the AWS CloudFormation template in templates/spoke-serverless-stack.yaml. This will create and configure the needed spoke account resources.

  3. Run the following commands to deploy the application pipeline stack:
    export STACKNAME=app && make deploy

    Figure 3: Example output of application pipeline deployment

    Figure 3: Example output of application pipeline deployment

After deploying the solution, each spoke account is required to configure stateful rules for every VPC in the configuration file and upload it to the S3 bucket. Each spoke account owner must verify the VPC’s connection to the firewall using the centralized deployment model. The configuration, presented in the YAML configuration language, might encompass multiple rule definitions. Each account must furnish one configuration file per VPC to establish accountability and non-repudiation.

Validation

Now that you’ve deployed the solution, follow the next steps to verify that it’s completed as expected, and then test the application.

To validate deployment

  1. Sign in to the AWS Management Console using the resource account and go to CodePipeline.
  2. Verify the existence of a pipeline named cpp-app-<aws_ organization_scope>-<project_name>-<module_name>-<STAGE> in the configured Region.
  3. Verify that stages exist in each pipeline for all configured Regions.
  4. Confirm that all pipeline stages exist. The LambdaStack and ServerlessStack stages must exist in the cpp-app-<aws_organization_scope>-<project_name>-<module_name>-<STAGE> stack. The StacksetStack stage must exist if you set the deploy_stacksets parameter to true in global.json.

To validate the application

  1. Sign in and open the Amazon S3 console using the spoke account.
  2. Follow the schema defined in app/RuleCollect/schema.json and create a file with naming convention ${Region}-config.yaml. Note that the Region in the config file is the destination Region for the firewall rules. Verify that the file has valid VPC data and rules.
    Figure 4: Example configuration file for eu-west-1 Region

    Figure 4: Example configuration file for eu-west-1 Region

  3. Upload the newly created config file to the S3 bucket named anfw-allowlist-<AWS_REGION for application stack>-<Spoke Account ID>-<STAGE>.
  4. If the data in the config file is invalid, you will see ERROR and WARN logs in the CloudWatch log group named cw-<aws_organization_scope>-<project_name>-<module_name>-CustomerLog-<STAGE>.
  5. If all the data in the config file is valid, you will see INFO logs in the same CloudWatch log group.
    Figure 5: Example of logs generated by the anfw-automate in a spoke account

    Figure 5: Example of logs generated by the anfw-automate in a spoke account

  6. After the successful processing of the rules, sign in to the Network Firewall console using the central account.
  7. Navigate to the Network Firewall rule groups and search for a rule group with a randomly assigned numeric name. This rule group will contain your Suricata rules after the transformation process.
    Figure 6: Rules created in Network Firewall rule group based on the configuration file in Figure 4

    Figure 6: Rules created in Network Firewall rule group based on the configuration file in Figure 4

  8. Access the Network Firewall rule group identified by the suffix reserved. This rule group is designated for administrators and global rules. Confirm that the rules specified in app/data/defaultdeny.yaml have been transformed into Suricata rules and are correctly placed within this rule group.
  9. Instantiate an EC2 instance in the VPC specified in the configuration file and try to access both the destinations allowed in the file and any destination not listed. Note that requests to destinations not defined in the configuration file are blocked.

Cleaning up

To avoid incurring future charges, remove all stacks and instances used in this walkthrough.

  1. Sign in to both the central account and the delegated admin account. Manually delete the stacks in the Regions configured for the app parameter in global.json. Ensure that the stacks are deleted for all Regions specified for the app parameter. You can filter the stack names using the keyword <aws_organization_scope>-<project_name>-<module_name> as defined in global.json.
  2. After deleting the stacks, remove the pipeline stacks using the same command as during deployment, replacing cdk deploy with cdk destroy.
  3. Terminate or stop the EC2 instance used to test the application.

Conclusion

This solution simplifies network security by combining distributed ANFW firewall configurations in a centralized policy. Automated rule management can help reduce operational overhead, reduces firewall change request completion times from minutes to seconds, offloads security and operational mechanisms such as input validation, state-management, and request throttling, and enables central security teams to enforce global firewall rules without compromising on the flexibility of user-defined rulesets.

In addition to using this application through S3 bucket configuration management, you can integrate this tool with GitHub Actions into your CI/CD pipeline to upload the firewall rule configuration to an S3 bucket. By combining GitHub actions, you can automate configuration file updates with automated release pipeline checks, such as schema validation and manual approvals. This enables your team to maintain and change firewall rule definitions within your existing CI/CD processes and tools. You can go further by allowing access to the S3 bucket only through the CI/CD pipeline.

Finally, you can ingest the AWS Network Firewall logs into one of our partner solutions for security information and event management (SIEM), security monitoring, threat intelligence, and managed detection and response (MDR). You can launch automatic rule updates based on security events detected by these solutions, which can help reduce the response time for security events.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Ajinkya Patil

Ajinkya Patil

Ajinkya is a Security Consultant at Amazon Professional Services, specializing in security consulting for AWS customers within the automotive industry since 2019. He has presented at AWS re:Inforce and contributed articles to the AWS Security blog and AWS Prescriptive Guidance. Beyond his professional commitments, he indulges in travel and photography.

Stephan Traub

Stephan Traub

Stephan is a Security Consultant working for automotive customers at AWS Professional Services. He is a technology enthusiast and passionate about helping customers gain a high security bar in their cloud infrastructure. When Stephan isn’t working, he’s playing volleyball or traveling with his family around the world.