Tag Archives: announcements

Top analytics announcements of AWS re:Invent 2024

Post Syndicated from Sakti Mishra original https://aws.amazon.com/blogs/big-data/top-analytics-announcements-of-aws-reinvent-2024/

AWS re:Invent 2024, the flagship annual conference, took place December 2–6, 2024, in Las Vegas, bringing together thousands of cloud enthusiasts, innovators, and industry leaders from around the globe. This premier event showcased groundbreaking advancements, keynotes from AWS leadership, hands-on technical sessions, and exciting product launches.

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

In this post, we walk you through the top analytics announcements from re:Invent 2024 and explore how these innovations can help you unlock the full potential of your data.

Amazon SageMaker

Introducing the next generation of Amazon SageMaker

AWS announces the next generation of Amazon SageMaker, a unified platform for data, analytics, and AI. This launch brings together widely adopted AWS machine learning (ML) and analytics capabilities and provides an integrated experience for analytics and AI with unified access to data and built-in governance.

The next generation of SageMaker also introduces new capabilities, including Amazon SageMaker Unified Studio (preview), Amazon SageMaker Lakehouse, and Amazon SageMaker Data and AI Governance. Amazon SageMaker Unified Studio brings together functionality and tools from the range of standalone studios, query editors, and visual tools available today in Amazon EMR, AWS Glue, Amazon Redshift, Amazon Bedrock, and the existing Amazon SageMaker Studio. Amazon SageMaker Lakehouse provides an open data architecture that reduces data silos and unifies data across Amazon Simple Storage Service (Amazon S3) data lakes, Redshift data warehouses, and third-party and federated data sources. Amazon SageMaker Data and AI Governance, including Amazon SageMaker Catalog built on Amazon DataZone, empowers you to securely discover, govern, and collaborate on data and AI workflows.

Amazon DynamoDB zero-ETL integration with Amazon SageMaker Lakehouse

Amazon DynamoDB zero-ETL integration with SageMaker Lakehouse automates the extraction and loading of data from a DynamoDB table into SageMaker Lakehouse, an open and secure lakehouse. Using the no-code interface, you can maintain an up-to-date replica of your DynamoDB data in the data lake by quickly setting up your integration to handle the complete process of replicating data and updating records. This zero-ETL integration reduces the complexity and operational burden of data replication to let you focus on deriving insights from your data. You can create and manage integrations using the AWS Management Console, the AWS Command Line Interface (AWS CLI), or the SageMaker Lakehouse APIs.

Amazon S3 Tables

Amazon S3 Tables – Fully managed Apache Iceberg tables optimized for analytics workloads

Amazon S3 Tables deliver the first cloud object store with built-in Apache Iceberg support, and the most straightforward way to store tabular data at scale. S3 Tables are specifically optimized for analytics workloads, resulting in up to 3 times faster query throughput and up to 10 times higher transactions per second compared to self-managed tables. S3 Tables are designed to perform continual table maintenance to automatically optimize query efficiency and storage cost over time, even as your data lake scales and evolves. S3 Tables integration with the AWS Glue Data Catalog is in preview, allowing you to stream, query, and visualize data—including Amazon S3 Metadata tables—using AWS analytics services such as Amazon Data Firehose, Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon QuickSight.

Amazon S3 Metadata (Preview) – Easiest and fastest way to manage your metadata

Amazon S3 Metadata is the simplest and fastest way to help you instantly discover and understand your S3 data with automated, queried metadata that updates in near real time. S3 Metadata supports object metadata, which includes system-defined details like size and the source of the object, and custom metadata, which allows you to use tags to annotate your objects with information like product SKU, transaction ID, or content rating, for example.

S3 Metadata is designed to automatically capture metadata from objects as they are uploaded into a bucket, and to make that metadata queryable in a read-only table. These metadata tables are stored in S3 Tables, the new S3 storage offering optimized for tabular data. Additionally, S3 Metadata integrates with Amazon Bedrock, allowing for the annotation of AI-generated videos with metadata that specifies its AI origin, creation timestamp, and the specific model used for its generation.

Additional resources:

AWS Glue

Introducing AWS Glue 5.0

With AWS Glue 5.0, you get improved performance, enhanced security, support for SageMaker Unified Studio and SageMaker Lakehouse, and more. AWS Glue 5.0 enables you to develop, run, and scale your data integration workloads and get insights faster.

AWS Glue 5.0 upgrades the engines to Apache Spark 3.5.2, Python 3.11, and Java 17, with new performance and security improvements. It also updates open table format support to Apache Hudi 0.15.0, Apache Iceberg 1.6.1, and Delta Lake 3.2.0. AWS Glue 5.0 adds Spark native fine-grained access control with AWS Lake Formation so you can apply table-, column-, row-, and cell-level permissions on S3 data lakes. Finally, AWS Glue 5.0 adds support for SageMaker Lakehouse to unify all your data across S3 data lakes and Redshift data warehouses.

Amazon S3 Access Grants now integrate with AWS Glue

Amazon S3 Access Grants now integrate with AWS Glue for analytics, ML, and application development workloads in AWS. S3 Access Grants map identities from your identity provider (IdP), such as Entra ID and Okta or AWS Identity and Access Management (IAM) principals, to datasets stored in Amazon S3. This integration gives you the ability to manage Amazon S3 permissions for end-users running jobs with AWS Glue 5.0 or later, without the need to write and maintain bucket policies or individual IAM roles. When end-users in the appropriate user groups access Amazon S3 using AWS Glue ETL for Apache Spark, they will then automatically have the necessary permissions to read and write data.

AWS Glue Data catalog now automates generating statistics for new tables

The AWS Glue Data Catalog now automates generating statistics for new tables. These statistics are integrated with a cost-based optimizer (CBO) from Amazon Redshift and Athena, resulting in improved query performance and potential cost savings. Previously, creating statistics for Iceberg tables in the Data Catalog required you to continuously monitor and update configurations for your tables. Now, the Data Catalog lets you generate statistics automatically for new tables with one-time catalog configuration. Amazon Redshift and Athena use the updated statistics to optimize queries, using optimizations such as optimal join order or cost-based aggregation pushdown. The Data Catalog console provides you visibility into the updated statistics and statistics generation runs.

AWS expands data connectivity for Amazon SageMaker Lakehouse and AWS Glue

SageMaker Lakehouse announces unified data connectivity capabilities to streamline the creation, management, and usage of connections to data sources across databases, data lakes, and enterprise applications. SageMaker Lakehouse unified data connectivity provides a connection configuration template, support for standard authentication methods like basic authentication and OAuth 2.0, connection testing, metadata retrieval, and data preview. You can create SageMaker Lakehouse connections through SageMaker Unified Studio (preview), the AWS Glue console, or a custom-built application using APIs under AWS Glue.

With the ability to browse metadata, you can understand the structure and schema of the data source and identify relevant tables and fields. SageMaker Lakehouse unified connectivity is available where SageMaker Lakehouse or AWS Glue is available.

Announcing generative AI troubleshooting for Apache Spark in AWS Glue (Preview)

AWS Glue announces generative AI troubleshooting for Apache Spark, a new capability that helps data engineers and scientists quickly identify and resolve issues in their Spark jobs. Spark Troubleshooting uses ML and generative AI technologies to provide automated root cause analysis for Spark job issues, along with actionable recommendations to fix identified issues. With Spark troubleshooting, you can initiate automated analysis of failed jobs with a single click on the AWS Glue console. Powered by Amazon Bedrock, Spark troubleshooting reduces debugging time from days to minutes.

The generative AI troubleshooting for Apache Spark preview is available for jobs running on AWS Glue 4.0.

Amazon EMR

Introducing Advanced Scaling in Amazon EMR Managed Scaling

We are excited to announce Advanced Scaling, a new capability in Amazon EMR Managed Scaling that provides you increased flexibility to control the performance and resource utilization of your Amazon EMR on EC2 clusters. With Advanced Scaling, you can configure the desired resource utilization or performance levels for your cluster, and Amazon EMR Managed Scaling will use your intent to intelligently scale the cluster and optimize cluster compute resources.

Advanced Scaling is available with Amazon EMR release 7.0 and later and is available in all AWS Regions where Amazon EMR Managed Scaling is available.

Additional resources:

Amazon Athena

Amazon SageMaker Lakehouse integrated access controls now available in Amazon Athena federated queries

SageMaker now supports connectivity, discovery, querying, and enforcing fine-grained data access controls on federated sources when querying data with Athena. Athena is a query service that makes it simple to analyze your data lake and federated data sources such as Amazon Redshift, DynamoDB, or Snowflake using SQL without extract, transform, and load (ETL) scripts. Now, data workers can connect to and unify these data sources within SageMaker Lakehouse. Federated source metadata is unified in SageMaker Lakehouse, where you apply fine-grained policies in one place, helping to streamline analytics workflows and secure your data.

Amazon Managed Service for Apache Flink

Amazon Managed Service for Apache Flink now supports Amazon Managed Service for Prometheus as a destination

AWS announced support for a new Apache Flink connector for Amazon Managed Service for Prometheus. The new connector, contributed by AWS for the Flink open source project, adds Amazon Managed Service for Prometheus as a new destination for Flink. You can use the new connector to send processed data to an Amazon Managed Service for Prometheus destination starting with Flink version 1.19. With Amazon Managed Service for Apache Flink, you can transform and analyze data in real time. There are no servers and clusters to manage, and there is no compute and storage infrastructure to set up.

Amazon Managed Service for Apache Flink now delivers to Amazon SQS queues

AWS announced support for a new Flink connector for Amazon Simple Queue Service (Amazon SQS). The new connector, contributed by AWS for the Flink open source project, adds Amazon SQS as a new destination for Apache Flink. You can use the new connector to send processed data from Amazon Managed Service for Apache Flink to SQS messages with Flink, a popular framework and engine for processing and analyzing streaming data.

Amazon Managed Service for Apache Flink releases a new Amazon Kinesis Data Streams connector

Amazon Managed Service for Apache Flink now offers a new Flink connector for Amazon Kinesis Data Streams. This open source connector, contributed by AWS, supports Flink 2.0 and provides several enhancements. It enables in-order reads during stream scale-up or scale-down, supports Flink’s native watermarking, and improves observability through unified connector metrics. Additionally, the connector uses the AWS SDK for Java 2.x, which supports enhanced performance and security features, and native retry strategy. You can use the new connector to read data from a Kinesis data stream starting with Flink version 1.19.

Amazon Redshift

Amazon SageMaker Lakehouse and Amazon Redshift support for zero-ETL integrations from eight applications

SageMaker Lakehouse and Amazon Redshift now support zero-ETL integrations from applications, automating the extraction and loading of data from eight applications, including Salesforce, SAP, ServiceNow, and Zendesk. As an open, unified, and secure lakehouse for your analytics and AI initiatives, SageMaker Lakehouse enhances these integrations to streamline your data management processes. These zero-ETL integrations are fully managed by AWS and minimize the need to build ETL data pipelines. Optimize your data ingestion processes and focus instead on analysis and gaining insights.

Amazon Redshift multi-data warehouse writes through data sharing is now generally available

AWS announces the general availability of Amazon Redshift multi-data warehouse writes through data sharing. You can now start writing to Redshift databases from multiple Redshift data warehouses in just a few clicks. With Redshift multi-data warehouse writes through data sharing, you can keep ETL jobs more predictable by splitting workloads between multiple warehouses, helping you meet your workload performance requirements with less time and effort. Your data is immediately available across AWS accounts and Regions after it’s committed, enabling better collaboration across your organization.

Announcing Amazon Redshift Serverless with AI-driven scaling and optimization

Amazon Redshift Serverless introduces the next generation of AI-driven scaling and optimization in cloud data warehousing. Redshift Serverless uses AI techniques to automatically scale with workload changes across all key dimensions—such as data volume changes, number of concurrent users, and query complexity—to meet and maintain your price-performance targets. Amazon internal tests demonstrate that this optimization can provide you up to 10 times better price performance for variable workloads, without manual intervention.

Redshift Serverless with AI-driven scaling and optimization is available in all AWS Regions where Redshift Serverless is available.

Amazon Redshift now supports incremental refresh on Materialized Views (MVs) for data lake tables

Amazon Redshift now supports incremental refresh of materialized views on data lake tables. This capability helps you improve query performance for your data lake queries in a cost-effective and efficient manner. By enabling incremental refresh for materialized views, you can maintain up-to-date data in a more efficient and affordable way.

Support for incremental refresh for materialized views on data lake tables is now available in all commercial Regions. To get started and learn more, visit Materialized views on external data lake tables in Amazon Redshift Spectrum.

AWS announces Amazon Redshift integration with Amazon Bedrock for generative AI

AWS announces the integration of Amazon Redshift with Amazon Bedrock, a fully managed service offering high-performing foundation models (FMs) making it simpler and faster for you to build generative AI applications. This integration enables you to use large language models (LLMs) from simple SQL commands alongside your data in Amazon Redshift.

The Amazon Redshift integration with Amazon Bedrock is now generally available in all Regions where Amazon Bedrock and Amazon Redshift ML are supported. To get started, see Amazon Redshift ML integration with Amazon Bedrock.

Announcing general availability of auto-copy for Amazon Redshift

Amazon Redshift announces the general availability of auto-copy, which simplifies data ingestion from Amazon S3 into Amazon Redshift. This new feature enables you to set up continuous file ingestion from your S3 prefix and automatically load new files to tables in your Redshift data warehouse without the need for additional tools or custom solutions.

Amazon Redshift auto-copy from Amazon S3 is now generally available for both Redshift Serverless and Amazon Redshift RA3 Provisioned data warehouses in all AWS commercial Regions.

Amazon DataZone

Data Lineage is now generally available in Amazon DataZone and next generation of Amazon SageMaker

AWS announces general availability of Data Lineage in Amazon DataZone and the next generation of SageMaker, a capability that automatically captures lineage from AWS Glue and Amazon Redshift to visualize lineage events from source to consumption. Being OpenLineage compatible, this feature allows data producers to augment the automated lineage with lineage events captured from OpenLineage-enabled systems or through an API, to provide a comprehensive data movement view to data consumers. This feature automates lineage capture of schema and transformations of data assets and columns from AWS Glue, Amazon Redshift, and Spark executions in tools to maintain consistency and reduce errors. Additionally, the data lineage feature versions lineage with each event, enabling you to visualize lineage at any point in time or compare transformations across an asset’s or job’s history.

Amazon DataZone now enhances data access governance with enforced metadata rules

Amazon DataZone now supports enforced metadata rules for data access workflows, providing organizations with enhanced capabilities to strengthen governance and compliance with their organization needs. This new feature allows domain owners to define and enforce mandatory metadata requirements, making sure data consumers provide essential information when requesting access to data assets in Amazon DataZone. By streamlining metadata governance, this capability helps organizations meet compliance standards, maintain audit readiness, and simplify access workflows for greater efficiency and control.

Amazon DataZone expands data access with tools like Tableau, Power BI, and more

Amazon DataZone now supports authentication with the Athena JDBC driver, enabling data consumers to query their project’s subscribed data lake assets in Amazon DataZone using popular business intelligence (BI) and analytics tools such as Tableau, Domino, Power BI, Microsoft Excel, SQL Workbench, and more. Data analysts and scientists can seamlessly access and analyze governed data in Amazon DataZone using a standard JDBC connection with their preferred tools.

This feature is now available in all the AWS commercial Regions where Amazon DataZone is supported. Check out Expanding data analysis and visualization options: Amazon DataZone now integrates with Tableau, Power BI, and more and Connecting Amazon DataZone with external applications via JDBC connectivity to learn more about how to connect Amazon DataZone to external analytics tools via JDBC.

Amazon QuickSight

Announcing scenarios analysis capability of Amazon Q in QuickSight (preview)

A new scenario analysis capability of Amazon Q in QuickSight is now available in preview. This new capability provides an AI-assisted data analysis experience that helps you make better decisions, faster. Amazon Q in QuickSight simplifies in-depth analysis with step-by-step guidance, saving hours of manual data manipulation and unlocking data-driven decision-making across your organization. You can ask a question or state your goal in natural language and Amazon Q in QuickSight guides you through every step of advanced data analysis—suggesting analytical approaches, automatically analyzing data, surfacing relevant insights, and summarizing findings with suggested actions.

Amazon QuickSight now supports prompted reports and reader scheduling for pixel-perfect reports

We are enabling QuickSight readers to generate filtered views of pixel-perfect reports and create schedules to deliver reports through email. Readers can create up to five schedules per dashboard for themselves. Previously, only dashboard owners could create schedules and only on the default (author published) view of the dashboard. Now, if an author has added controls to the pixel-perfect report, schedules can be created or updated to respect selections on the filter control.

Prompted reports and reader scheduling are now available in all supported QuickSight Regions—see Amazon QuickSight endpoints and quotas for QuickSight Regional endpoints.

Amazon Q in QuickSight unifies insights from structured and unstructured data

Amazon Q in QuickSight provides you with unified insights from structured and unstructured data sources through integration with Amazon Q Business. With data stories in Amazon Q in QuickSight, you can upload documents, or connect to unstructured data sources from Amazon Q Business, to create richer narratives or presentations explaining your data with additional context. This integration enables organizations to harness insights from all their data without the need for manual collation, leading to more informed decision-making, time savings, and a significant competitive edge.

Amazon Q Business now provides insights from your databases and data warehouses (preview)

AWS announces the public preview of the integration between Amazon Q Business and QuickSight, delivering a transformative capability that unifies answers from structured data sources (databases, warehouses) and unstructured data (documents, wikis, emails) in a single application.

With the QuickSight integration, you can now link your structured sources to Amazon Q Business through the extensive set of data source connectors available in QuickSight. This integration unifies insights across knowledge sources, helping organizations make more informed decisions while reducing the time and complexity traditionally required to gather insights.

Amazon OpenSearch Service

Amazon OpenSearch Service zero-ETL integration with Amazon Security Lake

Amazon OpenSearch Service now offers a zero-ETL integration with Amazon Security Lake, enabling you to query and analyze security data in-place directly through OpenSearch. This integration allows you to efficiently explore voluminous data sources that were previously cost-prohibitive to analyze, helping you streamline security investigations and obtain comprehensive visibility of your security landscape.

Amazon OpenSearch Ingestion now supports writing security data to Amazon Security Lake

Amazon OpenSearch Ingestion now allows you to write data into Amazon Security Lake in real time, allowing you to ingest security data from both AWS and custom sources and uncover valuable insights into potential security issues in near real time. With this feature, you can now use OpenSearch Ingestion to ingest and transform security data from popular third-party sources like Palo Alto, CrowdStrike, and SentinelOne into OCSF format before writing the data into Amazon Security Lake. After the data is written to Amazon Security Lake, it is available in the AWS Glue Data Catalog and Lake Formation tables for the respective source.

AWS Clean Rooms

AWS Clean Rooms now supports multiple clouds and data sources

AWS Clean Rooms announces support for collaboration with datasets from multiple clouds and data sources. This launch allows companies and their partners to collaborate with data stored in Snowflake and Athena, without having to move or share their underlying data among collaborators.

Conclusion

re:Invent 2024 showcased how AWS continues to push the boundaries of data and analytics, delivering tools and services that empower organizations to derive faster, smarter, and more actionable insights. From advancements in data lakes, data warehouses, and streaming solutions to the integration of generative AI capabilities, these announcements are designed to transform the way businesses interact with their data.

As we look ahead, it’s clear that AWS is committed to helping organizations stay ahead in an increasingly data-driven world. Whether you’re modernizing your analytics stack or exploring new possibilities with AI and ML, the innovations from re:Invent 2024 provide the building blocks to unlock value from your data.

Stay tuned for more deep dives into these announcements, and don’t hesitate to explore how these tools can accelerate your journey toward data-driven success!


About the Authors

Sakti Mishra serves as Principal Data and AI Solutions Architect at AWS, where he helps customers modernize their data architecture and define end-to end-data strategies, including data security, accessibility, governance, and more. He is also the author of Simplify Big Data Analytics with Amazon EMR and AWS Certified Data Engineer Study Guide books. Outside of work, Sakti enjoys learning new technologies, watching movies, and visiting places with family. He can be reached via LinkedIn.

Navnit Shukla serves as an AWS Specialist Solutions Architect with a focus on analytics. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices. Notably, Navnit Shukla is the accomplished author of the book titled “Data Wrangling on AWS.” He can be reached via LinkedIn.

Support Canada’s CCCS PBHVA overlay compliance with the Landing Zone Accelerator on AWS

Post Syndicated from Naranjan Goklani original https://aws.amazon.com/blogs/security/support-canadas-cccs-pbhva-overlay-compliance-with-the-landing-zone-accelerator-on-aws/

Organizations seeking to adhere to the Canadian Centre for Cyber Security (CCCS) Protected B High Value Assets (PBHVA) overlay requirements can use the Landing Zone Accelerator (LZA) on AWS solution with the CCCS Medium configuration to accelerate their compliance journey. To further support customers, AWS recently collaborated with Coalfire to assess and verify the LZA solution’s ability to support CCCS PBHVA overlay controls.

By implementing the PBHVA control overlay over a CCCS Medium baseline, you can better protect your organization’s most critical assets from potential threats and vulnerabilities, providing continuity of essential government operations and safeguarding sensitive information.

Understanding CCCS PBHVA overlay requirements

The CCCS PBHVA overlay consists of 137 controls designed to protect high-value assets, including 69 new controls and 68 controls from CCCS Medium. These controls provide enhanced data protection, particularly for integrity and availability, and are based on NIST SP 800-53 Revision 5.

Key findings from the Coalfire assessment

Coalfire’s assessment found that the LZA on AWS solution significantly supports CCCS PBHVA overlay compliance requirements:

  • 71 percent of in-scope controls (97 of 137) are supported by the AWS contribution to compliance in the shared responsibility model
  • The solution uses over 35 AWS services to provide comprehensive security capabilities
  • Strong network segmentation is achieved through network account and network-boundary VPC design
  • Infrastructure-as-code (IaC) enables reliable build and deployment results

The 29 percent of controls not addressed by the LZA are on the customer side of the shared responsibility model. They are addressed in the customer’s application stack or as non-technical controls such as policies and procedures.

Key security capabilities

The LZA solution implements several critical security features:

Implementation considerations

While the LZA solution provides significant compliance support, organizations should note:

  • The solution alone does not guarantee compliance
  • Organizations must implement their own policies, standards, and procedures
  • A thorough understanding of the shared responsibility model is essential

The AWS Landing Zone Accelerator Verified Reference Architecture documentation is available for customer download in AWS Artifact. This resource can help organizations reduce the time and effort required to deploy an environment that aligns with CCCS PBHVA overlay requirements.

Conclusion

The Coalfire assessment confirms that the LZA on AWS solution provides effective support for CCCS PBHVA overlay compliance objectives. However, organizations should remember that compliance is an ongoing process that requires active management and cannot be achieved through technology alone.

For more information about implementing the Landing Zone Accelerator for CCCS PBHVA overlay requirements, contact your AWS account team or the AWS Public Sector team directly.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Naranjan Goklani
Naranjan Goklani

Naranjan is an Audit Lead for Canada based in Toronto. He has experience leading audits, attestations, certifications, and assessments across North America and Europe. Naranjan has more than 15 years of experience in risk management, security assurance, and performing technology audits. Naranjan previously worked in one of the Big 4 accounting firms and supported clients from the financial services, technology, retail, e-commerce, and utilities industries as part of the first and third line of defense.
Michael Davie
Michael Davie

Michael is the Canada lead for Amazon Web Services (AWS) Compliance and Security Assurance. He works with customers, regulators, and AWS teams to help raise the bar on secure cloud adoption and usage. Michael has more than 20 years of experience working in the defence, intelligence, and technology sectors in Canada, and is a licensed professional engineer.
James Kierstead
James Kierstead

James is a senior solutions architect at Amazon Web Services (AWS) based in Ottawa, Canada. He is passionate about helping Canada’s federal government use AWS to deliver services to Canadians.

Anthropic’s Claude 3.7 Sonnet hybrid reasoning model is now available in Amazon Bedrock

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/anthropics-claude-3-7-sonnet-the-first-hybrid-reasoning-model-is-now-available-in-amazon-bedrock/

Amazon Bedrock is expanding its foundation model (FM) offerings as the generative AI field evolves. Today, we’re excited to announce the availability of Anthropic’s Claude 3.7 Sonnet foundation model in Amazon Bedrock. As Anthropic’s most intelligent model to date, Claude 3.7 Sonnet stands out as their first hybrid reasoning model capable of producing quick responses or extended thinking, meaning it can work through difficult problems using careful, step-by-step reasoning. Additionally, today we are adding Claude 3.7 Sonnet to the list of models used by Amazon Q Developer. Amazon Q is built on Bedrock, and with Amazon Q you can use the most appropriate model for a specific task such as Claude 3.7 Sonnet, for more advanced coding workflows that enable developers to accelerate building across the entire software development lifecycle.

Key highlights of Claude 3.7 Sonnet
Here are several notable features and capabilities of Claude 3.7 Sonnet in Amazon Bedrock.

The first Claude model with hybrid reasoning – Claude 3.7 Sonnet takes a different approach to how models think. Instead of using separate models—one for quick answers and another for solving complex problems—Claude 3.7 Sonnet integrates reasoning as a core capability within a single model. This combination is more similar to how the human brains works. After all, we use the same brain whether we’re answering a simple question or solving a difficult puzzle.

The model has two modes—standard and extended thinking mode—which can be toggled in Amazon Bedrock. In standard mode, Claude 3.7 Sonnet is an improved version of Claude 3.5 Sonnet. In extended thinking mode, Claude 3.7 Sonnet takes additional time to analyze problems in detail, plan solutions, and consider multiple perspectives before providing a response, allowing it to make further gains in performance. You can control speed and cost by choosing when to use reasoning capabilities. Extended thinking tokens count towards the context window and are billed as output tokens.

Anthropic’s most powerful model for coding – Claude 3.7 Sonnet is state-of-the art for coding, excelling in understanding context and creative problem solving, and according to Anthropic, achieves an industry-leading 70.3% for standard mode on SWE-bench Verified. Claude 3.7 Sonnet also performs better than Claude 3.5 Sonnet across the majority of benchmarks. These enhanced capabilities make Claude 3.7 Sonnet ideal for powering AI agents and complex workflows.

Claude 3.7 Sonnet benchmarks

Source: https://www.anthropic.com/news/claude-3-7-sonnet

Over 15x longer output capacity than its predecessor – Compared to Claude 3.5 Sonnet, this model offers significantly expanded output length. This enhanced capacity is particularly useful when you explicitly request more detail, ask for multiple examples, or request additional context or background information. To achieve long outputs, try asking for a detailed outline (for writing use cases, you can specify outline detail down to the paragraph level and include word count targets). Then, ask for the response to index its paragraphs to the outline and reiterate the word counts. Claude 3.7 Sonnet supports outputs up to 128K tokens long (up to 64K as generally available and up to 128K as a beta).

Adjustable reasoning budget – You can control the budget for thinking when you use Claude 3.7 Sonnet in Amazon Bedrock. This flexibility helps you weigh the trade-offs between speed, cost, and performance. By allocating more tokens to reasoning for complex problems or limiting tokens for faster responses, you can optimize performance for your specific use case.

Claude 3.7 Sonnet in action
As for any new model, I have to request access in the Amazon Bedrock console. In the navigation pane, I choose Model access under Bedrock configurations. Then, I choose Modify model access to request access for Claude 3.7 Sonnet.

Model access in Amazon Bedrock

To try Claude 3.7 Sonnet, I choose Chat / Text under Playgrounds in the navigation pane. Then I choose Select model and choose Anthropic under the Categories and Claude 3.7 Sonnet under the Models. To enable the extended thinking mode, I toggle Model reasoning under Configurations. I type the following prompt, and choose Run:

You're the manager of a small restaurant facing these challenges:

Three staff members called in sick for tonight's dinner service
You're expecting a full house (80 seats)
There's a large party of 20 coming at 7 PM
Your main chef is available but two kitchen helpers are among those who called in sick
You have 2 regular servers and 1 trainee available
How would you:

Reorganize the available staff to handle the situation
Prioritize tasks and service
Determine if you need to make any adjustments to reservations
Handle the large party while maintaining service quality
Minimize negative impact on customer experience
Explain your reasoning for each decision and discuss potential trade-offs


Chat / Text playground

Here’s the result with an animated image showing the reasoning process of the model.

Testing Claude 3.7 Sonnet reasoning

To test image-to-text vision capabilities, I upload an image of a detailed architectural site plan created using Amazon Bedrock. I receive a detailed analysis and reasoned insights of this site plan.

Claude 3.7 Sonnet can also be accessed through AWS SDK by using Amazon Bedrock API. To learn more about Claude 3.7 Sonnet’s features and capabilities, visit the Anthropic’s Claude in Amazon Bedrock product detail page.

Get started with Claude 3.7 Sonnet today
Claude 3.7 Sonnet’s enhanced capabilities can benefit multiple industry use cases. Businesses can create advanced AI assistants and agents that interact directly with customers. In fields such as healthcare, it can assist in medical imaging analysis and research summarization, and financial services can benefit from its abilities to solve complex financial modeling problems. For developers, it serves as a coding companion that can review code, explain technical concepts, and suggest improvements across different languages.

Anthropic’s Claude 3.7 Sonnet is available today in the US East (N. Virginia), US East (Ohio), and US West (Oregon) Regions. Check the full Region list for future updates.

Claude 3.7 Sonnet is priced competitively and matches the price of Claude 3.5 Sonnet. For pricing details, refer to the Amazon Bedrock pricing page.

To get started with Claude 3.7 Sonnet in Amazon Bedrock, visit the Amazon Bedrock console and Amazon Bedrock documentation.

— Esra

AWS Weekly Roundup: Cloud Club Captain Applications, Formula 1®, Amazon Nova Prompt Engineering, and more (Feb 24, 2025)

Post Syndicated from Elizabeth Fuentes original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-cloud-club-captain-applications-formula-1-amazon-nova-prompt-engineering-and-more-feb-24-2025/

AWS Developer Day 2025, held on February 20th, showcased how to integrate responsible generative AI into development workflows. The event featured keynotes from AWS leaders including Srini Iragavarapu, Director Generative AI Applications and Developer Experiences, Jeff Barr, Vice President of AWS Evangelism, David Nalley, Director Open Source Marketing of AWS, along with AWS Heroes and technical community members. Watch the full event recording on Developer Day 2025.

Cloud Club

Applications are now open through March 6th for the 2025 AWS Cloud Clubs Captains program. AWS Cloud Clubs are student-led groups for post-secondary and independent students, 18 years old and over. Find a club near you on our Meetup page.

Last week’s launches
Here are some launches that got my attention:

Amplify Hosting announces support for IAM roles for server-side rendered (SSR) applications  AWS Amplify Hosting now supports AWS Identity and Access Management (IAM) roles for SSR applications, enabling secure access to AWS services without managing credentials manually. Learn more in the IAM Compute Roles for Server-Side Rendering with AWS Amplify Hosting blog.

AWS WAF enhances Data Protection and logging experience  AWS WAF expands its Data Protection capabilities allowing sensitive data in logs to be replaced with cryptographic hashes (e.g. ‘ade099751d2ea9f3393f0f’) or a predefined static string (‘REDACTED’) before logs are sent to WAF Sample Logs, Amazon Security Lake, Amazon CloudWatch, or other logging destinations.

Announcing AWS DMS Serverless comprehensive premigration assessments AWS Database Migration Service Serverless (AWS DMS Serverless) now supports premigration assessments for replications to identify potential issues before database migrations begin. The tool analyzes source and target databases, providing recommendations for optimal DMS settings and best practices.

Amazon ECS increases the CPU limit for ECS tasks to 192 vCPUs – Amazon Elastic Container Service (Amazon ECS) now supports CPU limits of up to 192 vCPU for ECS tasks deployed on Amazon Elastic Compute Cloud (Amazon EC2) instances, an increase from the previous 10 vCPU limit. This enhancement allows customers to more effectively manage resource allocation on larger Amazon EC2 instances.

AWS Network Firewall introduces automated domain lists and insightsAWS Network Firewall now provides automated domain lists and insights by analyzing 30 days of HTTP/S traffic. This helps create and maintain allow-list policies more efficiently, at no extra cost.

AWS announces Backup Payment Methods for invoices AWS now enables you to set up backup payment methods that automatically activate if primary payment fails. This helps prevent service interruptions and reduces manual intervention for invoice payments.

Get updated with all the announcements of AWS announcements on the What’s New with AWS? page.

Other AWS news
Here are additional noteworthy items:

AWS Partner Network: Essential training resources for ISV partners To help scale solutions effectively, AWS provides essential training resources for Software Vendors (ISVs) partners in four key areas: AWS Marketplace fundamentals, Foundational Technical Review (FTR), APN Customer Engagement (ACE) program and co-selling, and Partner funding opportunities.

How Formula 1® uses generative AI to accelerate race-day issue resolution Formula 1® (F1) uses Amazon Bedrock to speed up race-day issue resolution, reducing troubleshooting time from weeks to minutes through a chatbot that analyzes root causes and suggests fixes.

How Formula 1® uses generative AI to accelerate race-day issue resolution

Reducing hallucinations in LLM agents with a verified semantic cache using Amazon Bedrock Knowledge Bases This blog introduces a solution using Amazon Bedrock Knowledge Bases and Amazon Bedrock Agents to reduce Large language models (LLMs) hallucinations by implementing a verified semantic cache that checks queries against curated answers before generating new responses, improving accuracy and response times.

Reducing hallucinations in LLM agents with a verified semantic cache using Amazon Bedrock Knowledge Bases

Orchestrate an intelligent document processing workflow using tools in Amazon Bedrock This blog demonstrates an intelligent document processing workflow using Amazon Bedrock tools that combines Anthropic’s Claude 3 Haiku for orchestration and Anthropic’s Claude 3.5 Sonnet (v2) for analysis to handle structured, semi-structured, and unstructured healthcare documents efficiently.

From community.aws
Here are my personal favorites posts from community.aws:

Tracing Amazon Bedrock Agents Learn how to track and analyze Amazon Bedrock Agents workflows using AWS X-Ray for better observability, by Randy D.

Testing Amazon ECS Network Resilience with AWS FISThis article demonstrates how to test network resilience in Amazon ECS using AWS FIS with guidance from Amazon Q Developer, by Sunil Govindankutty

Stop Using Default Arguments in AWS Lambda Functions Discover why your AWS Lambda costs might be spiralling out of control due to a common Python programming practice, by Stuart Clark.

Amazon Nova Prompt Engineering on AWS: A Field Guide by Brooke A field guide for using Amazon Nova models, covering prompt engineering patterns and best practices on AWS, by Brooke Jamieson.

Amazon Nova Prompt Engineering on AWS: A Field Guide by Brooke

Creating Deployment Configurations for EKS with Amazon Q Amazon Q Developer helps create EKS deployments by providing templates and best practices for Kubernetes configs, by Ricardo Tasso.

Processing WhatsApp Multimedia with Amazon Bedrock Agents: Images, Video, and DocumentsI invite you to read my latest blog, which explains how to create a WhatsApp AI assistant using Amazon Bedrock and Amazon Nova models to process multimedia content such as images, videos, documents, and audio.

Processing WhatsApp Multimedia with Amazon Bedrock Agents: Images, Video, and Documents

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

AWS GenAI Lofts – GenAI Lofts offer collaborative spaces and immersive experiences for startups and developers. You can join in-person GenAI Loft San Francisco events such as Hands-on with Agentic Graph RAG Workshop (February 25), Unstructured Data Meetup SF (February 26 – 27) and AI Tinkerers – San Francisco – February 2025 Demos + Science Fair (February 27 – 28). GenAI Loft Berlin has events and workshops on February 24 to March 7 that you can’t miss!

AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Milan, Italy (April 2), Bay Area – Security Edition (April 4), Timișoara, Romania (April 10), and Prague, Czeh Republic (April 29).

AWS Innovate: Generative AI + Data – Join a free online conference focusing on generative AI and data innovations. Available in multiple geographic regions: APJC and EMEA (March 6), North America (March 13), Greater China Region (March 14), and Latin America (April 8).

AWS Summits – Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Paris (April 9), Amsterdam (April 16), London (April 30), and Poland (May 5).

AWS re:Inforce – AWS re:Inforce (June 16–18) in Philadelphia, PA our annual learning event devoted to all things AWS cloud security. Registration opens in March, and be ready to join more than 5,000 security builders and leaders.

Create your AWS Builder ID and reserve your alias. Builder ID is a universal login credential that gives you access–beyond the AWS Management Console–to AWS tools and resources, including over 600 free training courses, community features, and developer tools such as Amazon Q Developer.

You can browse all upcoming in-person and virtual events.

That’s all for this week. Stay tuned for next week’s Weekly Roundup!

Eli

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

From log analysis to rule creation: How AWS Network Firewall automates domain-based security for outbound traffic

Post Syndicated from Mary Kay Sondecker original https://aws.amazon.com/blogs/security/from-log-analysis-to-rule-creation-how-aws-network-firewall-automates-domain-based-security-for-outbound-traffic/

When it comes to controlling incoming (ingress) and outgoing (egress) network traffic, organizations typically focus heavily on inbound traffic controls—carefully restricting what traffic can enter their network perimeter. However, this approach addresses only inbound security challenges. Modern applications rely heavily on third-party code through operating systems, libraries, and packages. This dependency can create potential security vulnerabilities. If these components are compromised, affected workloads might attempt to connect to unauthorized command and control servers or send sensitive data to unauthorized destinations on the internet.

This is why implementing strong outbound traffic controls—particularly through domain-based allowlisting—has become a critical security best practice. Rather than allowing unrestricted outbound access or maintaining an ever-growing denylist of low-reputation domains, many organizations are shifting to domain-based allowlisting. This approach restricts outbound communications to explicitly trusted domains, reduces potential risk surfaces, and helps to protect against both known and unknown threats. However, manually identifying and maintaining these allowlists has traditionally been a complex and time-consuming process.

AWS Network Firewall automated domain lists improve visibility into network traffic patterns and simplify outbound traffic control management. This feature provides analytics for HTTP and HTTPS network traffic, helping organizations understand domain usage patterns. It also automates firewall log analysis to create rules based on your network traffic. By combining increased visibility with automation, this feature enhances your security awareness and helps to improve the effectiveness of your firewall rules.

In this blog post, we’ll guide you through the implementation of the AWS Network Firewall automated domain list feature, providing a detailed overview, step-by-step instructions, and best practices to optimize your network security.

Overview of automated domain lists and traffic insights

Domain-based security allows you to control network traffic based on the domain names that your applications and users are trying to access. This approach offers a more intuitive and flexible way to create firewall rules, focusing on the destinations your network is trying to reach rather than just IP addresses. However, effectively configuring and managing firewall rules remains challenging for some customers, especially in large environments where connected devices, applications, and traffic patterns are continuously growing and changing. Organizations might struggle to keep up with these changes, leading to outdated or ineffective firewall rules and policies that are either too permissive, exposing the network to risks, or too restrictive, blocking legitimate traffic.

Let’s explore how automated domain lists address these challenges through various use cases and benefits:

Preventive and detective security controls

  1. Domain control through allowlisting – Establishing domain allowlists aligns with the security principle of least privilege for network traffic. A least-privilege model adjusts the scope of what a workload can do across the network, from infinite and undefined to scoped-down and well-defined, enabling better insight into potentially risky behaviors. By limiting outbound connections to only approved domains, organizations can more effectively control and monitor workload communications.
  2. Rule audit and compliance – Domain allowlisting makes it clear which domains are allowed, supporting alignment with standards like the Payment Card Industry Data Security Standard (PCI DSS), Health Insurance Portability and Accountability Act (HIPAA), Cybersecurity Maturity Model Certification (CMMC), and General Data Protection Regulation (GDPR).
  3. Preventive controls enable detection – Preventive controls also act as detective controls, establishing a baseline for normal domain access patterns. With a domain allowlist in place, security teams can better detect workloads that show signs of unauthorized activity.
  4. Incident response support – Domain reporting provides the latest list of workload domains accessed, enabling quick identification of potentially malicious domains during security incidents. This information helps teams prioritize which workloads may need immediate attention.

Operational value

  1. Initial firewall setup and management – Automated allowlisting involves analyzing existing traffic patterns and recommending domain-based rules, which simplifies the process of establishing baseline firewall rules. This helps organizations quickly deploy effective security policies, potentially reducing the time and expertise needed for initial firewall configuration and ongoing management.
  2. Application modernization – Allowlisting supports adjusting firewall rules to accommodate rapidly changing traffic patterns in microservices and containerized environments, helping security to keep pace with evolving architectures.
  3. Cross-environment consistency – Allowlisting enables consistent firewall rule creation and management across multi-cloud and hybrid environments, regardless of where applications or data reside.

How the automated domain list feature works

Automated domain lists work by analyzing your HTTP and HTTPS traffic, generating reports on frequently accessed domains, and providing a convenient way to create rules based on actual network traffic patterns. To begin using automated domain lists in AWS Network Firewall, sign in to the AWS Management Console, access the Network Firewall service, and either work with an existing firewall or create a new one. Then follow the rest of the steps in this post.

Step 1: Enable traffic analysis mode to capture HTTP and HTTPS traffic domain logs

After you’ve selected a firewall, in the left navigation pane, choose Configure advanced settings. Select the Enable traffic analysis mode checkbox to enable it, as shown in Figure 1. Network Firewall uses this logging mode to collect data on observed domains for HTTP and HTTPS traffic to create domain reports.

Figure 1: Enabling traffic analysis mode for a firewall

Figure 1: Enabling traffic analysis mode for a firewall

To stop collecting data on frequently accessed domains in your network traffic, clear the checkbox to disable traffic analysis mode, as shown in Figure 2. Note that if you disable traffic analysis mode, you won’t be able to generate domain reports.

Figure 2: Disabling traffic analysis mode

Figure 2: Disabling traffic analysis mode

Once traffic analysis mode is enabled, you’re ready to generate a domain report based on observed network traffic. Next, you can go to the Monitoring and observability tab and choose Create report.

Figure 3: Traffic analysis mode enabled: Now you’re ready to generate domain-based reports

Figure 3: Traffic analysis mode enabled: Now you’re ready to generate domain-based reports

Step 2: Create a domain report

The domain report summarizes the HTTP and HTTPS traffic observed by your firewall for up to 30 days (or for the duration since firewall activation if less than 30 days). Select the checkbox next to each traffic analysis type you want to include in the report—HTTP, HTTPS, or both.

Important: Use your monthly domain report to examine 30 days of traffic behavior. Each report type (HTTP, HTTPS) is available once every 30 days at no additional cost.

Figure 4: Create a domain report that includes traffic analysis types HTTP, HTTPS, or both

Figure 4: Create a domain report that includes traffic analysis types HTTP, HTTPS, or both

To see the status of your domain report, go to the Reports section in the console for your specific firewall. When the report is ready, you can review the report directly in the console or download it, as shown in Figure 5.

Figure 5: The list of domain reports in the Reports section of the console for your specific firewall

Figure 5: The list of domain reports in the Reports section of the console for your specific firewall

Step 3: Review the report details

The report details include the traffic type (HTTP or HTTPS) and the observation period (start and end dates). By default, the report covers the last 30 days, or the entire period since traffic analysis was enabled if that is less than 30 days. The report also shows these details:

  • The Domain list shows domains that are a fully qualified domain name (FQDN) observed in the network traffic, such as aws.com or subdomain.aws.com.
  • The Access attempt count refers to the overall count of connection requests to the domain, including both successful and failed attempts.
  • The Unique sources field shows the number of distinct source IP addresses connected to the domain, indicating its popularity. For example, if one workload connects to aws.com, then count = 1; if 1000 workloads connect to aws.com, then count = 1,000.
  • The First accessed field shows when the domain was first seen in your traffic, while Last accessed shows when it was most recently seen. This includes both successful and failed attempts to access the domain.
  • The Protocol field indicates how the domain was observed—through either HTTP or HTTPS traffic (in other words, HTTP headers or a TLS handshake).

An example report is shown in Figure 6.

Figure 6: Example domain report details: 30-day analysis

Figure 6: Example domain report details: 30-day analysis

Step 4: (Optional) Create a domain list rule group

You can copy the list of observed domains from the report to a stateful domain list rule group and update your firewall policy. To do so, in the Report details section, choose Create domain list group to use the firewall policy wizard to create or update your firewall rules. The selected domains are automatically copied to a domain list rule group, as shown in Figure 7. For detailed instructions, see the AWS Network Firewall documentation.

Figure 7: Option to copy over the observed domain lists and create a domain list rule group using the firewall policy wizard

Figure 7: Option to copy over the observed domain lists and create a domain list rule group using the firewall policy wizard

Best practices for implementing domain allowlists

When you implement domain allowlisting, consider the following guidelines for operational success. We recommend that you also consult your own internal compliance and security policies.

  1. Start with a strategy of generous allowlisting:
    • Begin with broader and more generous allowlist rules rather than a more refined list, initially, to reduce the risk of accidently blocking legitimate domains.
    • Focus on getting to a Default Deny policy so that you can benefit from its risk surface reduction.
    • Create flexible rules for trusted domains, including second-level domains and top-level domains, such as allowing access to subdomains under your registered second-level domain. Or allow access to second-level domains under top-level domains that your organization trusts—for example, .mil, .gov, or .edu.
    • Use custom Suricata rules with regex capabilities to handle complex traffic efficiently. See Examples of stateful rules for Network Firewall.
    • Remember that even a broad allowlist provides better security than having no allowlist at all.
  2. Make iterative improvements:
    • After you establish an initial generous allowlist and Default Deny rules, evaluate the rules to determine which areas you might want to start narrowing down further. Use alert rules before pass rules in order to log the specific domains a pass rule might be allowing access to.
    • Adjust logging levels based on domain trust levels and monitoring requirements.
    • Review and update rules based on operational insights and changing requirements.
    • Take a pragmatic and iterative approach to rule refinement rather than attempting to make the ruleset very strict.
  3. Set up robust logging:
  4. Additional considerations:
    • After you enable traffic analysis mode, the automated domain lists feature provides visibility into your network traffic, reporting on observed connections. Although it doesn’t distinguish between allowed and blocked traffic, the domain list report can help you identify the most critical domains to include in your firewall rules.
    • The domain traffic data used to generate the list of domain recommendations is available for up to the last 30 days after traffic analysis has been enabled. This allows you to focus on the most relevant and recent network activity when optimizing your firewall policies.
    • Data collection for automated domain lists is opt-in and performed independently of the firewall policy and logging configuration. Enabling the feature doesn’t impact the performance of the firewall itself.

Conclusion

With AWS Network Firewall automated domain lists, you can simplify your firewall management process, create more effective rules based on actual traffic patterns, and maintain a strong security posture with less manual effort. This feature helps you address common challenges such as keeping up with rapidly changing application landscapes, managing security across complex environments, and adhering to compliance requirements. To learn more about Network Firewall and its features, see the product page and service documentation.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Network Firewall re:Post forum or contact AWS Support.
 

Mary Kay Sondecker
Mary Kay Sondecker

Mary Kay is a Senior Product Manager at AWS, focused on AWS Network Firewall. With over two decades of experience in the technology industry, she is passionate about helping customers easily implement effective, scalable cloud solutions to drive better business outcomes.
Jesse Lepich
Jesse Lepich

Jesse is a Senior Security Solutions Architect at AWS based in Lake St. Louis, Missouri, focused on helping customers implement native AWS security services. Outside of cloud security, his interests include relaxing with family, barefoot waterskiing, snowboarding and snow skiing, surfing, boating, and mountain climbing.
Michael Leighty
Michael Leighty

Michael is a Senior Security Solutions Architect at AWS, based in Atlanta. He specializes in helping customers design and implement effective network security controls, drawing from extensive experience at leading network security vendors. At AWS, he works closely with service teams to drive continuous improvement in security services based on customer needs and feedback.
Jason Goode
Jason Goode

Jason is a Senior Security GTM Content Specialist at AWS, where he develops content strategies that bridge technical concepts with practical business solutions. Based in Austin, Texas, he leverages his creative background and expertise to help organizations understand and use native AWS security services.

Announcing CDK Garbage Collection

Post Syndicated from Kaizen Conroy original https://aws.amazon.com/blogs/devops/announcing-cdk-garbage-collection/

The AWS Cloud Development Kit (CDK) is an open source framework that enables developers to define cloud infrastructure using a familiar programming language. Additionally, CDK provides higher level abstractions (Constructs), which reduce the complexity required to define and integrate AWS services together when building on AWS. CDK also provides core functionality like CDK Assets, which gives users the ability to bundle application assets into their CDK applications. These assets can be local files (main.py), directories (python_app/), or Docker images (Dockerfile). CDK Assets are stored in an Amazon Simple Storage Service (Amazon S3) Bucket or Amazon Elastic Container Registry (Amazon ECR) Repository that is created during CDK bootstrapping.

For CDK developers that leverage assets at scale, they may notice over time that the bootstrapped bucket or repository accumulated old or unused data. If users wanted to clean this data on their own, CDK didn’t provide a clear way of determining which data is safe to delete. To solve this problem, we are excited to announce the preview launch of CDK Garbage Collection, a new feature of the CDK that automatically deletes old assets in your bootstrapped Amazon S3 Bucket and Amazon ECR Repository, saving users time and money. This feature is available starting in AWS CDK version 2.165.0.

We expect CDK Garbage Collection to help AWS CDK customers save on storage costs associated with using the product while not affecting how customers use CDK.

Quickstart

CDK Garbage Collection is exposed as a CDK CLI command named gc. To use CDK Garbage Collection in its default configuration, run the following command on a terminal in your CDK application.

cdk gc --unstable=gc

The --unstable flag is meant to acknowledge that CDK Garbage Collection is in preview mode. This indicates that the scope and API of the feature might still change, but otherwise the feature is generally production ready and fully supported.

Walkthrough

CDK Garbage Collection works at the environment level, so it will attempt to delete isolated assets in the AWS account / region that you call it in. For the purposes of this walkthrough, you will be re-bootstrapping the environment with a custom qualifier so that you do not delete isolated assets before you are ready.

cdk bootstrap --qualifier=abcdef --toolkit-stack-name=CDKToolkitDemo

You now have a new bootstrap template under the name CDKToolkitDemo and bootstrap resources associated with it. Next, set up a CDK application with both Amazon S3 and Amazon ECR assets:

mkdir garbage-collection-demo && cd garbage-collection-demo
cdk init -l typescript app

Your next step is to replace the existing code In lib/garbage-collection-demo-stack.ts with the following CDK Stack:

import * as path from 'path';
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class GarbageCollectionDemoStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const fn1 = new lambda.Function(this, 'my-function-s3', {
    code: lambda.Code.fromAsset(path.join(__dirname, '..', 'lambda')),
    runtime: lambda.Runtime.NODEJS_LATEST,
    handler: 'index.handler',
    });

    const fn2 = new lambda.Function(this, 'my-function-ecr', {
    code: lambda.Code.fromAssetImage(path.join(__dirname, '..', 'docker')),
    runtime: lambda.Runtime.FROM_IMAGE,
    handler: lambda.Handler.FROM_IMAGE,
    });
  }
}

This creates two AWS Lambda functions, one which uses an Amazon S3 asset as its source code and one that uses an Amazon ECR image as its source code. You need to add the assets that are referenced to our CDK application. In lambda/index.js add a simple Lambda function:

exports.handler = async function(event) {
  const response = require('./response.json');
  return response;
};

And in docker/Dockerfile add a simple Docker image:

FROM public.ecr.aws/docker/library/alpine:latest Now you can run cdk deploy and get your initial CDK application set up in your AWS Account.
cdk deploy \
  --toolkit-stack-name=CDKToolkitDemo \
  --context='@aws-cdk/core:bootstrapQualifier=abcdef'

At this point you can check to make sure that assets have been correctly added into the bootstrapped Amazon S3 bucket and Amazon ECR repository:

cdk assets inside s3 bucket

Two objects exist in the bootstrapped Amazon S3 Bucket after the initial AWS CDK Deploy.

1 Image exists in the bootstrapped Amazon ECR Repository after the initial AWS CDK Deploy.

One image exists in the bootstrapped Amazon ECR Repository after the initial AWS CDK Deploy.

The output shows that you have the data you expect in both bootstrapped resources. The Amazon S3 Bucket also stores the json file of the AWS CloudFormation Template that was generated when you ran cdk deploy.

You can now simulate a typical CDK development cycle by updating both assets. Add a small change to the Amazon S3 asset that lives in lambda/index.js:

exports.handler = async function(event) {
  console.log('hello world');
  const response = require('./response.json');
  return response;
};

And do the same in docker/Dockerfile:

FROM public.ecr.aws/docker/library/alpine:latest
CMD echo 'Hello World'

You can now run cdk deploy again, and both assets should be re-uploaded under a new hash.

4 Objects exist in the bootstrapped Amazon S3 Bucket after the second AWS CDK Deploy.

Four objects exist in the bootstrapped Amazon S3 Bucket after the second AWS CDK Deploy.

2 Images exist in the bootstrapped Amazon ECR Repository after the second AWS CDK Deploy.

Two images exist in the bootstrapped Amazon ECR Repository after the second AWS CDK Deploy.

This output confirms that everything is as expected and the new assets have been added in. Because you are using new bootstrapped resources, you can still tell which resources are currently isolated and which are not. Right now, only the zipfile prefixed with 50f409b9 is referenced in AWS CloudFormation, and in Amazon ECR, only the image prefixed a5801b5b is referenced. That means that every other asset — 3 objects in Amazon S3 and 1 object in Amazon ECR — are isolated and can be deleted.

One item to note is the additional files in Amazon S3 that are not your local assets — these are AWS CloudFormation templates that are uploaded to Amazon S3 as an intermediary step before being sent to AWS CloudFormation. They are not needed after being copied over and are a perfect candidate for deletion via CDK Garbage Collection.

Here is where CDK Garbage Collection comes in. With the right parameters, you are able to clean up the isolated objects while not disturbing the assets that are actively in use.

cdk gc \
  --unstable=gc \
  --bootstrap-stack-name=CDKToolkitDemo \
  --rollback-buffer-days=0 \
  --created-buffer-days=0

Because you want to delete assets immediately, and not tag them for deletion later, set rollback-buffer-days to 0. You also want to delete assets that were just created, so be sure to set created-buffer-days to 0 as well. The default for created-buffer-days is 1.

 ⏳ Garbage Collecting environment aws://912331974472/us-east-1...
Found 3 objects to delete based off of the following criteria:
- objects have been isolated for > 0 days
- objects were created > 0 days ago

Delete this batch (yes/no/delete-all)? 

CDK Garbage Collection found three assets to be deleted from Amazon S3, which is to be expected. It prompts you to verify that you want to delete, which you do, so enter yes. You will then get this response:

[100.00%] 4 files scanned: 0 assets (0.00 MiB) tagged, 3 assets (0.02 MiB) deleted.

Followed by:

Found 1 image to delete based off of the following criteria:
- images have been isolated for > 0 days
- images were created > 0 days ago

Delete this batch (yes/no/delete-all)?

Once again, this is to be expected for Amazon ECR, so you enter yes again. You then get the response:

[100.00%] 2 files scanned: 0 assets (0.00 MiB) tagged, 1 assets (3.90 MiB) deleted.

At this point, CDK Garbage Collection is finished.

Details

CDK Garbage Collection exposes some parameters to help you customize the experience to your specific scenario. These options help you determine how aggressive you want your garbage collection to be.

  • rollback-buffer-days: this is the amount of days an asset has to be marked as isolated before it is eligible for deletion.
  • created-buffer-days: this is the amount of days an asset must live before it is eligible for deletion.

Rollback Buffer Days should be considered when you are not using cdk deploy and instead use a deployment method that operates on templates only, like a pipeline. If your pipeline can rollback without any involvement of the CDK CLI, this parameter will help ensure that assets are not prematurely deleted. When used, instead of deleting unused objects, cdk gc tags them with the current date. Subsequent runs of cdk gc will check this tag and delete the asset only after it has been tagged for longer than the specified buffer days.

Created Buffer Days should be considered if you want to be extra safe about assets that have been recently uploaded. When used, cdk gc filters out any assets that have not persisted that number of days. Note that this may not include assets that have been shared across multiple CDK Apps CDK reuses assets that are identical, and its possible that a recent deploy of a CDK App references an asset that was uploaded earlier.

For example, if you want to ensure that only assets that are over a month old and have been isolated for a week are deleted, you can specify:

cdk gc --unstable --rollback-buffer-days=7 --created-buffer-days=30.
Decision flow diagram of an asset as it gets audited for garbage collection.

Decision flow diagram of an asset as it gets audited for garbage collection.

Limitations of CDK Garbage Collection

During CDK Garbage Collection, we collect all stack templates to see what assets are in use. If garbage collection runs between the asset upload and stack deployment, there is a chance that it does not pick up the latest stack deployment, but it does pick up the latest asset. In this scenario, CDK Garbage Collection may delete those assets.

We recommend not deploying stacks while running CDK Garbage Collection. If that is unavoidable, setting --created-buffer-days will help as garbage collection will avoid deleting assets that are recently created. Finally, if you do experience a failed deployment, the mitigation is to redeploy, as the asset upload step will be able to re-upload the missing asset. In practice, this race condition is only for a specific edge case and unlikely to happen. However, we are working on a new method of storing CDK Assets to reduce the risk of this race condition. That work is being tracked in this issue.

Conclusion

CDK Garbage Collection helps users manage the lifecycle of unused CDK Assets in their AWS account. As users continue to scale with the CDK, tools like CDK Garbage Collection will play a crucial role in maintaining clean, efficient, and cost-effective cloud environments. We encourage CDK users to explore this feature, provide feedback, and incorporate it into their workflows to optimize their AWS resource management.

Update on Support for Amazon Chime

Post Syndicated from Beth Styles original https://aws.amazon.com/blogs/messaging-and-targeting/update-on-support-for-amazon-chime/

Amazon Chime is a communications service that lets you meet, chat, and place business calls inside and outside your organization, all using a single application.

After careful consideration, we have decided to end support for the Amazon Chime service, including Business Calling features, effective February 20, 2026. Amazon Chime will no longer accept new customers beginning February 19, 2025. Existing customers can continue to use Amazon Chime features, including Business Calling, scheduling and hosting meetings, adding and managing users, and other capabilities supported through the Amazon Chime administration console. To qualify as an existing customer, you must have created an Amazon Chime Team or Enterprise account prior to February 19, 2025. After February 20, 2026, you will no longer be able to host Amazon Chime meetings, manage users, or use any of the Business Calling features. If you cannot delete your data prior to February 20, 2026, please reach out to us via Amazon Support for help. Note: This does not impact the availability of the Amazon Chime SDK service.

We know that you may need support to transition to other collaboration solutions. Please review our documentation which provides instructions and best practices for Amazon Chime IT administrators and users to move to alternate collaboration solutions. This may include solutions provided by AWS, such as AWS Wickr, or from AWS partners, such as Zoom from Zoom Video Communications Inc., Webex from Cisco Systems, Inc., and Slack from Salesforce, Inc. Visit the AWS Marketplace for more information about our AWS partners’ solutions. If you need additional assistance transitioning, please contact AWS Support.

Organizations with at least one Amazon Chime Team or Enterprise account can continue to use Amazon Chime and Business Calling features until February 20, 2026, when support for the service will end. Features that will no longer be supported include scheduling and hosting meetings, adding and managing users, and other capabilities available using the Amazon Chime console.

Additional Resources

Amazon Redshift announces history mode for zero-ETL integrations to simplify historical data tracking and analysis

Post Syndicated from Raks Khare original https://aws.amazon.com/blogs/big-data/amazon-redshift-announces-history-mode-for-zero-etl-integrations-to-simplify-historical-data-tracking-and-analysis/

In the ever-evolving landscape of cloud computing and data management, AWS has consistently been at the forefront of innovation. One of the groundbreaking developments in recent years is zero-ETL integration, a set of fully managed integrations by AWS that minimizes the need to build extract, transform, and load (ETL) data pipelines. This post will explore brief history of zero-ETL, its importance for customers, and introduce an exciting new feature: history mode for Amazon Aurora PostgreSQL-Compatible Edition, Amazon Aurora MySQL-Compatible Edition, Amazon Relational Database Service (Amazon RDS) for MySQL, and Amazon DynamoDB zero-ETL integration with Amazon Redshift.

A brief history of zero-ETL integrations

The concept of zero-ETL integrations emerged as a response to the growing complexities and inefficiencies in traditional ETL processes. Traditional ETL processes are time-consuming and complex to develop, maintain, and scale. Although not all use cases can be replaced with zero-ETL, it simplifies the replication and allows you to apply transformation post-replication. This eliminates the need for additional ETL technology between the source database and Amazon Redshift. We at AWS recognized the need for a more streamlined approach to data integration, particularly between operational databases and the cloud data warehouses. The journey of zero-ETL began in late 2022 when we introduced the feature for Aurora MySQL with Amazon Redshift. This feature marked a pivotal moment in streamlining complex data workflows, enabling near real-time data replication and analysis while eliminating the need for ETL processes.

Building on the success of our first zero-ETL integration, we’ve made continuous strides in this space by working backward from our customers’ needs and launching features like data filtering, auto and incremental refresh of materialized views, refresh interval, and more. Furthermore, we increased the breadth of sources to include Aurora PostgreSQL, DynamoDB, and Amazon RDS for MySQL to Amazon Redshift integrations, solidifying our commitment to making it seamless for you to run analytics on your data. The introduction of zero-ETL was not just a technological advancement; it represented a paradigm shift in how organizations could approach their data strategies. By removing the need for intermediate data processing steps, we opened up new possibilities for near real-time analytics and decision-making.

Introducing history mode: A new frontier in data analysis

Zero-ETL has already simplified the data integration, and we’re excited to further enhance the capabilities by announcing a new feature that takes it a step further: history mode with Amazon Redshift. Using history mode with zero-ETL integrations, you can streamline your historical data analysis by maintaining full change data capture (CDC) from the source in Amazon Redshift. History mode enables you to unlock the full potential of your data by seamlessly capturing and retaining historical versions of records across your zero-ETL data sources. You can perform advanced historical analysis, build look back reports, perform trend analysis, and create slowly changing dimensions (SCD) Type 2 tables on Amazon Redshift. This allows you to consolidate your core analytical assets and derive insights across multiple applications, gaining cost savings and operational efficiencies. History mode enables organizations to comply with regulatory requirements for maintaining historical records, facilitating comprehensive data governance and informed decision-making.

Zero-ETL integrations provide a current view of records in near real time, meaning only the latest changes from source databases are retained on Amazon Redshift. With history mode, Amazon Redshift introduces a revolutionary approach to historical data analysis. You can now configure your zero-ETL integrations to track every version of your records in source tables directly in Amazon Redshift, along with the source timestamp with every record version indicating when each record was inserted, modified, or deleted. Because data changes are tracked and retained by Amazon Redshift, this can help you meet your compliance requirements without having to maintain duplicate copies in data sources. In addition, you don’t have to maintain and manage partitioned tables to keep older data intact as separate partitions to version records, and maintain historical data in source databases.

In a data warehouse, the most common dimensional modeling techniques is a star schema, where there is a fact table at the center surrounded by a number of associated dimension tables. A dimension is a structure that categorizes facts and measures in order to enable users to answer business questions. To illustrate an example, in a typical sales domain, customer, time, or product are dimensions and sales transactions is a fact. An SCD is a data warehousing concept that contains relatively static data that can change slowly over a period of time. There are three major types of SCDs maintained in data warehousing: Type 1 (no history), Type 2 (full history), and Type 3 (limited history). CDC is a characteristic of a database that provides an ability to identify the data that changed between two database loads, so that an action can be performed on the changed data.

In this post, we demonstrate how to enable history mode for tables in a zero-ETL integration and capture the full historical data changes as SCD2.

Solution overview

In this use case, we explore how a fictional nationwide retail chain, AnyCompany, uses AWS services to gain valuable insights into their customer base. With multiple locations across the country, AnyCompany aims to enhance their understanding of customer behavior and improve their marketing strategies through two key initiatives:

  • Customer migration analysis – AnyCompany seeks to track and analyze customer relocation patterns, focusing on how geographical moves impact purchasing behavior. By monitoring these changes, the company can adapt its inventory, services, and local marketing efforts to better serve customers in their new locations.
  • Marketing campaign effectiveness – The retailer wants to evaluate the impact of targeted marketing campaigns based on customer demographics at the time of campaign execution. This analysis can help AnyCompany refine its marketing strategies, optimize resource allocation, and improve overall campaign performance.

By closely tracking changes in customer profiles for both geographic movement and marketing responsiveness, AnyCompany is positioning itself to make more informed, data-driven decisions.

In this demonstration, we begin by loading a sample dataset into the source table, customer, in Aurora PostgreSQL-Compatible. To maintain historical records, we enable history mode on the customer table, which automatically tracks changes in Amazon Redshift.

When history mode is turned on, the following columns are automatically added to the target table, customer, in Amazon Redshift to keep track of changes in the source.

Column name Data type Description
_record_is_active Boolean Indicates if a record in the target is currently active in the source. True indicates the record is active.
_record_create_time Timestamp Starting time (UTC) when the source record is active.
_record_delete_time Timestamp Ending time (UTC) when the source record is updated or deleted.

Next, we create a dimension table, customer_dim, in Amazon Redshift with an additional surrogate key column to show an example of creating an SCD table. To optimize query performance for different queries, some of which might be analyzing active or inactive records only while other queries might be analyzing data as of a certain date, we defined the sort key consisting of _record_is_active, _record_create_time, and _record_delete_time attributes in the customer_dim table.

The following figure provides the schema of the source table in Aurora PostgreSQL-Compatible, and the target table and target customer dimension table in Amazon Redshift.
schema

To streamline the data population process, we developed a stored procedure named SP_Customer_Type2_SCD(). This procedure is designed to populate incremental data into the customer_dim table from the replicated customer table. It handles various data changes, including updates, inserts, and deletes in the source table and implementing an SCD2 approach.

Prerequisites

Before you get started, complete the following steps:

  1. Configure your Aurora DB cluster and your Redshift data warehouse with the required parameters and permissions. For instructions, refer to Getting started with Aurora zero-ETL integrations with Amazon Redshift.
  2. Create an Aurora zero-ETL integration with Amazon Redshift.
  3. From an Amazon Elastic Compute Cloud (Amazon EC2) terminal or using AWS CloudShell, SSH into the Aurora PostgreSQL cluster and run the following commands to install psql:
sudo dnf install postgresql15
psql --version
  1. Load the sample source data:
    • Download the TPC-DS sample dataset for the customer table onto the machine running psql.
    • From the EC2 terminal, run the following command to connect to the Aurora PostgreSQL DB using the default super user postgres:
      psql -h <RDS Write Instance Endpoint> -p 5432 -U postgres

    • Run the following SQL command to create the database zetl:
      create database zetl template template1;

    • Change the connection to the newly created database:
      \c zetl

    • Create the customer table (the following example creates it in the public schema):
      CREATE TABLE customer(
          c_customer_id char(16) NOT NULL PRIMARY KEY,
          c_salutation char(10),
          c_first_name char(20),
          c_last_name char(30),
          c_preferred_cust_flag char(1),
          c_birth_day int4,
          c_birth_month int4,
          c_birth_year int4,
          c_birth_country varchar(20),
          c_login char(13),
          c_email_address char(50),
          ca_street_number char(10),
          ca_street_name varchar(60),
          ca_street_type char(15),
          ca_suite_number char(10),
          ca_city varchar(60),
          ca_county varchar(30),
          ca_state char(2),
          ca_zip char(10),
          ca_country varchar(20),
          ca_gmt_offset numeric(5, 2),
          ca_location_type char(20)
      );

    • Run the following command to load customer data from the downloaded dataset after changing the highlighted location of the dataset to your directory path:
      \copy customer from '/home/ec2-user/customer_sample_data.dat' WITH DELIMITER '|' CSV;

    • Run the following query to validate the successful creation of the table and loading of sample data:
      SELECT table_catalog, table_schema, table_name, n_live_tup AS row_count
      FROM information_schema.tables JOIN g_stat_user_tables ON table_name = relname
      WHERE table_type = 'BASE TABLE'
      ORDER BY row_count DESC;

The SQL output should be as follows:

table_catalog | table_schema | table_name | row_count
---------------+--------------+------------+-----------
zetl          | public       | customer   |   1200585
(1 row)

Create a target database in Amazon Redshift

To replicate data from your source into Amazon Redshift, you must create a target database from your integration in Amazon Redshift. For this post, we have already created a source database called zetl in Aurora PostgreSQL-Compatible as part of the prerequisites. Complete the following steps to create the target database:

  1. On the Amazon Redshift console, choose Query editor v2 in the navigation pane.
  2. Run the following commands to create a database called postgres in Amazon Redshift using the zero-ETL integration_id with history mode turned on.
-- Amazon Redshift SQL commands to create database
SELECT integration_id FROM svv_integration; -- copy this result, use in the next sql
CREATE DATABASE "postgres" FROM INTEGRATION '<result from above>' DATABASE "zetl" SET HISTORY_MODE = TRUE;

History mode turned on at the time of target database creation on Amazon Redshift will enable history mode for existing and new tables created in the future.

  1. Run the following query to validate the successful replication of the initial data from the source into Amazon Redshift:
select is_history_mode, table_name, table_state, * from svv_integration_table_state;

The table customer should show table_state as Synced with is_history_mode as true.
histmode-true

Enable history mode for existing zero-ETL integrations

History mode can be enabled for your existing zero-ETL integrations using either the Amazon Redshift console or SQL commands. Based on your use case, you can turn on history mode at the database, schema, or table level. To use the Amazon Redshift console, complete the following steps:

  1. On the Amazon Redshift console, choose Zero-ETL integrations in the navigation pane.
  2. Choose your desired integration.
  3. Choose Manage history mode.
    zelt-integratin

On this page, you can either enable or disable history mode for all tables or a subset of tables.

  1. Select Manage history mode for individual tables and select Turn on for the history mode for the customer
  2. Choose Save changes.
    table-hist-mode
  3. To confirm changes, choose Table statistics and make sure History mode is On for the customer.
    table-stats
  4. Optionally, you can run the following SQL command in Amazon Redshift to enable history mode for the customer table:
ALTER DATABASE "postgres" INTEGRATION SET HISTORY_MODE = TRUE FOR TABLE public.customer;
  1. Optionally, you can enable history mode for all current and tables created in the future in the database:
ALTER DATABASE "postgres" INTEGRATION SET HISTORY_MODE = TRUE FOR ALL TABLES;
  1. Optionally, you can enable history mode for all current and tables created in the future in one or more schemas. The following query enables history mode for all current and tables created in the future for the public schema:
ALTER DATABASE "postgres" INTEGRATION SET HISTORY_MODE = TRUE FOR ALL TABLES IN SCHEMA public;
  1. Run the following query to validate if the customer table has been successfully changed to history mode with the is_history_mode column as true so that it can begin tracking every version (including updates and deletes) of all records changed in the source:
select is_history_mode, table_name, table_state, * from svv_integration_table_state;

Initially, the table will be in ResyncInitiated state before changing to Synced.
table-synced

  1. Run the following query in the zetl database of Aurora PostgreSQL-Compatible to modify a source record and observe the behavior of history mode in the Amazon Redshift target:
UPDATE customer
SET
    ca_suite_number = 'Suite 100',
    ca_street_number = '500',
    ca_street_name = 'Main',
    ca_street_type = 'St.',
    ca_city = 'New York',
    ca_county = 'Manhattan',
    ca_state = 'NY',
    ca_zip = '10001'
WHERE c_customer_id = 'AAAAAAAAAAAKNAAA';
  1. Now run the following query in the postgres database of Amazon Redshift to see all versions of the same record:
SELECT   
    c_customer_id,
    ca_street_number,
    ca_street_name,
    ca_suite_number,
    ca_city,
    ca_county,
    ca_state,
    ca_zip,
    _record_is_active,
    _record_create_time,
    _record_delete_time
FROM postgres.public.customer
WHERE c_customer_id = 'AAAAAAAAAAAKNAAA';

Zero-ETL integrations with history mode has inactivated the old record with the _record_is_active column value to false and created a new record with _record_is_active as true. You can also see how it maintains the _record_create_time and _record_delete_time column values for both records. The inactive record has a delete timestamp that matches the active record’s create timestamp.
table-history

Load incremental data in an SCD2 table

Complete the following steps to create an SCD2 table and implement an incremental data load process in a regular database of Amazon Redshift, in this case dev:

  1. Create an empty customer SDC2 table called customer_dim with SCD fields. The table also has DISTSTYLE AUTO and SORTKEY columns _record_is_active, _record_create_time, and _record_delete_time. When you define a sort key on a table, Amazon Redshift can skip reading entire blocks of data for that column. It can do so because it tracks the minimum and maximum column values stored on each block and can skip blocks that don’t apply to the predicate range.
CREATE TABLE dev.public.customer_dim (
    c_customer_sk bigint NOT NULL DEFAULT 0 ENCODE raw distkey,
    c_customer_id character varying(19) DEFAULT '' :: character varying ENCODE lzo,
    c_salutation character varying(12) ENCODE bytedict,
    c_first_name character varying(24) ENCODE lzo,
    c_last_name character varying(36) ENCODE lzo,
    c_preferred_cust_flag character varying(1) ENCODE lzo,
    c_birth_day integer ENCODE az64,
    c_birth_month integer ENCODE az64,
    c_birth_year integer ENCODE az64,
    c_birth_country character varying(24) ENCODE bytedict,
    c_login character varying(15) ENCODE lzo,
    c_email_address character varying(60) ENCODE lzo,
    ca_street_number character varying(12) ENCODE lzo,
    ca_street_name character varying(72) ENCODE lzo,
    ca_street_type character varying(18) ENCODE bytedict,
    ca_suite_number character varying(12) ENCODE bytedict,
    ca_city character varying(72) ENCODE lzo,
    ca_county character varying(36) ENCODE lzo,
    ca_state character varying(2) ENCODE lzo,
    ca_zip character varying(12) ENCODE lzo,
    ca_country character varying(24) ENCODE lzo,
    ca_gmt_offset numeric(5, 2) ENCODE az64,
    ca_location_type character varying(24) ENCODE bytedict,
    _record_is_active boolean ENCODE raw,
    _record_create_time timestamp without time zone ENCODE az64,
    _record_delete_time timestamp without time zone ENCODE az64,
    PRIMARY KEY (c_customer_sk)
) SORTKEY (
    _record_is_active,
    _record_create_time,
    _record_delete_time
);

Next, you create a stored procedure called SP_Customer_Type2_SCD() to populate incremental data in the customer_dim SCD2 table created in the preceding step. The stored procedure contains the following components:

    • First, it fetches the max _record_create_time and max _record_delete_time for each customer_id.
    • Then, it compares the output of the preceding step with the ongoing zero-ETL integration replicated table for records created after the max creation time in the dimension table or the record in the replicated table with _record_delete_time after the max _record_delete_time in the dimension table for each customer_id.
    • The output of the preceding step captures the changed data between the replicated customer table and target customer_dim dimension table. The interim data is staged to a customer_stg table, which is ready to be merged with the target table.
    • During the merge process, records that need to be deleted are marked with _record_delete_time and _record_is_active is set to false, whereas newly created records are inserted into the target table customer_dim with _record_is_active as true.
  1. Create the stored procedure with the following code:
CREATE OR REPLACE PROCEDURE public.sp_customer_type2_scd()
LANGUAGE plpgsql
AS $$
    BEGIN

    DROP TABLE IF EXISTS cust_latest;

    -- Create temp table with latest record timestamps
         CREATE TEMP TABLE cust_latest DISTKEY (c_customer_id) 
    AS
        SELECT
            c_customer_id,
            max(_record_create_time) AS _record_create_time,
            max(_record_delete_time) AS _record_delete_time
        FROM customer_dim 
        GROUP BY c_customer_id;
    
    DROP TABLE IF EXISTS customer_stg;

    -- Identify and stage changed records
    CREATE TEMP TABLE customer_stg 
    AS           
    SELECT
            ABS(fnv_hash(cust.c_customer_id)) as customer_sk,
            cust.*
            FROM
                postgres.public.customer cust
LEFT OUTER JOIN cust_latest ON cust.c_customer_id = cust_latest.c_customer_id
WHERE (cust._record_create_time > NVL(cust_latest._record_create_time, '1099-01-01 01:01:01') AND cust._record_is_active is true)
OR (cust._record_delete_time > NVL(cust_latest._record_delete_time, '1099-01-01 01:01:01') AND cust._record_is_active is false);

    -- Merge changes to customer dimension table
    MERGE INTO public.customer_dim 
    USING customer_stg stg 
    ON customer_dim.c_customer_id = stg.c_customer_id
        AND customer_dim._record_is_active = TRUE
        AND stg._record_is_active = false
    WHEN MATCHED THEN
        UPDATE
        SET
            _record_is_active = stg._record_is_active,
            _record_create_time = stg._record_create_time,
            _record_delete_time = stg._record_delete_time
    WHEN NOT MATCHED THEN
        INSERT
        VALUES
            (
                stg.customer_sk,
                stg.c_customer_id,
                stg.c_salutation,
                stg.c_first_name,
                stg.c_last_name,
                stg.c_preferred_cust_flag,
                stg.c_birth_day,
                 	     stg.c_birth_month,
                stg.c_birth_year,
                stg.c_birth_country,
                stg.c_login,
                stg.c_email_address,
                stg.ca_street_number,
                stg.ca_street_name,
                stg.ca_street_type,
                stg.ca_suite_number,
                stg.ca_city,
                stg.ca_county,
                stg.ca_state,
                stg.ca_zip,
                stg.ca_country,
                stg.ca_gmt_offset,
                stg.ca_location_type,
                stg._record_is_active,
                stg._record_create_time,
                stg._record_delete_time
            );

    END;
    $$
  1. Run and schedule the stored procedure to load the initial and ongoing incremental data into the customer_dim SCD2 table:
CALL SP_Customer_Type2_SCD();
  1. Validate the data in the customer_dim table for the same customer with a changed address:
SELECT
    c_customer_id,
    ca_street_number,
    ca_street_name,
    ca_suite_number,
    ca_city,
    ca_county,
    ca_state,
    ca_zip,
    _record_is_active,
    _record_create_time,
    _record_delete_time
FROM customer_dim
WHERE c_customer_id = 'AAAAAAAAAAAKNAAA';

dim-history

You have successfully implemented an incremental load strategy for the customer SCD2 table. Going forward, all changes to customer will be tracked and maintained in this customer dimension table by running the stored procedure. This enables you to analyze customer data at a desired point in time for varying use cases, for example, performing customer migration analysis and seeing how geographical moves impact purchasing behavior, or marketing campaign effectiveness to analyze the impact of targeted marketing campaigns on customer demographics at the time of campaign execution.

Industry use cases for history mode

The following are other industry use cases enabled by history mode between operational data stores and Amazon Redshift:

  • Financial auditing or regulatory compliance – Track changes in financial records over time to support compliance and audit requirements. History mode allows auditors to reconstruct the state of financial data at any point in time, which is crucial for investigations and regulatory reporting.
  • Customer journey analysis – Understand how customer data evolves to gain insights into behavior patterns and preferences. Marketers can analyze how customer profiles change over time, informing personalization strategies and lifetime value calculations.
  • Supply chain optimization – Analyze historical inventory and order data to identify trends and optimize stock levels. Supply chain managers can review how demand patterns have shifted over time, improving forecasting accuracy.
  • HR analytics – Track employee data changes over time for better workforce planning and performance analysis. HR professionals can analyze career progression, salary changes, and skill development trends across the organization.
  • Machine learning model auditing – Data scientists can use historical data to train models, compare predictions vs. actuals to improve accuracy, and help explain model behavior and identify potential biases over time.
  • Hospitality and airline industry use cases – For example:
    • Customer service – Access historical reservation data to swiftly address customer queries, enhancing service quality and customer satisfaction.
    • Crew scheduling – Track crew schedule changes to help comply with union contracts, maintaining positive labor relations and optimizing workforce management.
    • Data science applications – Use historical data to train models on multiple scenarios from different time periods. Compare predictions against actuals to improve model accuracy for key operations such as airport gate management, flight prioritization, and crew scheduling optimization.

Best practices

If your requirement is to separate active and inactive records, you can use _record_is_active as the first sort key. For other patterns where you want to analyze data as of a specific date in the past, irrespective of whether data is active or inactive, _record_create_time and _record_delete_time can be added as sort keys.

History mode retains record versions, which will increase the table size in Amazon Redshift and could impact query performance. Therefore, periodically perform DML deletes for outdated record versions (delete data beyond a certain timeframe if not needed for analysis). When executing these deletions, maintain data integrity by deleting across all related tables. Vacuuming also becomes necessary when you perform DML deletes on records whose versioning is no longer required. To improve auto vacuum delete efficiency, Amazon Redshift auto vacuum delete is more efficient when operating on bulk deletes. You can monitor vacuum progression using the SYS_VACUUM_HISTORY table.

Clean up

Complete the following steps to clean up your resources:

  1. Delete the Aurora PostgreSQL cluster.
  2. Delete the Redshift cluster.
  3. Delete the EC2 instance.

Conclusion

Zero-ETL integrations have already made significant strides in simplifying data integration and enabling near real-time analytics. With the addition of history mode, AWS continues to innovate, providing you with even more powerful tools to derive value from your data.

As businesses increasingly rely on data-driven decision-making, zero-ETL with history mode will be crucial in maintaining a competitive edge in the digital economy. These advancements not only streamline data processes but also open up new avenues for analysis and insight generation.

To learn more about zero-ETL integration with history mode, refer to Zero-ETL integrations and Limitations. Get started with zero-ETL on AWS by creating a free account today!


About the Authors

Raks KhareRaks Khare is a Senior Analytics Specialist Solutions Architect at AWS based out of Pennsylvania. He helps customers across varying industries and regions architect data analytics solutions at scale on the AWS platform. Outside of work, he likes exploring new travel and food destinations and spending quality time with his family.

Jyoti Aggarwal is a Product Management Lead for AWS zero-ETL. She leads the product and business strategy, including driving initiatives around performance, customer experience, and security. She brings along an expertise in cloud compute, data pipelines, analytics, artificial intelligence (AI), and data services including databases, data warehouses and data lakes.

Gopal Paliwal is a Principal Engineer for Amazon Redshift, leading the software development of ZeroETL initiatives for Amazon Redshift.

Harman Nagra is a Principal Solutions Architect at AWS, based in San Francisco. He works with global financial services organizations to design, develop, and optimize their workloads on AWS.

Sumanth Punyamurthula is a Senior Data and Analytics Architect at Amazon Web Services with more than 20 years of experience in leading large analytical initiatives, including analytics, data warehouse, data lakes, data governance, security, and cloud infrastructure across travel, hospitality, financial, and healthcare industries.

AWS Weekly Roundup: AWS Developer Day, Trust Center, Well-Architected for Enterprises, and more (Feb 17, 2025)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-aws-developer-day-trust-center-well-architected-for-enterprises-and-more-feb-17-2025/

Join us for the AWS Developer Day on February 20! This virtual event is designed to help developers and teams incorporate cutting-edge yet responsible generative AI across their development lifecycle to accelerate innovation.

In his keynote, Jeff Barr, Vice President of AWS Evangelism, shares his thoughts on the next generation of software development based on generative AI, the skills needed to thrive in this changing environment, and how he sees it evolving in the future.

Get a first look at exciting technical deep-dive and product updates about Amazon Q Developer, AWS Amplify, and GitLab Duo with Amazon Q. You get the chance to explore real-world use cases, live coding demos, interactive sessions, and community spotlight sessions with Christian Bonzelet (AWS Community Builder), Hazel Saenz (AWS Serverless Hero), Matt Lewis (AWS Data Hero), and Johannes Koch (AWS DevTools Hero). Please sign up for this event now!

Last week’s launches
Here are some launches that got my attention:

Updating AWS SDK defaults for AWS STS – As we shared upcoming changes to the AWS Security Token Service (AWS STS) global endpoint to improve the resiliency and performance of your applications, we’re updating two defaults of AWS Software Development Kits (AWS SDKs) and AWS Command Line Interfaces (AWS CLIs) on July 31st 2025 – the default AWS STS service to regional, and the default retry strategy to standard. We recommend that you test your application before the release to avoid an unexpected experience after updating.

Introducing the AWS Trust CenterChris Betz, CISO at Amazon Web Services (AWS), shared AWS Trust Center, a new online resource communicating how we approach securing your assets in the cloud. This resource is a window into our security practices, compliance programs, and data protection controls that demonstrates how we work to earn your trust every day.

AWS CloudTrail network activity events for VPC endpoint – This feature provides you with a powerful tool to enhance your security posture, detect potential threats, and gain deeper insights into your VPC network traffic. This feature addresses your critical needs for comprehensive visibility and control over your AWS environments.

AWS Verified Access support for non-HTTP resources – AWS Verified Access now extends beyond HTTP apps to provide VPN-less, secure access to non-HTTP resources like Amazon Relational Database Service (Amazon RDS) databases, enabling improved security and enhanced user experience for both web applications and database connections. To learn more, visit the Verified Access endpoints page and a video tutorial.

New subnet management of Network Load Balancer (NLB) – NLBs were previously restricted to only adding subnets in new Availability Zones, and they now support full subnet management, including removal of subnets, matching the capabilities of Application Load Balancer (ALB). This enhancement offers organizations greater control over their network architecture and brings consistency to AWS load balancing services.

Meta SAM 2.1 and Falcon 3 models in Amazon SageMaker JumpStart – You can use Meta’s Segment Anything Model (SAM) 2.1 with state-of-the-art video and image segmentation capabilities in a single model. You can also use the Falcon 3 family with five models ranging from 1 to 10 billion parameters, with a focus on enhancing science, math, and coding capabilities. To learn more, visit SageMaker JumpStart pretrained models and Getting started with Amazon SageMaker JumpStart.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS? page.

Other AWS news
Here are some additional news items that you might find interesting:

AWS Documentation updateGreg Wilson, a lead of AWS Documentation, SDK, and CLI teams shared an insightful blog post about the progress, challenges, and what’s next for technical documentation for 200+ AWS services. It includes AWS Decision Guides for choosing the right service for specific needs; optimizing documents for readability, such as doubled code samples; and improving usability, such as dark mode and auto-suggest with top global navigation controls. You can also learn about how we use generative AI to help create technical documents.

AWS Well-Architected for Enterprises – This is a new free digital course designed for technical professionals who architect, build, and operate AWS solutions at scale. This intermediate-level course will help you optimize your cloud architecture while aligning to your business goals. The course takes approximately 1 hour to complete and includes a knowledge check at the end to reinforce your learning.

Integrating AWS with .NET Aspire – The .NET team at AWS has been working on integrations for connecting your .NET applications to AWS resources. Learn about how to automatically deploy AWS application resources using Aspire.Hosting.AWS NuGet package for NET Aspire, an open source framework building cloud-ready applications.

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

AWS Innovate: Generative AI + Data – Join a free online conference focusing on generative AI and data innovations. Available in multiple geographic regions: APJC and EMEA (March 6), North America (March 13), Greater China Region (March 14), and Latin America (April 8).

AWS Summits – Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Paris (April 9), Amsterdam (April 16), London (April 30), and Poland (May 5).

AWS GenAI Lofts – GenAI Lofts offer collaborative spaces and immersive experiences for startups and developers. You can join in-person GenAI Loft San Francisco events such as Built on Amazon Bedrock demo nights (April 19), SageMaker Unified Studio Demo for Startups (April 21), and Hands-on with Agentic Graph RAG Workshop (April 25). GenAI Loft Berlin has its Opening Day on February 24 and goes to March 7.

AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Karachi, Pakistan (February 22), Milan, Italy (April 2), Bay Area – Security Edition (April 4), Timișoara, Romania (April 10), and Prague, Czeh Republic (April 29).

AWS re:Inforce – Mark your calendars for AWS re:Inforce (June 16–18) in Philadelphia, PA. AWS re:Inforce is a learning conference focused on AWS security solutions, cloud security, compliance, and identity. You can subscribe for event updates now!

You can browse all upcoming in-person and virtual events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

Channy

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Introducing the AWS Trust Center

Post Syndicated from Chris Betz original https://aws.amazon.com/blogs/security/introducing-the-aws-trust-center/

At Amazon Web Services (AWS), earning trust isn’t just a goal—it’s one of our core Leadership Principles that guide every decision we make. As CISO of AWS, I’ve seen firsthand how this commitment to earning trust shapes our culture, our services, and our daily interactions with customers. Customers choose AWS over other providers because they trust us to provide the most secure infrastructure and services, and to tell them exactly how data is being protected.

To make that information even easier to find, we’re launching the AWS Trust Center, a new online resource that shares how we approach securing your assets in the cloud. The AWS Trust Center is a window into our security practices, compliance programs, and data protection controls that demonstrates how we work to earn your trust every day.

Building on a foundation of trust

Security has been our top priority since day one. When we launched AWS in 2006, we designed our infrastructure as the most secure cloud computing environment available. We knew we couldn’t just offer the same level of security as existing on-premises infrastructure. To earn customer trust, we had to disrupt and exceed the exacting industry standards of the world’s most security-conscious organizations. We continue to constantly reinforce security in our everyday decision making. With the Trust Center, we’re making it easier for you to understand how we protect your workloads, safeguard your data, and help you meet your compliance goals.

The Trust Center reflects our belief that more easily accessible information builds and maintains trust. Whether you’re looking for information about our data center controls, checking compliance certifications, or reviewing our shared responsibility model, you’ll now find the security and compliance information you need in one central location.

A single source of truth for security and compliance

In the Trust Center, you’ll find information about our approach to security at every level—from our physical data centers to our cloud infrastructure and our portfolio of cloud services. We’re including documentation about our security services and tools, helping you to understand both how we secure the cloud and how we help you secure your workloads within it. You’ll also find information about our compliance programs, including the certifications and attestations we maintain globally. This is valuable for teams working in regulated industries who need to demonstrate compliance to auditors and regulators.

The Trust Center highlights information about our data protection and privacy practices. Customers can learn how we protect your data and how we manage encryption. Further, we understand that customers are concerned about who can access their data and under what circumstances. We’ve consolidated detailed information about our operator access controls, which are designed to use the principle of least privilege at their core. You’ll learn about our zero-access designs for key services like AWS Key Management Service (AWS KMS) and Amazon Elastic Compute Cloud (Amazon EC2), our use of forward access sessions (FAS) to cryptographically enforce customer authorization, and our global monitoring systems.

The Trust Center provides a central place to find information about service health and security events, so you have the information you need to maintain operational excellence. You can stay up-to-date on security bulletins, and check real-time service health status. If you need to report a security concern or conduct your own security assessment, we’ve made those processes even easier to find. Resources are organized for ease of access, with direct links to the agreements, documentation, and resources you need to make informed decisions about your cloud security posture.

Empowering customers to drive secure innovation

What excites me most about the Trust Center is how it removes barriers for our customers. With detailed security information, easier links to compliance documentation, and operational insights now at your fingertips, you can move faster and innovate with confidence.

As we continue to innovate and expand AWS services, we’re committed to enhancing the Trust Center with the latest security information. This is a living resource that will evolve alongside our cloud and services. Maintaining your trust isn’t just about what we’ve built today—it’s about demonstrating, through both our commitments and delivery, that we’re worthy of being your trusted security partner. That’s our commitment to you, and that’s what the AWS Trust Center represents.

We invite you to explore the AWS Trust Center today, and we look forward to continuing to earn your trust, every day.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Chris Betz
Chris Betz

Chris is CISO at AWS. He oversees security teams and leads the development and implementation of security policies with the aim of managing risk and aligning the company’s security posture with business objectives. Chris joined Amazon in August 2023 after holding CISO and security leadership roles at leading companies. He lives in Northern Virginia with his family.

AWS CloudTrail network activity events for VPC endpoints now generally available

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/aws-cloudtrail-network-activity-events-for-vpc-endpoints-now-generally-available/

Today, I’m happy to announce the general availability of network activity events for Amazon Virtual Private Cloud (Amazon VPC) endpoints in AWS CloudTrail. This feature helps you to record and monitor AWS API activity traversing your VPC endpoints, helping you strengthen your data perimeter and implement better detective controls.

Previously, it was hard to detect potential data exfiltration attempts and unauthorized access to the resources within your network through VPC endpoints. While VPC endpoint policies could be configured to prevent access from external accounts, there was no built-in mechanism to log denied actions or detect when external credentials were used at a VPC endpoint. This often required you to build custom solutions to inspect and analyze TLS traffic, which could be operationally costly and negate the benefits of encrypted communications.

With this new capability, you can now opt in to log all AWS API activity passing through your VPC endpoints. CloudTrail records these events as a new event type called network activity events, which capture both control plane and data plane actions passing through a VPC endpoint.

Network activity events in CloudTrail provide several key benefits:

  • Comprehensive visibility – Log all API activity traversing VPC endpoints, regardless of the AWS account initiating the action.
  • External credential detection – Identify when credentials from outside your organization are accessing your VPC endpoint.
  • Data exfiltration prevention – Detect and investigate potential unauthorized data movement attempts.
  • Enhanced security monitoring – Gain insights into all AWS API activity at your VPC endpoints without the need to decrypt TLS traffic.
  • Visibility for regulatory compliance – Improve your ability to meet regulatory requirements by tracking all API activity passing through.

Getting started with network activity events for VPC endpoint logging
To enable network activity events, I go to the AWS CloudTrail console and choose Trails in the navigation pane. I choose Create trail to create a new one. I enter a name in the Trail name field and choose an Amazon Simple Storage Service (Amazon S3) bucket to store the event logs. When I create a trail in CloudTrail, I can specify an existing Amazon S3 bucket or create a new bucket to store my trail’s event logs.

If you set Log file SSE-KMS encryption to Enabled, you have two options: Choose New to create a new AWS Key Management Service (AWS KMS) key or choose Existing to choose an existing KMS key. If you chose New, you need to type an alias in the AWS KMS alias field. CloudTrail encrypts your log files with this KMS key and adds the policy for you. The KMS key and Amazon S3 must be in the same AWS Region. For this example, I use an existing KMS key. I enter the alias in the AWS KMS alias field and leave the rest as default for this demo. I choose Next for the next step.

In the Choose log events step, I choose Network activity events under Events. I choose the event source from the list of AWS services, such as cloudtrail.amazonaws.com, ec2.amazonaws.com, kms.amazonaws.com, s3.amazonaws.com, and secretsmanager.amazonaws.com. I add two network activity event sources for this demo. For the first source, I select ec2.amazonaws.com option. For Log selector template, I can use templates for common use cases or create fine-grained filters for specific scenarios. For example, to log all API activities traversing the VPC endpoint, I can choose the Log all events template. I choose Log network activity access denied events template to log only access denied events. Optionally, I can enter a name in the Selector name field to identify the log selector template, such as Include network activity events for Amazon EC2.

As a second example, I choose Custom to create custom filters on multiple fields, such as eventName and vpcEndpointId. I can specify specific VPC endpoint IDs or filter the results to include only the VPC endpoints that match specific criteria. For Advanced event selectors, I choose vpcEndpointId from the Field dropdown, choose equals as Operator, and enter the VPC endpoint ID. When I expand the JSON view, I can see my event selectors as a JSON block. I choose Next and after reviewing the selections, I choose Create trail.

After it’s configured, CloudTrail will begin logging network activity events for my VPC endpoints, helping me analyze and act on this data. To analyze AWS CloudTrail network activity events, you can use the CloudTrail console, AWS Command Line Interface (AWS CLI), and AWS SDK to retrieve relevant logs. You can also use CloudTrail Lake to capture, store and analyze your network activity events. If you are using Trails, you can use Amazon Athena to query and filter these events based on specific criteria. Regular analysis of these events can help you maintain security, comply with regulations, and optimize your network infrastructure in AWS.

Now available
CloudTrail network activity events for VPC endpoint logging provide you with a powerful tool to enhance your security posture, detect potential threats, and gain deeper insights into your VPC network traffic. This feature addresses your critical needs for comprehensive visibility and control over your AWS environments.

Network activity events for VPC endpoints are available in all commercial AWS Regions.

For pricing information, visit AWS CloudTrail pricing.

To get started with CloudTrail network activity events, visit AWS CloudTrail. For more information on CloudTrail and its features, refer to the AWS CloudTrail documentation.

— Esra

AWS Weekly Roundup: AWS Step Functions, AWS CloudFormation, Amazon Q Developer, and more (February 10, 2024)

Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-aws-step-functions-aws-cloudformation-amazon-q-developer-and-more-february-10-2024/

We are well settled into 2025 by now, but many people are still catching up with all the exciting new releases and announcements that came out of re:Invent last year. There have been hundreds of re:Invent recap events around the world since the beginning of the year, including in-person all-day official AWS events with multiple tracks to help you discover and dive deeper into the releases you care about, as well as community and virtual events.

Last month, I was lucky to be a co-host for AWS EMEA re:Invent re:Cap which was a nearly 4-hour livestream with experts featuring demos, whiteboard sessions, and a live Q&A. The good news is that you can now watch it on-demand! We had a great team and thousands of people enjoyed learning through the virtual experience. I recommend you check it out or share it with colleagues who have not been able to attend any re:Invent re:Cap events.

The Korean team also did an amazing job hosting their own virtual re:Invent re:Cap event, and it’s also now available on-demand. So if you speak Korean I do recommend you check it out.

If you’re more of a reader, then we have a treat for you. You can download the full official re:Invent re:Cap deck with all the slides covering releases across all areas by visiting community.aws! While there, you can also check all the upcoming in-person re:Invent re:Cap community events remaining across the globe for a chance to still attend one of those in a city near you.

But as we know, new releases, announcements, and updates don’t stop at re:Invent. Every week there are even more, and this is why we have this Weekly Roundup series that you can read every Monday to get the AWS news highlights from the week before.

So here’s what caught my attention last week.

Last week’s AWS Launches
If you use AWS Step Functions you may be interested in these:

Amazon Q Developer also got a couple of updates:

Here are some other releases that caught my attention this week from a variety of other AWS services:

AWS CloudFormation introduces stack refactoring – You can now split your CloudFormation stacks, move resources from one stack to another, and change the logical name of resources within the same stack. This adds a lot of flexibility enabling you to keep up with changes within your organization and architectures, such as streamlining resource lifecycle management for existing stacks, keeping up with naming convention changes, and other cases. You can refactor your stacks by using the AWS command line interface (CLI) or AWS SDK.

AWS Config now supports 4 new release typesAWS Config is great for monitoring resources across your AWS environment and help you towards ensuring alignment with your company and security policies as well as compliance requirements. It now has four new types of resources enabling you to monitor Amazon VPC block public access settings, any exceptions made within those settings, as well as monitor S3 Express One Zone bucket policies and directory bucket settings.

Automated recovery of Microsoft SQL Server on EC2 instan ces with VSS – You can now use a new feature called Volume Shadow Copy Services (VSS) to backup Microsoft SQL Server databases to Amazon Elastic Block Store (EBS) snapshots while the database is running. You can then use AWS Systems Manager Automation Runbook to set a recovery point of time of your preference and it will restore the database automatically from your VSS-based EBS snapshot without incurring any downtime.

Other updates
Upcoming changes to the AWS Security Token Service (AWS STS) global endpoint – To help improve the resiliency and performance of your applications, we are making changes to the AWS STS global endpoint (https://sts.amazonaws.com), with no action required from customers. Starting in early 2025, requests to the STS global endpoint will be automatically served in the same Region as your AWS deployed workloads. For example, if your application calls sts.amazonaws.com from the US West (Oregon) Region, your calls will be served locally in the US West (Oregon) Region instead of being served by the US East (N. Virginia) Region. These changes will be released in the coming weeks and we will gradually roll it out to AWS Regions that are enabled by default by mid-2025.

Upcoming AWS and community events

AWS Public Sector Day London, February 27 — Join public sector leaders and innovators to explore how AWS is enabling digital transformation in government, education, and healthcare.

AWS Innovate GenAI + Data Edition — A free online conference focusing on generative AI and data innovations. Available in multiple Regions: APJC and EMEA (March 6), North America (March 13), Greater China Region (March 14), and Latin America (April 8).

Browse more upcoming AWS led in-person and virtual developer-focused events.

Looking for some reading recommendations? At the beginning of every year Dr. Werner Vogles, VP and CTO of Amazon, publishes a list of recommended books that he believes should have your attention. This year’s list is looking particularly good in my opinion!

That’s it for this week! For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

See you next time 🙂

Matheus Guimaraes | @codingmatheus

2024 PiTuKri ISAE 3000 Type II attestation report available with 179 services in scope

Post Syndicated from Tariro Dongo original https://aws.amazon.com/blogs/security/2024-pitukri-isae-3000-type-ii-attestation-report-available-with-179-services-in-scope/

Amazon Web Services (AWS) is pleased to announce the issuance of the Criteria to Assess the Information Security of Cloud Services (PiTuKri) Type II attestation report with 179 services in scope.

The Finnish Transport and Communications Agency (Traficom) Cyber Security Centre published PiTuKri, which consists of 52 criteria that provide guidance across 11 domains for assessing the security of cloud service providers.

An independent third-party audit firm issued the report to assure customers that the AWS control environment is appropriately designed and operating effectively to demonstrate adherence with PiTuKri requirements. This attestation demonstrates the AWS commitment to adhere to security expectations for cloud service providers set by Traficom.

The latest report covers a 12-month period from October 1, 2023 to September 30, 2024. AWS has added the following 10 services to the current PiTuKri scope:

Customers can find the PiTuKri ISAE 3000 report on AWS Artifact. To learn more about the complete list of services in scope, see AWS Compliance Programs and AWS Services in Scope for PiTuKri.

AWS strives to continuously bring new services into the scope of its compliance programs to help you meet your architectural and regulatory needs. Contact your AWS account team for questions about the PiTuKri report.

To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page.

If you have feedback about this post, submit comments in the Comments section below.

Tariro Dongo
Tariro Dongo

Tari is a Security Assurance Program Manager at AWS, based in London. Tari is responsible for third-party and customer audits, attestations, certifications, and assessments across EMEA. Previously, Tari worked for over 12 years in security assurance and technology risk in the big four and financial services industry.

2024 FINMA ISAE 3000 Type II attestation report available with 179 services in scope

Post Syndicated from Tariro Dongo original https://aws.amazon.com/blogs/security/2024-finma-isae-3000-type-ii-attestation-report-available-with-179-services-in-scope/

Amazon Web Services (AWS) is pleased to announce the issuance of the Swiss Financial Market Supervisory Authority (FINMA) Type II attestation report with 179 services in scope.

The Swiss Financial Market Supervisory Authority (FINMA) has published several requirements and guidelines about engaging with outsourced services for the regulated financial services customers in Switzerland.

An independent third-party audit firm issued the report to assure customers that the AWS control environment is appropriately designed and operating effectively to support adherence with FINMA requirements.

The latest report covers the 12-month period from October 1, 2023 to September 30, 2024, for the following circulars:

  • 2018/03 “Outsourcing – banks, insurance companies and selected financial institutions under FinIA”
  • 2023/01 “Operational risks and resilience – banks”
  • Business Continuity Management (BCM) minimum standards proposed by the Swiss Insurance Association

AWS has added the following 10 services to the current FINMA scope:

Customers can find the FINMA ISAE 3000 report on AWS Artifact. To learn more about the complete list of services in scope, see AWS Compliance Programs and AWS Services in Scope for FINMA.

AWS strives to continuously bring new services into the scope of its compliance programs to help you meet your architectural and regulatory needs. Contact your AWS account team for questions about the FINMA report.

To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page.

If you have feedback about this post, submit comments in the Comments section below.

Tariro Dongo
Tariro Dongo

Tari is a Security Assurance Program Manager at AWS, based in London. Tari is responsible for third-party and customer audits, attestations, certifications, and assessments across EMEA. Previously, Tari worked for over 12 years in security assurance and technology risk in the big four and financial services industry.

AWS CodeBuild for macOS adds support for Fastlane

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/codebuild-for-macos-adds-support-for-fastlane/

I’m pleased to announce the availability of Fastlane in your AWS CodeBuild for macOS environments. AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages.

Fastlane is an open source tool suite designed to automate various aspects of mobile application development. It provides mobile application developers with a centralized set of tools to manage tasks such as code signing, screenshot generation, beta distribution, and app store submissions. It integrates with popular continuous integration and continuous deployment (CI/CD) platforms and supports both iOS and Android development workflows. Although Fastlane offers significant automation capabilities, developers may encounter challenges during its setup and maintenance. Configuring Fastlane can be complex, particularly for teams unfamiliar with the syntax and package management system of Ruby. Keeping Fastlane and its dependencies up to date requires ongoing effort, because updates to mobile platforms or third-party services may necessitate adjustments to existing workflows.

When we introduced CodeBuild for macOS in August 2024, we knew that one of your challenges was to install and maintain Fastlane in your build environment. Although it was possible to manually install Fastlane in a custom build environment, at AWS, we remove the undifferentiated heaving lifting from your infrastructure so you can spend more time on the aspects that matter for your business. Starting today, Fastlane is installed by default, and you can use the familiar command fastlane buildin your buildspec.yaml file.

Fastlane and code signing
To distribute an application on the App Store, developers must sign their binary with a private key generated on the Apple Developer portal. This private key, along with the certificate that validates it, must be accessible during the build process. This can be a challenge for development teams because they need to share the development private key (which allows deployment on selected test devices) among team members. Additionally, the distribution private key (which enables publishing on the App Store) must be available during the signing process before uploading the binary to the App Store.

Fastlane is a versatile build system in that it also helps developers with the management of development and distribution keys and certificates. Developers can use fastlane match to share signing materials in a team and make them securely and easily accessible on individual developers’ machines and on the CI environment. match allows the storage of private keys, the certificates, and the mobile provisioning profiles on a secured share storage. It makes sure that the local build environment, whether it’s a developer laptop or a server machine in the cloud, stays in sync with the shared storage. At build time, it securely downloads the required certificates to sign your app and configures the build machine to allow the codesign utility to pick them up.

match allows the sharing of signing secrets through GitHub, GitLab, Google Cloud Storage, Azure DevOps, and Amazon Simple Storage Service (Amazon S3).

If you already use one of these and you’re migrating your projects to CodeBuild, you don’t have much to do. You only need to make sure your CodeBuild build environment has access to the shared storage (see step 3 in the demo).

Let’s see how it works
If you’re new to Fastlane or CodeBuild, let’s see how it works.

For this demo, I start with an existing iOS project. The project is already configured to be built on CodeBuild. You can refer to my previous blog post, Add macOS to your continuous integration pipelines with AWS CodeBuild, to learn more details.

I’ll show you how to get started in three steps:

  • Import your existing signing materials to a shared private GitHub repository
  • Configure fastlane to build and sign your project
  • Use fastlanewith CodeBuild

Step 1: Import your signing materials

Most of the fastlane documentation I read explains how to create a new key pair and a new certificate to get started. Although this is certainly true for new projects, in real life, you probably already have your project and your signing keys. So, the first step is to import these existing signing materials.

Apple App Store uses different keys and certificates for development and distribution (there are also ad hoc and enterprise certificates, but these are outside the scope of this post). You must have three files for each usage (that’s a total of six files):

  • A .mobileprovision file that you can create and download from the Apple developer console. The provisioning profile links your identity, the app identity, and the entitlements the app might have.
  • A .cer file, which is the certificate emitted by Apple to validate your private key. You can download this from the Apple Developer portal. Select the certificate, then select Download.
  • A .p12 file, which contains your private key. You can download the key when you create it in the Apple Developer portal. If you didn’t download it but have it on your machine, you can export it from the Apple Keychain app. Note that the KeyChain.app is hidden in macOS 15.x. You can open it with open /System/Library/CoreServices/Applications/Keychain\ Access.app. Select the key you want to export and right click to select Export.
Export p12 files from Keychain

When you have these files, create a fastlane/Matchfile file with the following content:

git_url("https://github.com/sebsto/secret.git")
storage_mode("git")
type("development")
# or use appstore to use the distribution signing key and certificate
# type("appstore")

Be sure to replace the URL of your GitHub repository and make sure this repository is private. It will serve as a storage for your signing key and certificate.

Then, I import my existing files with the fastlane match import --type appstore command. I repeat the command for each environment: appstore and development.

The very first time, fastlane prompts me for my Apple Id username and password. It connects to App Store Connect to verify the validity of the certificates or to create new ones when necessary. The session cookie is stored in ~/.fastlane/spaceship/<your apple user id>/cookie.

fastlane match also asks for a password. It uses this password to generate a key to crypt the signing materials on the storage. Don’t forget this password because it will be used at build time to import the signing materials on the build machine.

Here is the command and its output in full:

 fastlane match import --type appstore

[✔] 🚀
[16:43:54]: Successfully loaded '~/amplify-ios-getting-started/code/fastlane/Matchfile' 📄

+-----------------------------------------------------+
| Detected Values from './fastlane/Matchfile'         |
+--------------+--------------------------------------+
| git_url.     | https://github.com/sebsto/secret.git |
| storage_mode | git                                  |
| type         | development                          |
+--------------+--------------------------------------+

[16:43:54]: Certificate (.cer) path:
./secrets/sebsto-apple-dist.cer
[16:44:07]: Private key (.p12) path:
./secrets/sebsto-apple-dist.p12
[16:44:12]: Provisioning profile (.mobileprovision or .provisionprofile) path or leave empty to skip
this file:
./secrets/amplifyiosgettingstarteddist.mobileprovision
[16:44:25]: Cloning remote git repo...
[16:44:25]: If cloning the repo takes too long, you can use the `clone_branch_directly` option in match.
[16:44:27]: Checking out branch master...
[16:44:27]: Enter the passphrase that should be used to encrypt/decrypt your certificates
[16:44:27]: This passphrase is specific per repository and will be stored in your local keychain
[16:44:27]: Make sure to remember the password, as you'll need it when you run match on a different machine
[16:44:27]: Passphrase for Match storage: ********
[16:44:30]: Type passphrase again: ********
security: SecKeychainAddInternetPassword <NULL>: The specified item already exists in the keychain.
[16:44:31]: 🔓 Successfully decrypted certificates repo
[16:44:31]: Repo is at: '/var/folders/14/nwpsn4b504gfp02_mrbyd2jr0000gr/T/d20250131-41830-z7b4ic'
[16:44:31]: Login to App Store Connect ([email protected])
[16:44:33]: Enter the passphrase that should be used to encrypt/decrypt your certificates
[16:44:33]: This passphrase is specific per repository and will be stored in your local keychain
[16:44:33]: Make sure to remember the password, as you'll need it when you run match on a different machine
[16:44:33]: Passphrase for Match storage: ********
[16:44:37]: Type passphrase again: ********
security: SecKeychainAddInternetPassword <NULL>: The specified item already exists in the keychain.
[16:44:39]: 🔒 Successfully encrypted certificates repo
[16:44:39]: Pushing changes to remote git repo...
[16:44:40]: Finished uploading files to Git Repo [https://github.com/sebsto/secret.git]

I verify that Fastlane imported my signing material to my Git repository.

Fastlane match - github repo after import

I can also configure my local machine to use these signing materials during the next build:

» fastlane match appstore 

[✔] 🚀 
[17:39:08]: Successfully loaded '~/amplify-ios-getting-started/code/fastlane/Matchfile' 📄

+-----------------------------------------------------+
|   Detected Values from './fastlane/Matchfile'       |
+--------------+--------------------------------------+
| git_url      | https://github.com/sebsto/secret.git |
| storage_mode | git                                  |
| type         | development                          |
+--------------+--------------------------------------+


+-------------------------------------------------------------------------------------------+
|                                 Summary for match 2.226.0                                 |
+----------------------------------------+--------------------------------------------------+
| type                                   | appstore                                         |
| readonly                               | false                                            |
| generate_apple_certs                   | true                                             |
| skip_provisioning_profiles             | false                                            |
| app_identifier                         | ["com.amazonaws.amplify.mobile.getting-started"] |
| username                               | xxxx@xxxxxxxxx                                   |
| team_id                                | XXXXXXXXXX                                       |
| storage_mode                           | git                                              |
| git_url                                | https://github.com/sebsto/secret.git             |
| git_branch                             | master                                           |
| shallow_clone                          | false                                            |
| clone_branch_directly                  | false                                            |
| skip_google_cloud_account_confirmation | false                                            |
| s3_skip_encryption                     | false                                            |
| gitlab_host                            | https://gitlab.com                               |
| keychain_name                          | login.keychain                                   |
| force                                  | false                                            |
| force_for_new_devices                  | false                                            |
| include_mac_in_profiles                | false                                            |
| include_all_certificates               | false                                            |
| force_for_new_certificates             | false                                            |
| skip_confirmation                      | false                                            |
| safe_remove_certs                      | false                                            |
| skip_docs                              | false                                            |
| platform                               | ios                                              |
| derive_catalyst_app_identifier         | false                                            |
| fail_on_name_taken                     | false                                            |
| skip_certificate_matching              | false                                            |
| skip_set_partition_list                | false                                            |
| force_legacy_encryption                | false                                            |
| verbose                                | false                                            |
+----------------------------------------+--------------------------------------------------+

[17:39:08]: Cloning remote git repo...
[17:39:08]: If cloning the repo takes too long, you can use the `clone_branch_directly` option in match.
[17:39:10]: Checking out branch master...
[17:39:10]: Enter the passphrase that should be used to encrypt/decrypt your certificates
[17:39:10]: This passphrase is specific per repository and will be stored in your local keychain
[17:39:10]: Make sure to remember the password, as you'll need it when you run match on a different machine
[17:39:10]: Passphrase for Match storage: ********
[17:39:13]: Type passphrase again: ********
security: SecKeychainAddInternetPassword <NULL>: The specified item already exists in the keychain.
[17:39:15]: 🔓  Successfully decrypted certificates repo
[17:39:15]: Verifying that the certificate and profile are still valid on the Dev Portal...
[17:39:17]: Installing certificate...

+-------------------------------------------------------------------------+
|                          Installed Certificate                          |
+-------------------+-----------------------------------------------------+
| User ID           | XXXXXXXXXX                                          |
| Common Name       | Apple Distribution: Sebastien Stormacq (XXXXXXXXXX) |
| Organisation Unit | XXXXXXXXXX                                          |
| Organisation      | Sebastien Stormacq                                  |
| Country           | US                                                  |
| Start Datetime    | 2024-10-29 09:55:43 UTC                             |
| End Datetime      | 2025-10-29 09:55:42 UTC                             |
+-------------------+-----------------------------------------------------+

[17:39:18]: Installing provisioning profile...

+-------------------------------------------------------------------------------------------------------------------+
|                                          Installed Provisioning Profile                                           |
+---------------------+----------------------------------------------+----------------------------------------------+
| Parameter           | Environment Variable                         | Value                                        |
+---------------------+----------------------------------------------+----------------------------------------------+
| App Identifier      |                                              | com.amazonaws.amplify.mobile.getting-starte  |
|                     |                                              | d                                            |
| Type                |                                              | appstore                                     |
| Platform            |                                              | ios                                          |
| Profile UUID        | sigh_com.amazonaws.amplify.mobile.getting-s  | 4e497882-d80f-4684-945a-8bfec1b310b9         |
|                     | tarted_appstore                              |                                              |
| Profile Name        | sigh_com.amazonaws.amplify.mobile.getting-s  | amplify-ios-getting-started-dist             |
|                     | tarted_appstore_profile-name                 |                                              |
| Profile Path        | sigh_com.amazonaws.amplify.mobile.getting-s  | /Users/stormacq/Library/MobileDevice/Provis  |
|                     | tarted_appstore_profile-path                 | ioning                                       |
|                     |                                              | Profiles/4e497882-d80f-4684-945a-8bfec1b310  |
|                     |                                              | b9.mobileprovision                           |
| Development Team ID | sigh_com.amazonaws.amplify.mobile.getting-s  | XXXXXXXXXX                                   |
|                     | tarted_appstore_team-id                      |                                              |
| Certificate Name    | sigh_com.amazonaws.amplify.mobile.getting-s  | Apple Distribution: Sebastien Stormacq       |
|                     | tarted_appstore_certificate-name             | (XXXXXXXXXX)                                 |
+---------------------+----------------------------------------------+----------------------------------------------+

[17:39:18]: All required keys, certificates and provisioning profiles are installed 🙌

Step 2: Configure Fastlane to sign your project

I create a Fastlane build configuration file in fastlane/Fastfile (you can use fastlane init command to get started):

default_platform(:ios)

platform :ios do
  before_all do
    setup_ci
  end

  desc "Build and Sign the binary"
  lane :build do
    match(type: "appstore", readonly: true)
    gym(
      scheme: "getting started",
      export_method: "app-store"
    )
  end
end

Make sure that the setup_ci action is added to the before_all section of Fastfile for the match action to function correctly. This action creates a temporary Fastlane keychain with correct permissions. Without this step, you may encounter build failures or inconsistent results.

And I test a local build with the command fastlane build. I enter the password I used when importing my keys and certificate, then I let the system build and sign my project. When everything is correctly configured, it produces a similar output.

...
[17:58:33]: Successfully exported and compressed dSYM file
[17:58:33]: Successfully exported and signed the ipa file:
[17:58:33]: ~/amplify-ios-getting-started/code/getting started.ipa

+---------------------------------------+
|           fastlane summary            |
+------+------------------+-------------+
| Step | Action           | Time (in s) |
+------+------------------+-------------+
| 1    | default_platform | 0           |
| 2    | setup_ci         | 0           |
| 3    | match            | 36          |
| 4    | gym              | 151         |
+------+------------------+-------------+

[17:58:33]: fastlane.tools finished successfully 🎉

Step 3: Configure CodeBuild to use Fastlane

Next, I create a project on CodeBuild. I’m not going into the step-by-step guide to help you to do so. You can refer to my previous post or to the CodeBuild documentation.

There is just one Fastlane-specific configuration. To access the signing materials, Fastlane requires access to three secret values that I’ll pass as environment variables:

  • MATCH_PASSWORD, the password I entered when importing the signing material. Fastlane uses this password to decipher the encrypted files in the GitHub repository
  • FASTLANE_SESSION, the value of the Apple Id session cookie, located at ~/.fastlane/spaceship/<your apple user id>/cookie. The session is valid from a couple of hours to multiple days. When the session expires, reauthenticate with the command fastlane spaceauth from your laptop and update the value of FASTLANE_SESSION with the new value of the cookie.
  • MATCH_GIT_BASIC_AUTHORIZATION, a base 64 encoding of your GitHub username, followed by a colon, followed by a personal authentication token (PAT) to access your private GitHub repository. You can generate PAT on the GitHub console in Your Profile > Settings > Developers Settings > Personal Access Token. I use this command to generate the value of this environment variable: echo -n my_git_username:my_git_pat | base64.

Note that for each of these three values, I can enter the Amazon Resource Name (ARN) of the secret on AWS Secrets Manager or the plain text value. We strongly recommend using Secrets Manager to store security-sensitive values.

I’m a security-conscious user, so I store the three secrets in Secrets Manager with these commands:

aws --region $REGION secretsmanager create-secret --name /CodeBuild/MATCH_PASSWORD --secret-string MySuperSecretPassword
aws --region $REGION secretsmanager create-secret --name /CodeBuild/FASTLANE_SESSION --secret-string $(cat ~/.fastlane/spaceship/my_appleid_username/cookie)
aws --region $REGION secretsmanager create-secret --name /CodeBuild/MATCH_GIT_BASIC_AUTHORIZATION --secret-string $(echo -n my_git_username:my_git_pat | base64)

If your build project refers to secrets stored in Secrets Manager, the build project’s service role must allow the secretsmanager:GetSecretValue action. If you chose New service role when you created your project, CodeBuild includes this action in the default service role for your build project. However, if you chose Existing service role, you must include this action to your service role separately.

For this demo, I use this AWS Identity and Access Management (IAM) policy:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"secretsmanager:GetSecretValue"
			],
			"Resource": [
				"arn:aws:secretsmanager:us-east-2:012345678912:secret:/CodeBuild/*"
			]
		}
	]
}

After I created the project in the CodeBuild section of the AWS Management Console, I enter the three environment variables. Notice that the value is the name of the secret in Secrets Manager.

Codebuild - environment variables for Fastlane match

You can also define the environment variables and their Secrets Manager secret name in your buildpsec.yaml file.

Next, I modify the buildspec.yaml file at the root of my project to use fastlane to build and sign the binary. My buildspec.yaml file now looks like this one:

# buildspec.yml
version: 0.2
phases:
  install:
    commands:
      - code/ci_actions/00_install_rosetta.sh
  pre_build:
    commands:
      - code/ci_actions/02_amplify.sh
  build:
    commands:
      - (cd code && fastlane build)
artifacts:
  name: getting-started-$(date +%Y-%m-%d).ipa
  files:
    - 'getting started.ipa'
  base-directory: 'code'

The Rosetta and Amplify scripts are required to receive the Amplify configuration for the backend. If you don’t use AWS Amplify in your project, you don’t need these.

Notice that there is nothing in the build file that downloads the signing key or prepares the keychain in the build environment; fastlane match will do that for me.

I add the new buildspec.yaml file and my ./fastlane directory to Git. I commit and push these files. git commit -m "add fastlane support" && git push

When everything goes well, I can see the build running on CodeBuild and the Succeeded message.

Codebuild - succeed message

Pricing and availability
Fastlane is now pre-installed at no extra cost on all macOS images that CodeBuild uses, in all Regions where CodeBuild for macOS is available. At the time of this writing, these are US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Sydney), and Europe (Frankfurt).

In my experience, it takes a bit of time to configure fastlane match correctly. When it’s configured, having it working on CodeBuild is pretty straightforward. Before trying this on CodeBuild, be sure it works on your local machine. When something goes wrong on CodeBuild, triple-check the values of the environment variables and make sure CodeBuild has access to your secrets on AWS Secrets Manager.

Now go build (on macOS)!

AWS renews MTCS Level 3 certification under the SS584:2020 standard

Post Syndicated from Joseph Goh original https://aws.amazon.com/blogs/security/aws-renews-mtcs-level-3-certification-under-the-ss5842020-standard/

Amazon Web Services (AWS) is pleased to announce the renewal of the Multi-Tier Cloud Security (MTCS) Level 3 certification under the SS584:2020 standard in December 2024 for the Asia Pacific (Singapore), Asia Pacific (Seoul), and United States AWS Regions, excluding AWS GovCloud (US) Regions. This achievement reaffirms our commitment to maintaining the highest security standards for our global customers, particularly those in Singapore and the Asia-Pacific.

AWS was the first cloud service provider (CSP) to attain MTCS Level 3 certification for Singapore in 2014. We continued this leadership by being among the first CSPs certified under the updated SS584:2020 Level 3 standard in 2021. Our dedication to expanding our security coverage is evident in the significant increase of in-scope services from 145 to 184, representing a 27% growth since 2021.

The MTCS standard is recognized as the world’s first cloud security standard to specify a multi-tiered management system for cloud security. This standard can be applied by CSPs to support differing cloud user needs for data sensitivity and business criticality, and the use of MTCS is mandated by the Singapore government as a requirement for public sector agencies and regulated organizations.

As part of our commitment to transparency, AWS fulfills the self-disclosure requirement for CSPs, providing detailed service-oriented information typically found in service level agreements. This allows our customers to make informed decisions about their cloud security needs.

The MTCS framework establishes three levels of security, with Level 3 being the most stringent:

  1. Level 1: Designed for non-business-critical data and systems with baseline security controls.
  2. Level 2: Addresses the needs of organizations that run business-critical data and systems in public or third-party cloud systems.
  3. Level 3: Tailored for regulated organizations with specific and more stringent security requirements, including industry-specific regulations.

Benefits of the MTCS Level 3 certification

By achieving MTCS Level 3 certification, AWS helps Singapore customers in regulated industries to securely host applications and systems with highly sensitive information. This includes confidential business data, financial records, and medical records in a Level-3-compliant MTCS environment.

As cloud technology continues to evolve, AWS remains dedicated to maintaining and exceeding the highest security standards. Our renewed MTCS Level 3 certification under the SS584:2020 standard is a testament to this commitment, enabling our customers in Singapore and around the world to use AWS services with confidence for their most sensitive and critical workloads.

You can now download the latest MTCS certificates and the MTCS Self-Disclosure Form in AWS Artifact. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact. For a full list of AWS services that are certified under MTCS, see the AWS Multi-Tier Cloud Security (MTCS) page

To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page.

If you have feedback about this post, submit comments in the Comments section below.

Joseph Goh

Joseph Goh
Joseph is the APJ ASEAN Lead at AWS, based in Singapore. He leads security audits, certifications, and regulatory compliance programs across the unique regulatory landscapes in the Asia Pacific region. Joseph is passionate about delivering programs that strengthen trust with customers and providing them assurance on cloud security.

How AWS Network Firewall session state replication maximizes high availability for your application traffic

Post Syndicated from Tushar Jagdale original https://aws.amazon.com/blogs/security/how-aws-network-firewall-session-state-replication-maximizes-high-availability-for-your-application-traffic/

AWS Network Firewall is a managed, stateful network firewall and intrusion protection service that you can use to implement firewall rules for fine grained control over your network traffic. With Network Firewall, you can filter traffic at the perimeter of your virtual private cloud (VPC); including filtering traffic going to and coming from an internet gateway, NAT gateway, over VPN or AWS Direct Connect. In this post, we show you how AWS Network Firewall uses session state replication to help maintain high availability for your application traffic.

Network Firewall session state replication

The stateful engine used by Network Firewall applies network security policies according to customer-defined rules. When inspecting stateful flows, Network Firewall maintains a reconstructed state—stored in a flow table—for each connection. Network Firewall is a distributed service, spreading traffic and the connection state over many backend firewall hosts using Gateway Load Balancer endpoints.

There are several operational reasons why backend hosts might be brought in and out of service, such as autoscaling events to adjust for changing traffic levels or installing software for security and other service updates. During these operations, if the traffic contains long-lived traffic flows, Network Firewall allows these flows to drain for several minutes before replacing the hosts and re-balancing them to the newer hosts.

When an existing flow is rebalanced onto a new host that doesn’t contain its connection state, the connection is handled according to the firewall policy’s configured stream exception policy. Network Firewall will either drop or reject these connections, or Network Firewall can be configured to continue applying the firewall policy’s rules without the context from earlier in the connection. Both choices have implications: using the drop or reject action maintains security by forcing connections to be restarted and re-inspected but at the cost of some broken connections, while the continue action requires writing firewall rules that can accept connections that are broken midstream.

In December 2024, AWS introduced the ability for Network Firewall to replicate the session state between backend hosts, reducing the number of cases where the stream exception policy needs to be applied to broken connections. Now, the majority of these failed-over flows go to a new host that already contains the correct flow state, allowing those connections to continue without interruption. This feature is automatically enabled by default on all firewalls and no action is required by you. The stream exception policy will continue to be applied in rare cases where the state cannot automatically be replicated or when connections are broken for other reasons such as routing changes in the network.

Figure 1: Session state replication flowFigure 1: Session state replication flow

Figure 1 shows the sequence of events to maintain persistent connection during operations to replace a backend firewall host:

  1. A network flow arrives at the Network Firewall endpoint and is forwarded to a firewall backend host (firewall host 1) by Gateway Load Balancer.
  2. Firewall host 1 is de-registered from the Gateway Load Balancer target group, which causes Gateway Load Balancer to stop assigning new flows to the host but maintains existing ones.
  3. The service exports the remaining session state table from backend firewall host 1.
  4. The service replicates the session table data to other healthy backend hosts.
  5. A flow flush operation causes Gateway Load Balancer to reassign the remaining flows on host 1 to other in-service hosts, where the ongoing flows will continue being inspected by the stateful inspection rules configured on the firewall.

Some of the key considerations for your own workloads:

Conclusion

In this post, we outlined how AWS Network Firewall uses its ability to replicate connection state across multiple backend firewall hosts to maintain high availability for your application traffic. This feature is enabled by default for existing and new customers and there are no additional costs or configuration changes required to use this feature.

To learn more about AWS Network Firewall, see the AWS Network Firewall product page and the service documentation. To see which AWS Regions AWS Network Firewall is available in, see AWS Services by Region.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Tushar Jagdale
Tushar Jagdale

Tushar is a Specialist Solutions Architect focused on Networking at AWS, where he helps customers build and design scalable, highly available, secure, resilient, and cost effective networks. He has over 15 years of experience building and securing data center and cloud networks.
Amish Shah
Amish Shah

Amish is a seasoned product leader with over 15 years’ experience developing innovative and scalable solutions for networking, security, and cloud use cases. He currently leads the AWS Network Firewall service, where he helps develop security solutions that protect AWS workloads. Outside of work, Amish enjoys playing cricket and soccer, loves to travel, and has recently started collecting niche fragrances.
Vikram Saurabh
Vikram Saurabh

Vikram is an experienced engineering leader with 20 years of experience in software engineering, primarily in building firewall products and services. He currently leads the AWS Network Firewall engineering team and has previously led the engineering team of Route53 DNS Firewall. Outside of work, Vikram enjoys playing cricket, hiking, and solving math puzzles.
Jamie Lavigne
Jamie Lavigne

Jamie is a Principal Software Dev Engineer with over 10 years of experience building and operating highly resilient network security services at AWS. Jamie has been a technical lead of the AWS Network Firewall service since its inception and continues to focus on ensuring that it meets the security, compliance, and availability needs of its internal and external customers.

AWS Weekly Roundup: DeepSeek-R1, S3 Metadata, Elastic Beanstalk updates, and more (February 3, 2024)

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-deepseek-r1-s3-metadata-elastic-beanstalk-updates-and-more-february-3-2024/

Last week, I had an amazing time attending AWS Community Day Thailand in Bangkok. This event came at an exciting time, following the recent launch of the AWS Asia Pacific (Bangkok) Region. We had over 300 attendees and featured 15 speakers from the community, including an AWS Hero and 4 AWS Community Builders who shared their technical expertise and experiences.

The highlight was definitely Jeff Barr, AWS Vice President & Chief Evangelist, delivering an inspiring keynote titled “Next-Generation Software Development”, which set the perfect tone for the day. The day kicked off with welcoming remarks from Vatsun Thirapatarapong, AWS Country Manager for Thailand, and was made even more special thanks to the tremendous support from both the AWS User Group volunteers and the AWS Thailand team.

Here’s a photo capturing the excitement from the event: 

Last week’s AWS Launches
There are 30+ launches last week and here are some launches that caught my attention:

DeepSeek-R1 models now available on AWS — Channy wrote on how you can now deploy DeepSeek-R1 models in Amazon Bedrock and Amazon SageMaker AI. This helps you to build and scale generative AI applications with minimal infrastructure investment.

Amazon S3 Tables increases table limit to 10,000 per bucket — S3 Tables now supports creating up to 10,000 tables in each table bucket, allowing you to scale up to 100,000 tables across 10 buckets within an AWS Region per account.

Amazon S3 Metadata now generally available — S3 Metadata provides automated and easily queried metadata that updates in near real-time, simplifying business analytics and real-time inference applications. It supports both system-defined and custom metadata, including integration with AWS analytics services.

AWS Amplify adds TypeScript Data client support for Lambda functions — Developers can now use the Amplify Data client within AWS Lambda functions, enabling consistent type-safe data operations across frontend and backend applications.

AWS Elastic Beanstalk adds Python 3.13, .NET 9, and PHP 8.4 support on Amazon Linux 2023 — AWS Elastic Beanstalk brings the latest language features and improvements to application deployments while benefiting from Amazon Linux 2023 enhanced security and performance features.

From community.aws
Here’s my top 5 personal favorites posts from community.aws:

Upcoming AWS and community events
Check your calendars and sign up for upcoming AWS and community events:

  • AWS Korea re:Invent reCap Online, February 2-4 — A virtual event recapping key announcements and innovations from re:Invent 2023 for the Korean audience.
  • AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs. Upcoming AWS Community Day is in Ahmedabad (February 8).
  • AWS Public Sector Day London, February 27 — Join public sector leaders and innovators to explore how AWS is enabling digital transformation in government, education, and healthcare.
  • AWS Innovate GenAI + Data Edition — A free online conference focusing on generative AI and data innovations. Available in multiple Regions: APJC and EMEA (March 6), North America (March 13), Greater China Region (March 14), and Latin America (April 8).

Browse more upcoming AWS led in-person and virtual developer-focused events.

AWS Community re:Invent re:Caps

Lastly, if you want to learn about top announcements and innovations from AWS re:Invent, the AWS Community shares a summary from a community perspective of these announcements so you can get up to speed. Download the AWS Community re:Invent re:Caps deck

That’s all for this week. Check back next Monday for another Weekly Roundup!

Donnie

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Enhancing Code Generation with Real-Time Execution in Amazon Q Developer

Post Syndicated from Sundaresh Iyer original https://aws.amazon.com/blogs/devops/enhancing-code-generation-with-real-time-execution-in-amazon-q-developer/

As AI continues to drive rapid innovation in software development, a reliable runtime environment for real-time testing is essential to promote high-quality code generation. Developers often face delays in feature delivery as they spend significant time debugging and iterating on AI-generated code to verify it meets project requirements. Previously, the Amazon Q Developer development agent focused on code generation. With its latest update, the agent can now build and test code in real time, validating changes before a developer’s review. This new capability directly addresses community feedback regarding code recommendation quality, detecting errors, keeping generated code in sync with the project’s current state, and accelerating the development process by streamlining both code generation and testing workflows.

With natural language input and project-specific context, the Amazon Q Developer agent is designed to assist in implementing complex multi-file features, bug fixes, and test suites. For example, a developer can request Amazon Q Developer agent to add a checkout feature to an e-commerce application. The agent then analyzes the existing codebase, makes all necessary code changes and tests within minutes, including running any unit tests and building the code to verify the code is ready for review. This approach significantly improves development efficiency and reduces errors. To use the Amazon Q Developer agent in your IDE, simply install the Amazon Q extension and use the /dev command in the chat window to initiate requests.

Once the /dev command is entered in the IDE, the agent packages the project and securely uploads it to Amazon Q, initiating project-specific code generation. The Amazon Q Developer agent not only focuses on code generation, but also maintains a real-time connection with the developer, providing updates throughout the process and delivering a polished patch or implementation for the requested feature.

This real-time execution is powered by a Devfile, which defines the development environment and commands the agent can use. If a project doesn’t already have a Devfile, Amazon Q Developer will prompt users to create one after their first run of /dev. Without a Devfile, the agent will develop solutions without the additional feedback provided by running builds or unit tests, limiting developers’ ability to receive real-time feedback during the development process.

Core enhancements in the latest Amazon Q Developer update

  • Customizable Commands: Developers can specify commands in a Devfile to control which commands the AI agent runs, reducing unnecessary steps and improving accuracy.
  • Flexible Environment Setup: Developers can use custom Docker images preloaded with dependencies for faster startup times, providing the agent has all necessary tools.
  • Sandboxed Security: Amazon Q Developer secures the execution within isolated environments, offering comprehensive logging and robust permission controls to safeguard any changes made.

With this setup, Amazon Q Developer can execute tests, apply migrations, and run installation commands directly within a sandbox, providing feedback to the agent for iterative improvements.

Security and isolation

Given the security-sensitive nature of executing AI-generated code, the Amazon Q Developer agent introduces several safeguards:

Environment Isolation: Commands are executed within an isolated, managed sandbox environment configured without credentials to access non-public internet resources, ensuring that only authorized actions are performed securely.
Devfile driven: This feature requires a Devfile, and the Devfile configuration allows developers to control which commands the agent uses during the development process.

Getting started with the Amazon Q Developer agent

To get started, you need to have an AWS Builder ID or be part of an organization with an AWS IAM Identity Center instance that allows you to use Amazon Q. To use Amazon Q Developer agent for software development in Visual Studio Code, start by installing the Amazon Q extension. Find the latest version of the extension on the Amazon Q Developer page. The extension is also available for JetBrains, Eclipse (Preview), and Visual Studio IDEs. For a detailed list of supported IDEs and the features available in each, refer to the Amazon Q Developer documentation.

Amazon_Q_Developer_AI_Assistant

After authenticating, you can invoke the feature development agent by entering /dev in Amazon Q’s chat window.

Amazon_Q_Developer_AI_Assistant_Dev

Amazon Q Developer leverages an isolated sandbox environment to securely execute code generated by the Amazon Q Developer agent. This keeps the generated code running safely and in sync with the original codebase. Here’s a breakdown of how the process works:

  • Initiating the Execution Environment: Upon receiving a prompt, the Amazon Q Developer agent initiates a sandbox instance or the customer specified docker container, which serves as a sandbox environment for code execution.
  • Executing Commands Safely: The Amazon Q Developer agent safely executes a curated list of shell commands based on customer specifications in a Devfile. Devfiles model the configuration and dependencies of a development environment, enabling consistent environment reproduction and reducing manual setup effort. Developers can define custom commands within the Devfile to control actions in the sandbox, such as installing dependencies, running tests, applying database migrations, or executing build scripts, improving accuracy and efficiency.
  • Feedback and Sync: After each command runs, changes to the code are tracked and the AI agent is provided with real-time feedback and enabling iterative improvements.

Use case example 1: Adding a test suite to an existing project

Let’s say you want to enhance the functionality of a React-based application, like the example react-solitaire from GitHub. As you add new features, it’s crucial to ensure that existing functionality remains intact and doesn’t break with each update. To achieve this, you aim to create a test suite for continuous testing and iteration of your code.

To illustrate this, we’ll clone the React project from GitHub and add a Devfile to define the environment and dependencies. The Devfile configures the sandbox to execute and test code changes safely, allowing updates to be made without affecting the working features.

Amazon_Q_Developer_AI_Assistant_Devfile

Once the repository is cloned, place the Devfile in the root of the project folder. Then, open the Amazon Q IDE in Visual Studio Code and enter the /dev command to prompt the creation of a tailored test suite for the repository.

Amazon_Q_Developer_AI_Assistant_Feature_Development

The Amazon Q Developer agent then begins analyzing your codebase, sharing real-time updates on the changes it’s making and files it’s working with. The agent starts by exploring the project structure, planning the necessary updates, and generating the test suite.

Amazon_Q_Developer_AI_Assistant_Summary_of_Changes

After a few steps, the agent has created the required test suites.

Amazon_Q_Developer_AI_Assistant_Chat

Then, the agent executes the tests, continuously monitoring for any failures. When an issue is detected, it doesn’t stop immediately—it actively improves the code based on feedback from the tests, repeating this process up to three times. If the issue remains unresolved after three iterations, the agent aborts the process. However, if the issue is fixed, it moves on to the next step. For instance, when the agent identified that Enzyme didn’t support React 18, it addressed the issue and re-ran the tests in the testing environment.

Amazon_Q_Developer_AI_Assistant_QDev

Once the issue was resolved, the agent moved on to the next step, displaying all the changes and files it has modified in the sandbox and asks if you want to accept the changes or provide feedback.

Amazon_Q_Developer_AI_Assistant_Accept_Code

If you are satisfied with the output, you can accept the changes or you can provide feedback to the agent and request that it regenerates the code again.

Use case example 2: Re-run tests when a feature is updated

After successfully creating and executing the tests, the agent was prompted to add a new feature that displays the name of the game in the UI. The agent analyzed the repository, identified the files requiring updates, and determined the precise locations to implement the changes.

Amazon_Q_Developer_AI_Assistant_Summary_of_Changes

After applying the updates, the agent executes tests to validate the new feature, promoting seamless integration with the existing codebase and maintaining reliability throughout the development process.

Amazon_Q_Developer_AI_Assistant_NPM_Install

Upon accepting the changes made by the agent, the index.html file is updated to include the text ‘Solitaire,’ integrating the new content smoothly into the existing project.

Amazon_Q_Developer_AI_Assistant_Solitaire

Conclusion

The launch of this new update in Amazon Q Developer marks a significant advancement in AI-driven development, transforming the Amazon Q Developer agent from a tool focused on code generation to a robust execution engine. By enabling developers to validate and test code changes in real-time, this enhancement can improve the accuracy and reliability of AI-generated files and fixes.

With flexible options to use AWS managed sandbox or bring custom environments, developers gain control to maximize the Amazon Q Developer agent’s potential. The new execution capability empowers teams to iterate faster, make informed adjustments, and leverage a secure, intelligent platform tailored to their needs.

You can try it out today by updating or installing your Amazon Q Developer extension on VS Code or JetBrains.

DeepSeek-R1 models now available on AWS

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/deepseek-r1-models-now-available-on-aws/

During this past AWS re:Invent, Amazon CEO Andy Jassy shared valuable lessons learned from Amazon’s own experience developing nearly 1,000 generative AI applications across the company. Drawing from this extensive scale of AI deployment, Jassy offered three key observations that have shaped Amazon’s approach to enterprise AI implementation.

First is that as you get to scale in generative AI applications, the cost of compute really matters. People are very hungry for better price performance. The second is actually quite difficult to build a really good generative AI application. The third is the diversity of the models being used when we gave our builders freedom to pick what they want to do. It doesn’t surprise us, because we keep learning the same lesson over and over and over again, which is that there is never going to be one tool to rule the world.

As Andy emphasized, a broad and deep range of models provided by Amazon empowers customers to choose the precise capabilities that best serve their unique needs. By closely monitoring both customer needs and technological advancements, AWS regularly expands our curated selection of models to include promising new models alongside established industry favorites. This ongoing expansion of high-performing and differentiated model offerings helps customers stay at the forefront of AI innovation.

This leads us to Chinese AI startup DeepSeek. DeepSeek launched DeepSeek-V3 on December 2024 and subsequently released DeepSeek-R1, DeepSeek-R1-Zero with 671 billion parameters, and DeepSeek-R1-Distill models ranging from 1.5–70 billion parameters on January 20, 2025. They added their vision-based Janus-Pro-7B model on January 27, 2025. The models are publicly available and are reportedly 90-95% more affordable and cost-effective than comparable models. Per Deepseek, their model stands out for its reasoning capabilities, achieved through innovative training techniques such as reinforcement learning.

Today, you can now deploy DeepSeek-R1 models in Amazon Bedrock and Amazon SageMaker AI. Amazon Bedrock is best for teams seeking to quickly integrate pre-trained foundation models through APIs. Amazon SageMaker AI is ideal for organizations that want advanced customization, training, and deployment, with access to the underlying infrastructure. Additionally, you can also use AWS Trainium and AWS Inferentia to deploy DeepSeek-R1-Distill models cost-effectively via Amazon Elastic Compute Cloud (Amazon EC2) or Amazon SageMaker AI.

With AWS, you can use DeepSeek-R1 models to build, experiment, and responsibly scale your generative AI ideas by using this powerful, cost-efficient model with minimal infrastructure investment. You can also confidently drive generative AI innovation by building on AWS services that are uniquely designed for security. We highly recommend integrating your deployments of the DeepSeek-R1 models with Amazon Bedrock Guardrails to add a layer of protection for your generative AI applications, which can be used by both Amazon Bedrock and Amazon SageMaker AI customers.

You can choose how to deploy DeepSeek-R1 models on AWS today in a few ways: 1/ Amazon Bedrock Marketplace for the DeepSeek-R1 model, 2/ Amazon SageMaker JumpStart for the DeepSeek-R1 model, 3/ Amazon Bedrock Custom Model Import for the DeepSeek-R1-Distill models, and 4/ Amazon EC2 Trn1 instances for the DeepSeek-R1-Distill models.

Let me walk you through the various paths for getting started with DeepSeek-R1 models on AWS. Whether you’re building your first AI application or scaling existing solutions, these methods provide flexible starting points based on your team’s expertise and requirements.

1. The DeepSeek-R1 model in Amazon Bedrock Marketplace
Amazon Bedrock Marketplace offers over 100 popular, emerging, and specialized FMs alongside the current selection of industry-leading models in Amazon Bedrock. You can easily discover models in a single catalog, subscribe to the model, and then deploy the model on managed endpoints.

To access the DeepSeek-R1 model in Amazon Bedrock Marketplace, go to the Amazon Bedrock console and select Model catalog under the Foundation models section. You can quickly find DeepSeek by searching or filtering by model providers.

After checking out the model detail page including the model’s capabilities, and implementation guidelines, you can directly deploy the model by providing an endpoint name, choosing the number of instances, and selecting an instance type.

You can also configure advanced options that let you customize the security and infrastructure settings for the DeepSeek-R1 model including VPC networking, service role permissions, and encryption settings. For production deployments, you should review these settings to align with your organization’s security and compliance requirements.

With Amazon Bedrock Guardrails, you can independently evaluate user inputs and model outputs. You can control the interaction between users and DeepSeek-R1 with your defined set of policies by filtering undesirable and harmful content in generative AI applications. The DeepSeek-R1 model in Amazon Bedrock Marketplace can only be used with Bedrock’s ApplyGuardrail API to evaluate user inputs and model responses for custom and third-party FMs available outside of Amazon Bedrock. To learn more, read Implement model-independent safety measures with Amazon Bedrock Guardrails.

Amazon Bedrock Guardrails can also be integrated with other Bedrock tools including Amazon Bedrock Agents and Amazon Bedrock Knowledge Bases to build safer and more secure generative AI applications aligned with responsible AI policies. To learn more, visit the AWS Responsible AI page.

Refer to this step-by-step guide on how to deploy the DeepSeek-R1 model in Amazon Bedrock Marketplace. To learn more, visit Deploy models in Amazon Bedrock Marketplace.

2. The DeepSeek-R1 model in Amazon SageMaker JumpStart
Amazon SageMaker JumpStart is a machine learning (ML) hub with FMs, built-in algorithms, and prebuilt ML solutions that you can deploy with just a few clicks. To deploy DeepSeek-R1 in SageMaker JumpStart, you can discover the DeepSeek-R1 model in SageMaker Unified Studio, SageMaker Studio, SageMaker AI console, or programmatically through the SageMaker Python SDK.

In the Amazon SageMaker AI console, open SageMaker Unified Studio or SageMaker Studio. In case of SageMaker Studio, choose JumpStart and search for “DeepSeek-R1” in the All public models page.

You can select the model and choose deploy to create an endpoint with default settings. When the endpoint comes InService, you can make inferences by sending requests to its endpoint.

You can derive model performance and ML operations controls with Amazon SageMaker AI features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in an AWS secure environment and under your virtual private cloud (VPC) controls, helping to support data security.

As like Bedrock Marketpalce, you can use the ApplyGuardrail API in the SageMaker JumpStart to decouple safeguards for your generative AI applications from the DeepSeek-R1 model. You can now use guardrails without invoking FMs, which opens the door to more integration of standardized and thoroughly tested enterprise safeguards to your application flow regardless of the models used.

Refer to this step-by-step guide on how to deploy DeepSeek-R1 in Amazon SageMaker JumpStart. To learn more, visit Discover SageMaker JumpStart models in SageMaker Unified Studio or Deploy SageMaker JumpStart models in SageMaker Studio.

3. DeepSeek-R1-Distill models using Amazon Bedrock Custom Model Import
Amazon Bedrock Custom Model Import provides the ability to import and use your customized models alongside existing FMs through a single serverless, unified API without the need to manage underlying infrastructure. With Amazon Bedrock Custom Model Import, you can import DeepSeek-R1-Distill Llama models ranging from 1.5–70 billion parameters. As I highlighted in my blog post about Amazon Bedrock Model Distillation, the distillation process involves training smaller, more efficient models to mimic the behavior and reasoning patterns of the larger DeepSeek-R1 model with 671 billion parameters by using it as a teacher model.

After storing these publicly available models in an Amazon Simple Storage Service (Amazon S3) bucket or an Amazon SageMaker Model Registry, go to Imported models under Foundation models in the Amazon Bedrock console and import and deploy them in a fully managed and serverless environment through Amazon Bedrock. This serverless approach eliminates the need for infrastructure management while providing enterprise-grade security and scalability.

Refer to this step-by-step guide on how to deploy DeepSeek-R1 models using Amazon Bedrock Custom Model Import. To learn more, visit Import a customized model into Amazon Bedrock.

4. DeepSeek-R1-Distill models using AWS Trainium and AWS Inferentia
AWS Deep Learning AMIs (DLAMI) provides customized machine images that you can use for deep learning in a variety of Amazon EC2 instances, from a small CPU-only instance to the latest high-powered multi-GPU instances. You can deploy the DeepSeek-R1-Distill models on AWS Trainuim1 or AWS Inferentia2 instances to get the best price-performance.

To get started, go to Amazon EC2 console and launch a trn1.32xlarge EC2 instance with the Neuron Multi Framework DLAMI called Deep Learning AMI Neuron (Ubuntu 22.04).

Once you have connected to your launched ec2 instance, install vLLM, an open-source tool to serve Large Language Models (LLMs) and download the DeepSeek-R1-Distill model from Hugging Face. You can deploy the model using vLLM and invoke the model server.

To learn more, refer to this step-by-step guide on how to deploy DeepSeek-R1-Distill Llama models on AWS Inferentia and Trainium.

You can also visit the DeepSeek-R1-Distill-Llama-8B or deepseek-ai/DeepSeek-R1-Distill-Llama-70B model cards on Hugging Face. Choose Deploy and then Amazon SageMaker. From the AWS Inferentia and Trainium tab, copy the example code for deploy DeepSeek-R1-Distill Llama models.

Since the release of DeepSeek-R1, various guides of its deployment for Amazon EC2 and Amazon Elastic Kubernetes Service (Amazon EKS) have been posted. Here is some additional material for you to check out:

Things to know
Here are a few important things to know.

  • Pricing – For publicly available models like DeepSeek-R1, you are charged only the infrastructure price based on inference instance hours you select for Amazon Bedrock Markeplace, Amazon SageMaker JumpStart, and Amazon EC2. For the Bedrock Custom Model Import, you are only charged for model inference, based on the number of copies of your custom model is active, billed in 5-minute windows. To learn more, check out the Amazon Bedrock Pricing, Amazon SageMaker AI Pricing, and Amazon EC2 Pricing pages.
  • Data security – You can use enterprise-grade security features in Amazon Bedrock and Amazon SageMaker to help you make your data and applications secure and private. This means your data is not shared with model providers, and is not used to improve the models. This applies to all models—proprietary and publicly available—like DeepSeek-R1 models on Amazon Bedrock and Amazon SageMaker. To learn more, visit Amazon Bedrock Security and Privacy and Security in Amazon SageMaker AI.

Now available
DeepSeek-R1 is generally available today in Amazon Bedrock Marketplace and Amazon SageMaker JumpStart. You can also use DeepSeek-R1-Distill models using Amazon Bedrock Custom Model Import and Amazon EC2 instances with AWS Trainum and Inferentia chips.

Give DeepSeek-R1 models a try today in the Amazon Bedrock console, Amazon SageMaker AI console, and Amazon EC2 console, and send feedback to AWS re:Post for Amazon Bedrock and AWS re:Post for SageMaker AI or through your usual AWS Support contacts.

Channy