Security updates for Wednesday

Post Syndicated from daroc original https://lwn.net/Articles/1048195/

Security updates have been issued by AlmaLinux (bind, binutils, delve and golang, expat, firefox, haproxy, kernel, libsoup3, libssh, libtiff, openssh, openssl, pam, podman, python-kdcproxy, shadow-utils, squid, thunderbird, vim, xorg-x11-server-Xwayland, and zziplib), Debian (cups-filters, libsdl2, linux-6.1, net-snmp, pdfminer, rails, and tryton-sao), Fedora (chromium, docker-buildkit, docker-buildx, and sudo-rs), Gentoo (librnp), Mageia (webkit2), SUSE (amazon-ssm-agent, buildah, curl, dpdk, fontforge-20251009, kernel, libIex-3_4-33, librnp0, python311, rclone, and sssd), and Ubuntu (linux, linux-aws, linux-aws-6.8, linux-ibm, linux-lowlatency, linux-lowlatency-hwe-6.8, linux-nvidia, linux-nvidia-6.8, linux-nvidia-lowlatency, linux-oracle, linux-aws-6.14, linux-oracle-6.14, linux-aws-fips, linux-fips, linux-gcp-fips, linux-realtime, linux-realtime-6.8, mupdf, openjdk-17, openjdk-8, and openjdk-lts).

Huawei and Chinese Surveillance

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/11/huawei-and-chinese-surveillance.html

This quote is from House of Huawei: The Secret History of China’s Most Powerful Company.

“Long before anyone had heard of Ren Zhengfei or Huawei, Wan Runnan had been China’s star entrepreneur in the 1980s, with his company, the Stone Group, touted as “China’s IBM.” Wan had believed that economic change could lead to political change. He had thrown his support behind the pro-democracy protesters in 1989. As a result, he had to flee to France, with an arrest warrant hanging over his head. He was never able to return home. Now, decades later and in failing health in Paris, Wan recalled something that had happened one day in the late 1980s, when he was still living in Beijing.

Local officials had invited him to dinner.

This was unusual. He was usually the one to invite officials to dine, so as to curry favor with the show of hospitality. Over the meal, the officials told Wan that the Ministry of State Security was going to send agents to work undercover at his company in positions dealing with international relations. The officials cast the move to embed these minders as an act of protection for Wan and the company’s other executives, a security measure that would keep them from stumbling into unseen risks in their dealings with foreigners. “You have a lot of international business, which raises security issues for you. There are situations that you don’t understand,” Wan recalled the officials telling him. “They said, ‘We are sending some people over. You can just treat them like regular employees.’”

Wan said he knew that around this time, state intelligence also contacted other tech companies in Beijing with the same request. He couldn’t say what the situation was for Huawei, which was still a little startup far to the south in Shenzhen, not yet on anyone’s radar. But Wan said he didn’t believe that Huawei would have been able to escape similar demands. “That is a certainty,” he said.

“Telecommunications is an industry that has to do with keeping control of a nation’s lifeline…and actually in any system of communications, there’s a back-end platform that could be used for eavesdropping.”

It was a rare moment of an executive lifting the cone of silence surrounding the MSS’s relationship with China’s high-tech industry. It was rare, in fact, in any country. Around the world, such spying operations rank among governments’ closest-held secrets. When Edward Snowden had exposed the NSA’s operations abroad, he’d ended up in exile in Russia. Wan, too, might have risked arrest had he still been living in China.

Here are two book reviews.

Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

Post Syndicated from Grab Tech original https://engineering.grab.com/real-time-data-quality-monitoring

Introduction

In today’s data-driven landscape, monitoring data quality has become a critical need for ensuring reliable and efficient data usage across domains. High-quality data is the backbone of AI innovation, driving efficiency and unlocking new opportunities. As decentralized data ownership grows, the ability to effectively monitor data quality is essential for maintaining reliability in data systems.

Kafka streams, as a vital component of real-time data processing, play a significant role in this ecosystem. However, unreliable data within Kafka streams can lead to errors and inefficiencies for downstream users, and monitoring the quality of data within these streams has always been a challenge. This blog introduces a solution that empowers stream users to define a data contract, specifying the rules that Kafka stream data must adhere to. By leveraging this user-defined data contract, the solution performs automated real-time data quality checks, identifies problematic data as it occurs, and promptly notifies stream owners. This ensures timely action, enabling effective monitoring and management of Kafka stream data quality while supporting the broader goals of data mesh and AI-driven innovation.

Problem statement

In the past, monitoring Kafka stream data processing lacked an effective solution for data quality validation. This limitation made it challenging to identify bad data, notify users in a timely manner, and prevent the cascading impact on downstream users from further escalating.

Challenges in syntactic and semantic issue identification:

  • Syntactic issues: Refers to schema mismatches between producers and consumers, which can lead to deserialization errors. While schema backward compatibility can be validated upon schema evolution, there are scenarios where the actual data in the Kafka topic does not align with the defined schema. For example, this can occur when a rogue Kafka producer is not using the expected schema for a given Kafka topic. Identifying the specific fields causing these syntactic issues is a typical challenge.
  • Semantic issues: Refers to inconsistencies or misalignments between producers and consumers about the expected pattern or significance of each field. Unlike Kafka stream schemas, which act as a data structure contract between producers and consumers, there is no existing framework for stakeholders to define and enforce field-level semantic rules, for example, the expected length or pattern of an identifier.

Timeliness challenge in data quality monitoring: There is no real-time mechanism to automatically validate data against predefined rules, timely identify quality issues, and promptly alert stream stakeholders. Without real-time stream validation, data quality issues can sometimes persist for periods of time, impacting various online and offline downstream systems before being discovered.

Observability challenge for troubleshooting bad data: Even when problematic data is identified, stream users face difficulties in pinpointing the exact “poison data” and understanding which fields are incompatible with the schema or violate semantic rules. This lack of visibility complicates Root Cause Analysis and resolution efforts.

Solution

Our Coban platform offers a standardized data quality test and observability solution at the platform level, consisting of the following components:

  • Data Contract Definition: Enables Kafka stream stakeholders to define contracts that include schema agreements, semantic rules that Kafka topic data must comply with, and Kafka stream ownership details for alerting and notifications.
  • Automated Test Execution: Provides a long running Test Runner to automatically execute real-time tests based on the defined contract.
  • Real-time Data Quality Issue Identification: Detects data issues at both syntactic and semantic levels in real-time.
  • Alerts and Result Observability: Alerts users, simplifying observation of data quality issues via the platform.

Architecture details

The solution includes three components: Data Contract Definition, Test Execution & Data Quality Issue Identification, and Result Observability as shown in the architecture diagram in figure 1. All mentions of “Flow” from here onwards refer to the corresponding processes illustrated in figure 1.

Figure 1. Real-time Kafka Stream Data Quality Monitoring Architecture diagram.

Data Contract Definition

The Coban Platform streamlines the process of defining Kafka stream data contracts, serving as a formal agreement among Kafka stream stakeholders. This includes the following components:

  • Kafka Stream Schema: Represents the schema used by the Kafka topic under test and helps the Test Runner to validate schema compatibility across data streams (Flow 1.1).
  • Kafka Stream Configuration: Encompasses essential configurations such as the endpoint and topic name, which the platform automatically populates (Flow 1.2).
  • Observability Metadata: Provides contact information for notifying Kafka stream stakeholders about data quality issues and includes alert configurations for monitoring (Flow 1.3).
  • Kafka Stream Semantic Test Rules: Empowers users to define intuitive semantic test rules at the field level. These rules include checks for string patterns, number ranges, constant values, etc. (Flow 1.5).
  • LLM-Based Semantic Test Rules Recommendation: Defining dozens if not hundreds of field-specific test rules can overwhelm users. To simplify this process, the Coban Platform uses LLM-based recommendations to predict semantic test rules using provided Kafka stream schemas and anonymized sample data (Flow 1.4). This feature helps users set up semantic rules efficiently, as demonstrated in the sample UI in figure 2.
Figure 2. Sample UI showcasing LLM-based Kafka stream schema field-level semantic test rules. Note that the data shown is entirely fictional.

Data Contract Transformation

Once defined, the Coban Platform’s transformation engine converts the data contract into configurations that the Test Runner can interpret (Flow 2.1). This transformation process includes:

  • Kafka Stream Schema: Translates the schema defined in the data contract into a schema reference that the Test Runner can parse.
  • Kafka Stream Configuration: Sets up the Kafka stream as a source for the Test Runner.
  • Observability metadata: Sets contact information as configurations of the Test Runner.
  • Kafka Stream Semantic Test Rules: Transforms human-readable semantic test rules into an inverse SQL query to capture the data that violates the defined rules.
Figure 3. Illustration of semantic test rules being converted from human-readable formats into inverse SQL queries.

Test Execution & Data Quality Issue Identification

Once the Test Configuration Transformation Engine generates the Test Runner configuration (Flow 2.1), the platform automatically deploys the Test Runner.

Test Runner

The Test Runner utilises FlinkSQL as the compute engine to execute the tests. FlinkSQL was selected for its flexibility in defining test rules as straightforward SQL statements, enabling our platform to efficiently convert data contracts into enforceable rules.

Test Execution Workflow And Problematic Data Identification

FlinkSQL consumes data from the Kafka topic under test (Flow 2.2) using its own consumer group, ensuring it doesn’t impact other consumers. It runs the inverse SQL query (Flow 2.3) to identify any data that violates the semantic rules or that is syntactically incorrect in the first place. Test Runner captures such data, packages it into a data quality issue event enriched with a test summary, the total count of bad records, and sample bad data, and publishes it to a dedicated Kafka topic (Flow 3.2). Additionally, the platform sinks all such data quality events to an AWS S3 bucket (Flow 3.1) to enable deeper observability and analysis.

Result Observability

Grab’s in-house data quality observability platform, Genchi, consumes problematic data captured by the Test Runner (Flow 3.3).

Alerting

Genchi sends Slack notifications (Flow 3.5) to stream owners specified in the data contract observability metadata. These notifications include detailed information about stream issues, such as links to sample data in Coban UI, observed windows, counts of bad records, and other relevant details.

Figure 4. Sample Slack notifications

Observability

Users can access the Coban UI (Flow 3.4), displaying Kafka stream test rules and sample bad records, highlighting fields and values that violate rules.

Figure 5. In this Sample Test Result, the highlighted fields indicate violations of the semantic test rules.

Impact

Since its deployment earlier this year, the solution has enabled Kafka stream users to define contracts with syntactic and semantic rules, automate test execution, and alert users when problematic data is detected, prompting timely action. It has been actively monitoring data quality across 100+ critical Kafka topics. The solution offers the capability to immediately identify and halt the propagation of invalid data across multiple streams.

Conclusion

We implemented and rolled out a solution to assist Grab engineers in effectively monitoring data quality in their Kafka streams. This solution empowers them to establish syntactic and semantic tests for their data. Our platform’s automatic testing feature enables real-time tracking of data quality, with instant alerts for any discrepancies. Additionally, we provide detailed visibility into test results, facilitating the easy identification of specific data fields that violate the rules. This accelerates the process of diagnosing and resolving issues, allowing users to swiftly address production data challenges.

What’s next

While our current solution emphasizes monitoring the quality of Kafka streaming data, further exploration will focus on tracing producers to pinpoint the origin of problematic data, as well as enabling more advanced semantic tests such as cross-field validations. Additionally, we aim to expand monitoring capabilities to cover broader aspects like data completeness and freshness, and integrate with Gable AI to detect Data Transfer Object (DTO) changes and semantic regressions in Go producers upon committing code to the Git repository. These enhancements will pave the way for a more robust, multidimensional data quality testing solution across a wider range.

References

Driving Data Quality with Data Contracts: A Comprehensive Guide to Building Reliable, Trusted, and Effective Data Platforms by Andrew Jones

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility and digital financial services sectors. Serving over 800 cities in eight Southeast Asian countries, Grab enables millions of people everyday to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line – we aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Разследване на Валя Ахчиева „Елените” – отмъщението на реката

Post Syndicated from Екип на Биволъ original https://bivol.bg/elenite-apocalips.html

сряда 26 ноември 2025


Трагедията, която не е природно бедствие Ужасът от 3 октомври във ваканционно селище „Елените“ не е просто „природно бедствие“, а резултат от едно дългогодишно, внимателно подредено престъпление срещу закона и…

Материалът <span style='color:#ff0000;font-size:12px;'>Разследване на Валя Ахчиева</span> <BR> <H1 class='post-title single-post-title entry-title'>„Елените” – отмъщението на реката</H1> е публикуван за пръв път на Bivol!.

Orchestrating data processing tasks with a serverless visual workflow in Amazon SageMaker Unified Studio

Post Syndicated from Suba Palanisamy original https://aws.amazon.com/blogs/big-data/orchestrating-data-processing-tasks-with-a-serverless-visual-workflow-in-amazon-sagemaker-unified-studio/

Automation of data processing and data integration tasks is essential for data engineers and analysts to maintain up-to-date data pipelines and reports. Amazon SageMaker Unified Studio is a single data and AI development environment where you can find and access the data in your organization and act on it using the ideal tools for your use case. SageMaker Unified Studio offers multiple ways to integrate with data through its editorial tools, including Visual ETL, Query Editor, and JupyterLab builders.

Recently, AWS launched the visual workflow experience in SageMaker Unified Studio IAM-based domains. With visual workflows, you don’t need to code Python DAGs manually or have deep expertise in Apache Airflow. Instead, you can visually define orchestration workflows through an intuitive drag-and-drop interface in SageMaker Unified Studio. The visual definition is automatically converted to workflow definitions that leverage Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless, providing enterprise-grade orchestration capabilities with a simplified user experience.

In this post, we show how to use the new visual workflow experience in SageMaker Unified Studio IAM-based domains to orchestrate an end-to-end machine learning workflow. The workflow ingests weather data, applies transformations, and generates predictions—all through a single, intuitive interface, without writing any orchestration code.

For more details on Amazon MWAA Serverless, see Introducing Amazon MWAA Serverless.

Example use case

To demonstrate how SageMaker Unified Studio simplifies end-to-end workflow orchestration, let’s walk through a real-world scenario from agricultural analytics. The following diagram shows a weather data processing workflow that we will orchestrate using the visual workflow experience in SageMaker Unified Studio.

A regional agricultural extension office collects hourly weather data from multiple stations across farming communities. Their goal is to analyze this data and provide farmers with actionable insights into weather patterns and their impact on crop conditions. To achieve this, the team built a ML–powered analytics workflow using SageMaker Unified Studio to automate the processing of incoming weather data and predict irrigation needs.

In this walkthrough, we demonstrate how the visual workflow experience in Unified Studio can orchestrate an end-to-end data pipeline that:

  • Monitors and ingests hourly weather data from Amazon Simple Storage Service (Amazon S3)
  • Transforms raw weather measurements using Visual ETL jobs (type casting, SQL operations, and data cleansing)
  • Generates seasonal irrigation predictions and crop impact insights using JupyterLab notebooks

Whenever new weather data arrives, the workflow automatically routes it through a series of transformation steps, and produces ready-to-use insights—all visually orchestrated in SageMaker Unified Studio with no custom orchestration code required.

Prerequisites

Before you begin, complete the following steps:

  1. Signup for an AWS account and create a user with administrative access using the setup guide.
  2. Setup your SageMaker Unified Studio IAM-based domain:
    1. Navigate to the Amazon SageMaker console and use the Region selector in the top navigation bar to choose the appropriate AWS Region.
    2. On the Amazon SageMaker home page, choose Get started.
    3. For Project data access, choose to Auto-create a new role with admin permissions.
    4. Select the checkbox for S3 table integration with AWS Analytics services, for Data encryption choose Use AWS owned key, and then Set up.
  3. Go back to the Amazon SageMaker home page and choose Open to access the SageMaker Unified Studio experience.
  4. From the SageMaker Studio UI you can access the project in the SageMaker Unified Studio IAM-based domain. This project curates all assets accessible through the designated Execution IAM role.

Workflow implementation steps

In this section, we use Amazon SageMaker Studio to create an end-to-end visual workflow in IAM-based domain.

Step 1: Set up data storage and import weather dataset

First, we’ll prepare the Amazon S3 storage locations for raw and processed data:

  1. Download this weather dataset file to your local environment.
  2. From the left menu of the project, choose Files. Under Shared, create two new folders raw_data and processed_data.
  3. Upload the weather dataset file downloaded locally into raw_data folder.

Step 2: Create the weather data transformation job using Visual ETL

Next, create a Visual ETL job to transform the raw weather data through type casting, SQL transformations, and data cleansing:

  1. From the left menu, under Data Analytics, choose Visual ETL and Create Visual Job.
  2. Choose the + sign, and under Data sources, choose Amazon S3.
  3. For the Amazon S3 node settings, choose the following:
    • S3 URI: Choose Browse S3 and Select
    • Delimiter: ,
    • Multiline: Disabled
    • Header: Enabled
    • Infer schema: Disabled
    • Recursive file lookup: Disabled

  4. Choose the + sign next to the Amazon S3 box to add another node, under Transforms select Change columns.
  5. Connect the Amazon S3 node to the change columns node.
  6. Select the Change columns node to open the configuration window.
    • Choose Add type cast. Select temperature_2m (¬∞C) as the source column and add temperature_celsius as the target column. Select float as the Type.
    • Select precipitation (mm) as the source column and add Precipitation_mm as the target column. Select float as the Type.
    • Select rain (mm) as the source column and add Rain_mm as the target column. Select float as the Type.
    • Select windspeed_10m (km/h) as the source column and add windspeed as the target column. Select float as the Type.
    • Close the configuration window.

  7. Choose the + sign to add another node, under Transforms select SQL query . In the configuration window, paste in the following SQL statement: 
    SELECT 
    MAKE_TIMESTAMP(2016, 1, day_of_year, hour, 0, 0) as timestamp,
    (temperature_celsius * 9 / 5) + 32 as temp_f,
    rain_mm * 25.4 as rain_inches,
    windspeed
FROM {myDataSource}

  8. Choose the + sign to add another node, under Data targets, choose Amazon S3 and provide the following options:
    • S3 URI: Choose Browse S3 and select the processed_data folder created in Step 1.
    • Format: CSV
    • Update catalog: true
    • Database: sagemaker_sample_db
    • Table: weather_data
    • Include header: true
    • Ouput to a single file: false

  9. Connect the nodes to create a complete job.
  10. Save the Visual ETL and name it DataProcessing.

Step 3: Create the analysis and prediction notebook using JupyterLab

Now, we’ll set up the JupyterLab notebook that performs seasonal irrigation analysis and crop impact predictions based on temperature, rainfall, and wind speed patterns.Complete the following steps:

  1. Download the Crop Irrigation Prediction Python notebook to your local environment.
  2. In the SageMaker Unified Studio, from the left menu, choose JupyterLab. Wait for a few seconds for JupyterLab to be set up if you are trying for the first time.
  3. Upload CropIrrigationPrediction.ipynb using the upload files option.
  4. Review the notebook code to understand how it processes the weather data and generates irrigation predictions.

Step 4: Orchestrate the workflow

Finally, we will use the visual workflow to orchestrate tasks. With visual workflows, you can define a collection of tasks organized as a directed acyclic graph (DAG) that can run on a user-defined schedule.

  1. Choose Workflows from the left menu.
  2. Choose Create new Workflow.
  3. Rename the workflow to WeatherDataProcessingOrchestration.
  4. Create S3 task for monitoring and ingesting raw weather data:
    1. Choose the + sign, then choose S3 Key Sensor.
    2. Select S3-task to open the configuration window.
    3. For Bucket key choose Browse S3 and choose the synthetic_weather_hourly_data.csv file from the shared/raw_data S3 folder.



  5. Create a Glue task to transform the weather data:
    1. Choose the + Sign and add Data Processing Job / Glue Job Operator.
    2. Select Glue-task node to open the configuration window. For Operation type select Choose an existing Glue job.
    3. For Job name, choose Browse Jobs and select DataProcessing (this is the visual ETL job we created in the previous step.


  6. Choose the + sign and add SageMaker Unified Studio Jupyter Notebook Operator.
  7. Select the Notebook-task to open the configuration window. For Source, choose Browse Files and choose CropIrrigationPrediction.ipynb.

  8. Connect the tasks to create the complete workflow.
  9. Review the Workflow settings and choose Save.
    1. Provide a workflow description, “Workflow for Weather Data Processing”
    2. For Trigger, choose Manual only, because in this example you will trigger the workflow manually. You can also configure the workflow to trigger automatically on a schedule or disable it from running

Step 5: Execute and monitor the workflow

To run your workflow, complete the following steps:

  1. Choose Run to trigger workflow execution.
  2. Choose View runs to see the running workflow.
  3. Choose the Run ID for detailed logs on the execution.
  4. When the run is complete, you can review the task logs by choosing the Task ID.

The model’s output is written to the S3 processed data output folder. You can review the crop irrigation prediction results to verify they reflect realistic weather patterns and field conditions. If any results appear unexpected or unclear, examine the upstream transformation steps or adjust the notebook logic to refine the outputs.

Clean up

To avoid incurring future charges, clean up the resources you created during this walkthrough. Leaving these resources running may result in ongoing costs for storage and compute.To clean up your resources:

  1. On the workflows page, select your workflow, and under Actions, choose Delete workflow.
  2. In Visual ETL, select your weather data transformation flow, and under Actions, choose Delete job.
  3. In Query Editor, use the three dots next to the name of the table weather_data and choose Drop table.
  4. In JupyterLab, in the File Browser sidebar, choose (right-click) your notebook and choose Delete.
  5. In Files, choose the folder raw_data and under Actions, choose Delete. Repeat the steps for the folders processed_data and output.

Conclusion

In this post, you learned how you can use the visual workflow experience in Amazon SageMaker Unified Studio to build end-to-end data processing pipelines through an intuitive, no-code interface. This experience removes the need to write orchestration logic manually while still offering production-grade reliability and scalability powered by Amazon MWAA Serverless. Whether you’re processing weather data for agricultural insights or building more complex machine learning pipelines, the visual workflow experience accelerates development and makes workflow automation accessible to data engineers, analysts, and data scientists alike.As organizations increasingly rely on automated data pipelines to drive business decisions, the visual workflow experience provides the perfect balance of simplicity and power. We encourage you to explore this new capability in Amazon SageMaker Unified Studio and discover how it can transform your data processing workflows.

To learn more, visit the Amazon SageMaker Unified Studio page.

About the authors

Suba Palanisamy

Suba Palanisamy

Suba is an Enterprise Support Lead, helping customers achieve operational excellence on AWS. Suba is passionate about all things data and analytics. She enjoys traveling with her family and playing board games.

Vinod Jayendra

Vinod Jayendra

Vinod is an Enterprise Support Lead in ISV accounts at Amazon Web Services, where he helps customers in solving their architectural, operational, and cost optimization challenges. With a particular focus on Serverless & Analytics technologies, he draws from his extensive background in application development to deliver top-tier solutions. Beyond work, he finds joy in quality family time, embarking on biking adventures, and coaching youth sports team.

Kamen Sharlandjiev

Kamen Sharlandjiev

Kamen is a Senior Worldwide Specialist SA, Big Data expert. He’s on a mission to make life easier for customers who are facing complex data integration and orchestration challenges. His secret weapon? Fully managed AWS services that can get the job done with minimal effort. Follow Kamen on LinkedIn to keep up to date with the latest MWAA and AWS Glue features and news!

Yuhang Huang

Yuhang Huang

Yuhang is a Software Development Manager on the Amazon SageMaker Unified Studio team. He leads the engineering team to design, build, and operate scheduling and orchestration capabilities in SageMaker Unified Studio. In his free time, he enjoys playing tennis.

Vasudevan Venkataramanan

Vasudevan Venkataramanan

Vasudevan is a Senior Software Engineer on the Amazon SageMaker Unified Studio team. He is responsible for technical direction of scheduling and orchestration within SageMaker Unified Studio. Outside of his professional work, he enjoys spending time with his kid and playing pickleball and cricket.

Gal Heyne

Gal Heyne

Gal is a Senior Technical Product Manager for AWS Analytics services with a strong focus on AI/ML and data engineering. She is passionate about developing a deep understanding of customers’ business needs and collaborating with engineers to design simple-to-use data products.

AWS Secrets Manager launches Managed External Secrets for Third-Party Credentials

Post Syndicated from Rohit Panjala original https://aws.amazon.com/blogs/security/aws-secrets-manager-launches-managed-external-secrets-for-third-party-credentials/

Although AWS Secrets Manager excels at managing the lifecycle of Amazon Web Services (AWS) secrets, managing credentials from third-party software providers presents unique challenges for organizations as they scale usage of their cloud applications. Organizations using multiple third-party services frequently develop different security approaches for each provider’s credentials because there hasn’t been a standardized way to manage them. When storing these third-party credentials in Secrets Manager, organizations frequently maintain additional metadata within secret values to facilitate service connections. This approach requires updating entire secret values when metadata changes and implementing provider-specific secret rotation processes that are manual and time consuming. Organizations looking to automate secret rotation typically develop custom functions tailored to each third-party software provider, requiring specialized knowledge of both third-party and AWS systems.

To help customers streamline third-party secrets management, we’re introducing a new feature in AWS Secrets Manager called managed external secrets. In this post, we explore how this new feature simplifies the management and rotation of third-party software credentials while maintaining security best practices.

Introducing managed external secrets

AWS Secrets Manager has a proven track record of helping customers secure and manage secrets for AWS services such as Amazon Relational Database Service (Amazon RDS) or Amazon DocumentDB through managed rotation capabilities. Building on this success, Secrets Manager now introduces managed external secrets, a new secret type that extends this same seamless experience to third-party software applications like Salesforce, simplifying secret management challenges through standardized formats and automated rotation.

You can use this capability to store secrets vended by third-party software providers in predefined formats. These formats were developed in collaboration with trusted integration partners to define both the secret structure and required metadata for rotation, eliminating the need for you to define your own custom storage strategies. Managed external secrets also provides automated rotation by directly integrating with software providers. With no rotation functions to maintain, you can reduce your operational overhead while benefiting from essential security controls, including fine-grained permissions management using AWS Identity and Access Management (IAM), secret access monitoring through Amazon CloudWatch and AWS CloudTrail, and automated secret-specific threat detection through Amazon GuardDuty. Moreover, you can implement centralized and consistent secret management practices across both AWS and third-party secrets from a single service, eliminating the need to operate multiple secrets management solutions at your organization. Managed external secrets follows standard Secrets Manager pricing, with no additional cost for using this new secret type.

Prerequisites

To create a managed external secret, you need an active AWS account with appropriate access to Secrets Manager. The account must have sufficient permissions to create and manage secrets, including the ability to access the AWS Management Console or programmatic access through the AWS Command Line Interface (AWS CLI) or AWS SDKs. At minimum, you need IAM permissions for the following actions: secretsmanager:DescribeSecret, secretsmanager:GetSecretValue, secretsmanager:UpdateSecret, and secretsmanager:UpdateSecretVersionStage.

You must have valid credentials and necessary access permissions for the third-party software provider you plan to have AWS manage secrets for.

For secret encryption, you must decide whether to use an AWS managed AWS Key Management Service (AWS KMS) key or a customer managed KMS key. For customer managed keys, make sure you have the necessary key policies configured. The AWS KMS key policy should allow Secrets Manager to use the key for encryption and decryption operations.

Create a managed external secret

Today, managed external secrets supports three integration partners: Salesforce, Snowflake, and BigID. Secrets Manager will continue to expand its partner list and more third-party software providers will be added over time. For the latest list, refer to Integration Partners.

To create a managed external secret, follow the steps in the following sections.

Note: This example demonstrates the steps for retrieving Salesforce External Client App Credentials, but a similar process can be followed for other third-party vendor credentials integrated with Secrets Manager.

To select secret type and add details:

  1. Go to the Secrets Manager service in the AWS Management Console and choose Store a new secret.
  2. Under Secret type, select Managed external secret.
  3. In the AWS Secrets Manager integrated third-party vendor credential section, select your provider from the available options. For this walkthrough, we select Salesforce External Client App Credential.
  4. Enter your configurations in the Salesforce External Client App Credential secret details section. The Salesforce External Client App credentials consist of several key components:
    1. The Consumer key (client ID), which serves as the credential identifier for OAuth 2.0. You can retrieve the consumer key directly from the Salesforce External Client App Manager OAuth settings.
    2. The Consumer secret (client secret), which functions as the private password for OAuth 2.0 authentication. You can retrieve the consumer secret directly from the Salesforce External Client App Manager OAuth settings.
    3. The Base URI, which is your Salesforce org’s base URL (formatted as https://MyDomainName.my.salesforce.com), is used to interact with Salesforce APIs.
    4. The App ID, which identifies your Salesforce External Client Apps (ECAs) and can be retrieved by calling the Salesforce OAuth usage endpoint.
    5. The Consumer ID, which identifies your Salesforce ECA, can be retrieved by calling the Salesforce OAuth credentials by App ID endpoint. For a list of commands, refer to Stage, Rotate, and Delete OAuth Credentials for an External Client App in the Salesforce documentation.
  5. Select the Encryption key from the dropdown menu. You can use an AWS managed KMS key or a customer managed KMS key.
  6. Choose Next.
Figure 1: Choose secret type

Figure 1: Choose secret type

To configure a secret:

  1. In this section, you need to provide information for your secret’s configuration.
  2. In Secret name, enter a descriptive name and optionally enter a detailed Description that helps identify the secret’s purpose and usage. You also have additional configuration choices available: you can attach Tags for better resource organization, set specific Resource permissions to control access, and select Replicate secret for multi-Region resilience.
  3. Choose Next.
Figure 2: Configure secret

Figure 2: Configure secret

To configure rotation and permissions (optional):

In the optional Configure rotation step, the new secret configuration introduces two key sections focused on metadata management, which are stored separately from the secret value itself.

  1. Under Rotation metadata, specify the API version your Salesforce app is using. To find the API version, refer to List Available REST API Versions in the Salesforce documentation. Note: The minimum version needed is v65.0.
  2. Select an Admin secret ARN, which contains the administrative OAuth credentials that are used to rotate the Salesforce client secret.
  3. In the Service permissions for secret rotation section, Secrets Manager automatically creates a role with necessary permissions to rotate your secret values. These default permissions are transparently displayed in the interface for review when you choose view permission details. You can deselect the default permissions for more granular control over secret rotation management.
  4. Choose Next.
Figure 3: Configure rotation

Figure 3: Configure rotation

To review:

In the final step, you’ll be presented with a summary of your secret’s configuration. On the Review page, you can verify parameters before proceeding with secret creation.

After confirming that the configurations are correct, choose Store to complete the process and create your secret with the specified settings.

Figure 4: Review

Figure 4: Review

After successful creation, your secret will appear on the Secrets tab. You can view, manage, and monitor aspects of your secret, including its configuration, rotation status, and permissions. After creation, review your secret configuration, including encryption settings and resource policies for cross-account access, and examine the sample code provided for different AWS SDKs to integrate secret retrieval into your applications. The Secrets tab provides an overview of your secrets, allowing for central management of secrets. Select your secret to view Secret details.

Figure 5: View secret details

Figure 5: View secret details

Your managed external secret has been successfully created in Secrets Manager. You can access and manage this secret through the Secrets Manager console or programmatically using AWS APIs.

Onboard as an integration partner with Secrets Manager

With the new managed external secret type, third-party software providers can integrate with Secrets Manager and offer their customers a programmatic way to securely manage secrets vended by their applications on AWS. This integration provides their customers with a centralized solution for managing both the lifecycle of AWS and third-party secrets, including automatic rotation capabilities from the moment of secret creation. Software providers like Salesforce are already using this capability.

“At Salesforce, we believe security shouldn’t be a barrier to innovation, it should be an enabler. Our partnership with AWS on managed external secrets represents security-by-default in action, removing operational burden from our customers while delivering enterprise-grade protection. With AWS Secrets Manager now extending to partners and automated zero-touch rotation eliminating human risk, we’re setting a new industry standard where secure credentials become seamless without specialized expertise or additional costs.” — Jay Hurst, Sr. Vice President, Product Management at Salesforce

There is no additional cost to onboard with Secrets Manager as an integration partner. To get started, partners must follow the process listed on the partner onboarding guide. If you have questions about becoming an integration partner, contact our team at [email protected] with Subject: [Partner Name] Onboarding request.

Conclusion

In this post, we introduced managed external secrets, a new secret type in Secrets Manager that addresses the challenges of securely managing the lifecycle of third-party secrets through predefined formats and automated rotation. By eliminating the need to define custom storage strategies and develop complex rotation functions, you can now consistently manage your secrets—whether from AWS services, custom applications, or third-party providers—from a single service. Managed external secrets provide the same security features as standard Secrets Manager secrets, including fine-grained permissions management, observability, and compliance controls, while adding built-in integration with trusted partners at no additional cost.

To get started, refer to the technical documentation. For information on migrating your existing partner secrets to managed external secrets, see Migrating existing secrets. The feature is available in all AWS Regions where AWS Secrets Manager is available. For a list of Regions where Secrets Manager is available, see the AWS Region table. If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on Secrets Manager re:Post or contact AWS Support.

Rohit Panjala

Rohit is a Security Specialist at AWS, focused on data protection and cryptography services. He is responsible for developing and implementing go-to-market (GTM) strategies and driving customer and partner adoptions for AWS data protection services on a global scale. Before joining AWS, he worked in security product management at IBM and electrical engineering roles. He holds a BS in engineering from The Ohio State University.

Rochak Karki

Rochak is a Security Specialist Solutions Architect at AWS, focusing on threat detection, incident response, and data protection to help customers build secure environments. Rochak is a US Army veteran and holds a BS in engineering from the University of Wyoming. Outside of work, he enjoys spending time with family and friends, hiking, and traveling.

Node.js 24 runtime now available in AWS Lambda

Post Syndicated from Andrea Amorosi original https://aws.amazon.com/blogs/compute/node-js-24-runtime-now-available-in-aws-lambda/

You can now develop AWS Lambda functions using Node.js 24, either as a managed runtime or using the container base image. Node.js 24 is in active LTS status and ready for production use. It is expected to be supported with security patches and bugfixes until April 2028.

The Lambda runtime for Node.js 24 includes a new implementation of the Runtime Interface Client (RIC), which integrates your functions code with the Lambda service. Written in TypeScript, the new RIC streamlines and simplifies Node.js support in Lambda, removing several legacy features. In particular, callback-based function handlers are no longer supported.

Node.js 24 includes several additions to the language, such as Explicit Resource Management, as well as changes to the runtime implementation and the standard library. With this release, Node.js developers can take advantage of these new features and enhancements when creating serverless applications on Lambda.

You can develop Node.js 24 Lambda functions using the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS SDK for JavaScript, AWS Serverless Application Model (AWS SAM), AWS Cloud Development Kit (AWS CDK), and other infrastructure as code tools. You can use Node.js 24 with Powertools for AWS Lambda (TypeScript), a developer toolkit to implement serverless best practices and increase developer velocity. Powertools includes libraries to support common tasks such as observability, AWS Systems Manager Parameter Store integration, idempotency, batch processing, and more. You can also use Node.js 24 with Lambda@Edge to customize low-latency content delivered through Amazon CloudFront.

This blog post highlights important changes to the Node.js runtime, notable Node.js language updates, and how you can use the new Node.js 24 runtime in your serverless applications.

Node.js 24 runtime changes

The Lambda Runtime for Node.js 24 includes the following changes relative to the Node.js 22 and earlier runtimes.

Removing support for callback-based function handlers

Starting with the Node.js 24 runtime, Lambda no longer supports the callback-based handler signature for asynchronous operations. Callback-based handlers take three parameters, with the third parameter a callback. For example:

export const handler = (event, context, callback) => {
    try {
        // Some processing...
        
        // Success case
        // First parameter (error) is null, second is the result
        callback(null, {
            statusCode: 200,
            body: JSON.stringify({
                message: "Operation completed successfully"
            })
        });
        
    } catch (error) {
        // Error case
        // First parameter contains the error
        callback(error);
    }
};

The modern approach to asynchronous programming in Node.js is to use the async/await pattern. Lambda introduced support for async handlers with the Node.js 8 runtime, launched in 2018. Here’s how the above function looks when using an async handler:

export const handler = async (event, context) => {
    try {
	  // Some processing
        
        return {
            statusCode: 200,
            body: JSON.stringify({
                message: "Operation completed successfully"
            })
        };
        
    } catch (error) {
        throw error;
    }
};

The Node.js 24 runtime still supports synchronous function handlers that do not use callbacks:

export const handler = (event, context) => {
    // Perform some synchronous data processing
    // Return response
    return {
        statusCode: 200,
        body: JSON.stringify(response)
    };
};

And Node.js 24 still supports response streaming, enabling more responsive applications by accelerating the time-to-first-byte:

export const handler = awslambda.streamifyResponse(async (event, responseStream, context) => {
    // Convert event to a readable stream
    const requestStream = Readable.from(Buffer.from(JSON.stringify(event)));
    // Stream the response using pipeline
    await pipeline(requestStream, responseStream);
});

This change to remove support for callback-based function handlers only affects Node.js 24 (and later) runtimes. Existing runtimes for Node.js 22 and earlier continue to support callback-based function handlers. When migrating functions that use callback-based handlers to Node.js 24, you need to modify your code to use one of the supported function handler signatures

As part of this change, context.callbackWaitsForEmptyEventLoop is removed. In addition, the previously deprecated context.succeed, context.fail, and context.done methods have also been removed. This aligns the runtime with modern Node.js patterns for clearer, more consistent error and result handling.

Harmonizing streaming and non-streaming behavior for unresolved promises

The Node.js 24 runtime also resolves a previous inconsistency in how unresolved promises were handled. Previously, Lambda would not wait for unresolved promises once the handler returns except when using response streaming. Starting with Node.js 24, the response streaming behavior is now consistent with non-streaming behavior, and Lambda no longer waits for unresolved promises once your handler returns or the response stream ends. Any background work (for example, pending timers, fetches, or queued callbacks) is not awaited implicitly. If your response depends on additional asynchronous operations, ensure you await them in your handler or integrate them into the streaming pipeline before closing the stream or returning, so the response only completes after all required work has finished.

Experimental Node.js features

Node.js enables certain experimental features by default in the upstream language releases. Such features include support for importing modules using require() in ECMAScript modules (ES modules) and automatically detecting ES vs CommonJS modules. As they are experimental, these features may be unstable or undergo breaking changes in future Node.js updates. To provide a stable experience, Lambda disables these features by default in the corresponding Lambda runtimes.

Lambda allows you to re-enable these features by adding the --experimental-require-module flag or the --experimental-detect-module flag to the NODE_OPTIONS environment variable. Enabling experimental Node.js features may affect performance and stability, and these features can change or be removed in future Node.js releases; such issues are not covered by AWS Support or the Lambda SLA.

ES modules in CloudFormation inline functions

With AWS CloudFormation inline functions, you provide your function code directly in the CloudFormation template. They’re particularly useful when deploying custom resources. With inline functions, the code filename is always index.js, which by default Node.js interprets as a CommonJS module. With the Node.js 24 runtime, you can use ES modules when authoring inline functions by passing the --experimental-detect-module flag via the NODE_OPTIONS environment variable. Previously, you needed a zip or container package to use ES modules. With Node.js 24, you can write inline functions using standard ESM syntax (import/export) and top‑level await), which simplifies small utilities and bootstrap logic without requiring a packaging step.

Node.js 24 language features

Node.js 24 introduces several language updates and features that enhance developer productivity and improve application performance.

Node.js 24 includes Undici 7, a newer version of the HTTP client that powers global ⁠fetch. This version brings performance improvements and broader protocol capabilities. Network‑heavy Lambda functions that call AWS services or external APIs can benefit from better connection management and throughput, especially when reusing clients or using HTTP/2 where supported. Most applications should work without changes, but you should validate behavior for advanced scenarios, such as custom headers or streaming bodies, and continue to define HTTP clients outside of the handler to maximize connection reuse across invocations.

The JavaScript Explicit Resource Management syntax (⁠using and ⁠await using) enables deterministic clean-up of resources when a block completes. For Lambda handlers, this makes it easier to ensure short‑lived objects, such as streams, temporary buffers, or file handles, are disposed of promptly, which reduces the risk of resource leaks across warm invocations. You should continue to define long‑lived clients, for example SDK clients or database pools, outside the handler to benefit from connection reuse, and apply explicit disposal only to resources you want to tear down at the end of each invocation.

Finally, the ⁠AsyncLocalStorage API now uses ⁠AsyncContextFrame by default, improving the performance and reliability of async context propagation. This benefits common serverless patterns such as timers, correlating logs, managing tracing IDs and request‑scoped metadata across async and await boundaries, and streams without manual parameter threading. If you already use ⁠AsyncLocalStorage‑based libraries for logging or observability, you may see lower overhead and more consistent context propagation in Node.js 24.

For a detailed overview of Node.js 24 language features, see the Node.js 24 release blog post and the Node.js 24 changelog.

Performance considerations

At launch, new Lambda runtimes receive less usage than existing established runtimes. This can result in longer cold start times due to reduced cache residency within internal Lambda sub-systems. Cold start times typically improve in the weeks following launch as usage increases. As a result, AWS recommends not drawing conclusions from side-by-side performance comparisons with other Lambda runtimes until the performance has stabilized. Since performance is highly dependent on workload, customers with performance-sensitive workloads should conduct their own testing, instead of relying on generic test benchmarks.

Builders should continue to measure and test function performance and optimize function code and configuration for any impact. To learn more about how to optimize Node.js performance in Lambda, see our blog post Optimizing Node.js dependencies in AWS Lambda.

Migration from earlier Node.js runtimes

We’ve already discussed changes that are new to the Node.js 24 runtime, such as removing support for callback-based function handlers. As a reminder, we’ll recap some previous changes for customers upgrading from older Node.js functions.

AWS SDK for JavaScript

Up until Node.js 16, Lambda’s Node.js runtimes included the AWS SDK for JavaScript version 2. This has since been superseded by the AWS SDK for JavaScript version 3, which was released in December 2024. Starting with Node.js 18, and continuing with Node.js 24, the Lambda Node.js runtimes include version 3. When upgrading from Node.js 16 or earlier runtimes and using the included version 2, you must upgrade your code to use the v3 SDK.

For optimal performance, and to have full control over your code dependencies, we recommend bundling and minifying the AWS SDK in your deployment package, rather than using the SDK included in the runtime. For more information, see Optimizing Node.js dependencies in AWS Lambda.

Amazon Linux 2023

The Node.js 24 runtime is based on the provided.al2023 runtime, which is based on the Amazon Linux 2023 minimal container image. The Amazon Linux 2023 minimal image uses microdnf as a package manager, symlinked as dnf. This replaces the yum package manager used in Node.js 18 and earlier AL2-based images. If you deploy your Lambda function as a container image, you must update your Dockerfile to use dnf instead of yum when upgrading to the Node.js 24 base image from Node.js 18 or earlier.

Learn more about the provided.al2023 runtime in the blog post Introducing the Amazon Linux 2023 runtime for AWS Lambda and the Amazon Linux 2023 launch blog post.

Using the Node.js 24 runtime in AWS Lambda

Finally, we’ll review how to configure your functions to use Node.js 24, using a range of deployment tools.

AWS Management Console

When using the AWS Lambda Console, you can choose Node.js 24.x in the Runtime dropdown when creating a function:

Creating Node.js function in the AWS Management Console

Creating Node.js function in the AWS Management Console

To update an existing Lambda function to Node.js 24, navigate to the function in the Lambda console, click Edit in the Runtime settings panel, then choose Node.js 24.x from the Runtime dropdown:

Editing Node.js function runtime

Editing Node.js function runtime

AWS Lambda container image

Change the Node.js base image version by modifying the FROM statement in your Dockerfile.

FROM public.ecr.aws/lambda/nodejs:24
# Copy function code
COPY lambda_handler.mjs ${LAMBDA_TASK_ROOT}

AWS Serverless Application Model

In AWS SAM, set the Runtime attribute to node24.x to use this version:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: nodejs24.x
      CodeUri: my_function/.
      Description: My Node.js Lambda Function

AWS SAM supports generating this template with Node.js 24 for new serverless applications using the sam init command. For more information, refer to the AWS SAM documentation.

AWS Cloud Development Kit (AWS CDK)

In AWS CDK, set the runtime attribute to Runtime.NODEJS_24_X to use this version.

import * as cdk from "aws-cdk-lib";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as path from "path";
import { Construct } from "constructs";
export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    // The code that defines your stack goes here
    // The Node.js 24 enabled Lambda Function
    const lambdaFunction = new lambda.Function(this, "node24LambdaFunction", {
      runtime: lambda.Runtime.NODEJS_24_X,
      code: lambda.Code.fromAsset(path.join(__dirname, "/../lambda")),
      handler: "index.handler",
    });
  }
}

Conclusion

AWS Lambda now supports Node.js 24 as a managed runtime and container base image. This release uses a new runtime interface client, removes support for callback-based function handlers, and includes several other changes to streamline and simplify Node.js support in Lambda.

You can build and deploy functions using Node.js 24 using the AWS Management Console, AWS CLI, AWS SDK, AWS SAM, AWS CDK, or your choice of infrastructure as code tool. You can also use the Node.js 24 container base image if you prefer to build and deploy your functions using container images.

To find more Node.js examples, use the Serverless Patterns Collection. For more serverless learning resources, visit Serverless Land

Performance benefits of new Amazon EC2 R8a memory-optimized instances

Post Syndicated from Tyler Jones original https://aws.amazon.com/blogs/compute/performance-benefits-of-new-amazon-ec2-r8a-memory-optimized-instances/

Recently we announced the availability of Amazon Elastic Compute Cloud (Amazon EC2) R8a instances, the latest addition to the AMD memory-optimized instance family. These instances are powered by the 5th Generation AMD EPYC (codename Turin) processors with a maximum frequency of 4.5 GHz. In this post I take these instances for a spin and benchmark MySQL later on, but first I discuss the top things you should know about these instances.

Notable characteristics of R8a instances

Each vCPU on an R8a instance corresponds to a physical CPU core (something we started on 7th generation AMD instances). This means that there is no simultaneous multi-threading (SMT). Each vCPU mapped to a dedicated physical core, which means that you get more predictable and consistent performance because there’s no resource sharing or potential interference between threads, which is particularly crucial for performance-sensitive workloads where consistent latency is essential. When evaluating and adopting R8a instances, make sure that you’re re-evaluating your thresholds for CPU usage. You can likely squeeze more out of each instance’s CPU without impacting any of your workload’s SLA metrics.

R8a instances feature sizes of up to 192 vCPU with 1,536 GiB RAM. The following table shows the detailed specs:

Instance size vCPU Memory (GiB) Instance storage Network bandwidth (Gbps) EBS bandwidth (Gbps)
r8a.medium 1 8 EBS Only Up to 12.5 Up to 10
r8a.large 2 16 EBS Only Up to 12.5 Up to 10
r8a.xlarge 4 32 EBS Only Up to 12.5 Up to 10
r8a.2xlarge 8 64 EBS Only Up to 15 Up to 10
r8a.4xlarge 16 128 EBS Only Up to 15 Up to 10
r8a.8xlarge 32 256 EBS Only 15 10
r8a.12xlarge 48 384 EBS Only 22.5 15
r8a.16xlarge 64 512 EBS Only 30 20
r8a.24xlarge 96 768 EBS Only 40 30
r8a.48xlarge 192 1536 EBS Only 75 60
r8a.metal-24xl 96 768 EBS Only 40 30
r8a.metal-48xl 192 1536 EBS Only 75 60

Testing MySQL performance using HammerDB

R8a instances are a great choice for MySQL databases, so I thought that would be a great place to showcase some of these instances capabilities. To test MySQL, I used a series of scripts written by my colleagues to track MySQL performance across software versions and different EC2 instances. These scripts are stored in the repro-collection repository, which is an open source, extensible framework for performance testing that addresses real-world workloads rather than micro-benchmarks. It is built to provide a performance measurement reference usable across multiple organizations, and it’s currently centered on MySQL and actively used in discussions with Linux Kernel developers and maintainers. Furthermore, it helps track any performance impacts created by code changes to MySQL. The scripts contained in this repository set up a MySQL database to be tested, and a load generator running the HammerDB benchmark.

For this benchmark I used an r6a.24xlarge instance for the load generator, and an r6a.xlarge, r7a.xlarge, and r8a.xlarge instances for the MySQL database server all deployed in the same AWS Availability Zone (AZ). I chose a single AZ setup to minimize any latency variability from crossing multiple AZs. This is not meant to be a production-like setup, and I highly recommend using multiple AZs for production workloads. Each MySQL instance was tested separately using the same HammerDB load generator. Each test was run three times, and the results were averaged across the three runs. A diagram of the architecture is shown in the following figure:

Performance testing architecture showing r6a/r7a/r8a instance types with HammerDB load generator executing 9 test runs

HammerDB overall results

R8a instances show great results in the HammerDB benchmark for MySQL databases. For HammerDB’s overall score category, R8a instances outscored R7a instances by 55% and outscored R6a instances by 74%.

Performance comparison chart showing r6a, r7a, and r8a instance scores

HammerDB transactions per minute test

R8a instances also showed a notable improvement in this category. When compared to previous generation R7a instances, R8a out performed R7a by 32%. When compared to R6a instances, R8a outperformed by 63%.

Performance comparison showing r6a (91,105), r7a (112,686), and r8a (148,478) transactions per minute

HammerDB P99 latency results

R8a instances showed improvement in P99 latency results, showing the efficiency gains driven by the new 5th Generation AMD EPYC CPUs and higher memory bandwidth. R8a shows an 14% latency reduction when compared to R7a, and a 25% latency reduction when compared to R6a.

P99 latency comparison showing decrease from 39.93ms (r6a) to 30.02ms (r8a) across instance generations

Conclusion

Built on the AWS Nitro System using sixth generation Nitro Cards, R8a instances are ideal for high performance, memory-intensive workloads, such as SQL and NoSQL databases, as demonstrated by the bench-marking shown in this post, as well as distributed web scale in-memory caches, in-memory databases, real-time big data analytics, and Electronic Design Automation (EDA) applications. R8a instances offer 12 sizes, including 2 bare metal sizes. Amazon EC2 R8a instances are SAP-certified, and providing 38% more SAPS when compared to R7a instances. If you’re still running 6th generation R6a instances, then I highly encourage you to migrate to the 8th generation instances to use their clear price performance benefits. Staying on modern infrastructure is a great way to drive down costs and provide more features for your customers, and there are clear gains to be had based on the testing shown in this post.

Start optimizing your high performance memory intensive workloads today by migrating to R8a instances. Visit the Amazon EC2 R8a instances page to learn more and get started on your upgrades to use the increased price performance of R8a instances today!

The attendee’s guide to hybrid cloud and edge computing at AWS re:Invent 2025

Post Syndicated from Rachel Zheng original https://aws.amazon.com/blogs/compute/the-attendees-guide-to-hybrid-cloud-and-edge-computing-at-aws-reinvent-2025/

AWS re:Invent 2025 returns to Las Vegas, Nevada, from December 1–5, 2025. This year, we’re offering a comprehensive lineup of sessions and booth activities to help you build resilient, performant, and scalable applications wherever you need them—in the cloud, on premises, or at the edge.

Session types

These sessions are available in the following formats. Most of the sessions are under the topic of Hybrid Cloud and Multicloud (HMC) in the event catalog. If you plan to attend in person, lightning talks and theater sessions are walk-up only. For all other session types, you can reserve your seat through the event portal (login required) or join as a walk-up based on availability.

  • Breakout sessions – Lecture-style 60-minute presentations led by AWS experts and customers.
  • Lightning talks – 20-minute content on specific topics. Each Hybrid Cloud lightning talk features a real-world customer implementation.
  • Chalk talks – 60-minute interactive sessions where AWS experts lead discussions and whiteboard in real time.
  • Code talks – 60-minute sessions featuring coding demonstrations and technical implementations.
  • Workshops – Hands-on, 120-minute sessions where you work directly with AWS services in a guided environment.
  • Theater sessions – 15-minute quick sessions on the Expo floor, typically featuring partner solutions.

Our Hybrid Cloud sessions explore how you can extend AWS infrastructure, services, and tools to distributed locations for low latency, data residency, and local processing needs. These sessions focus on AWS Local Zones, AWS Outposts, and AWS Dedicated Local Zones. We’ve curated the following sessions by theme to help you navigate re:Invent and find content most relevant to your needs.

Leadership session

HMC202 | Breakout Session | AWS wherever you need it: From the cloud to the edge

Wednesday, Dec 3 | 2:30 PM – 3:30 PM PST

Wynn | Convention Promenade | Lafite 7 | Content Hub | Turquoise Theater

Presented by our engineering and product management leaders, Spencer Dillard and Madhura Kale, this session provides an overview of all our latest innovations for hybrid cloud and edge computing, and how they help you address infrastructure requirements in digital sovereignty, generative AI with local data processing needs, and migration and modernization with on-premises dependencies.

Theme 1: Generative AI and agentic AI with local data processing

As you scale generative AI implementations from pilots to production, you need to balance speed of innovation with data sovereignty requirements, low-latency edge inference needs, and space, power, and cost efficiency. In the following sessions, you will learn how to address these challenges.

HMC308 | Breakout session | Build generative and agentic AI applications on-premises and at the edge

Thursday, Dec 4 | 2:30 PM – 3:30 PM PST

Wynn | Upper Convention Promenade | Cristal 7

This session shares reference architectures, best practices, and demos for running small language models (SLMs), Retrieval Augmented Generation (RAG), and agentic AI with AWS Hybrid and Edge services. Gain insights into strategies for model selection and optimization.

HMC324 | Lightning talk | BCC: Hybrid architecture for generative AI to meet regulatory needs

Monday, Dec 1 | 10:00AM – 10:20AM PST

Mandalay Bay | Oceanside C | Content Hub | Lightning Theater

Learn how Bank CenterCredit (BCC) implemented two generative AI use cases: anonymizing personally identifiable information (PII) in customer service calls with Outposts before sending it to the parent AWS Region for foundation model (FM) fine-tuning, and implementing local RAG with regulated data to improve HR efficiency.

HMC311 | Chalk talk | Developing end-to-end SLM pipelines from the cloud to the edge

Thursday, Dec 4 | 11:30AM – 12:30 PM PST

MGM | Level 3 | Chairman’s 362

This session presents a comprehensive approach to deploying SLMs to Local Zones and Outposts using Amazon SageMaker and Amazon EKS. Learn how to deliver domain-specific, fine-tuned SLMs from Regions to edge locations for low-latency inference.

HMC312 | Chalk talk | Implement RAG while meeting data residency requirements

Wednesday, Dec 3 | 5:30PM – 6:30 PM PST

Mandalay Bay | Level 3 South | South Seas A

This session explores how to implement RAG with on-premises and edge data to help you meet data residency and digital sovereignty needs.

HMC317 | Code talk | Implement Agentic AI at the edge for industrial automation

Monday, Dec 1 | 10:30AM – 11:30AM PST

Mandalay Bay | Level 3 South | Jasmine H

Manufacturing and industrial operations demand real-time, intelligent decision-making in low-connectivity environments. Learn how to deploy SmolVLMs (small vision language models) and AI agents to automate site operations using Outposts and Strands Agents.

HMC302 | Workshop | Implementing agentic AI solutions on-premises and at the edge

Wednesday, Dec 3 | 4:00PM – 6:00PM PST

MGM | Level 3 | Chairman’s 368

In this workshop, learn how to extend Amazon Bedrock AgentCore to Outposts and Local Zones to build distributed agentic applications using Model Context Protocol (MCP) and agent-to-agent (A2A) communication with on-premises data.

HMC305 | Workshop | Low-latency SLM deployment: Optimizing inference on AWS Hybrid and Edge services

Monday, Dec 1 | 08:00AM – 10:00AM PST

MGM | Level 1 | Grand 117

This workshop demonstrates a fully local deployment approach for running SLMs at the edge using Local Zones and Outposts. The implementation focuses on achieving low-latency inference and enabling data sovereignty compliance through RAG within local infrastructure.

Theme 2: Migration and modernization with on-premises or edge dependencies

Certain workloads need to stay on-premises or closer to end users. These can be driven by data residency and digital sovereignty requirements or the need to access legacy on-premises systems at a low latency. When a Region is not close enough to meet these needs, AWS brings AWS infrastructure and AWS services closer to where you need them to accelerate migration and modernization.

HMC309 | Breakout session | Migrating your VMware workloads with on-premises dependencies

Thursday, Dec 4 | 11:30AM – 12:30PM PST

Caesars Forum | Level 1 | Summit 212 | Content Hub | Orange Theater

Learn how AWS can help you migrate VMware-based workloads while addressing data residency requirements and latency-sensitive application interdependencies. Gain insights from a real-world implementation at Caesars Entertainment.

HMC217 | Lightning talk | Rivian: Modernize mission-critical manufacturing applications with AWS

Wednesday, Dec 3 | 2:30PM – 2:50PM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

Manufacturing applications like Ignition, SCADA, MES, and robotic control require low-latency connectivity to on-premises manufacturing equipment. Learn how Rivian is modernizing mission-critical and latency-sensitive manufacturing applications with Outposts.

HMC313 | Chalk talk | Extend Amazon EKS clusters for on-premises and edge use cases

Wednesday, Dec 3 | 4:00PM – 5:00PM PST

MGM | Level 3 | Chairman’s 356

Dive deep into strategies for modernizing distributed applications with Amazon EKS across Regions, Local Zones, and Outposts.

HMC303 | Workshop | Migrate and modernize VMware workloads with on-premises dependencies

Tuesday, Dec 2 | 12:30PM – 2:30PM PST

Wynn | Convention Promenade | Margaux 2

This workshop guides you through migrating VMware and other on-premises applications to Outposts while modernizing them through containerization.

Theater session | Deploying robust disaster recovery for mission-critical workloads

Wednesday, Dec 3 | 1:00PM – 1:15PM PST

The Venetian | Level 2 | Expo Hall B | NetApp Booth (#1039)

With Outposts third-party storage integration, you can modernize right inside your data centers while leveraging your investment in on-premises storage solutions. Join this session to learn how you can implement a robust disaster recovery solution for mission-critical workloads using Amazon FSx for NetApp ONTAP and on-premises ONTAP with Outposts. Learn how to perform real-time SnapMirror replication between Regions and on-premises environments and monitor replication status and RPO metrics.

Theme 3: Data residency and digital sovereignty

As organizations scale innovative solutions globally, they need to navigate complex digital sovereignty requirements. Learn how AWS Hybrid and Edge services help you adopt a consistent approach to security, monitoring, management, and auditing across jurisdictions while meeting regulatory obligations.

HMC310 | Breakout session | Digital sovereignty and data residency with AWS Hybrid and Edge services

Tuesday, Dec 2 | 4:30PM – 5:30PM PST

Caesars Forum | Level 1 | Summit 212 | Content Hub | Yellow Theater

This session examines best practices for data residency, security controls, and operational consistency across distributed locations. Hear how AWS helped Geidea, a leading fintech company, accelerate business expansion in the Middle East while meeting country-specific data residency requirements.

HMC214 | Lightning talk | DraftKings: Meeting gaming regulations at Super Bowl scale with AWS Local Zones

Wednesday, Dec 3 | 2:00PM – 2:20PM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

Learn how DraftKings built a scalable edge strategy using Regions and Local Zones to meet Federal Wire Act requirements while accelerating expansion into 26 US states.

HMC316 | Chalk talk | Address digital sovereignty with hybrid cloud solutions

Monday, Dec 1 | 1:30PM – 2:30PM PST

Mandalay Bay | Lower Level North | South Pacific D

Learn how to choose the best AWS infrastructure option for your sovereign needs and architect applications for data residency and resiliency. Discover how to implement security controls to regulate how data can be stored, processed, and transferred, and how to prevent unauthorized data access.

Theme 4: Optimizing for low and ultra-low latency

Systems like online ticketing, real-time threat detection, manufacturing control, and financial trading require network latency ranging from double-digit milliseconds to low double-digit microseconds. The Hybrid Cloud track discusses how AWS brings cloud services closer to end users and data generation points to satisfy various latency profiles.

SPF206 | Breakout session | Ticketmaster: Enhancing live event experience for fans with AWS Local Zones

Thursday, Dec 4 | 2:30PM – 3:30PM PST

Caesars Forum | Level 1 | Sports Forum | Mainstage

Discover how Ticketmaster delivers superior live event experiences by bringing its online ticket sales platform closer to fans using Local Zones.

HMC216 | Lightning talk | LSEG: Modernizing critical financial systems with multicloud and hybrid cloud

Tuesday, Dec 2 | 3:30PM – 3:50PM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

Learn how LSEG is transforming its global PriceStream FX trading platform, implementing a sophisticated architecture for ultra-low latency with managed services. Additionally, explore how LSEG is modernizing clearing systems as critical national infrastructure, balancing regulatory compliance, operational excellence, and business continuity with strict RPO/RTO requirements.

HMC215 | Lightning talk | AWS Local Zones – Sophos’ new edge in the global race against cyber-attacks

Tuesday, Dec 2 | 3:00PM – 3:20PM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

Discover how AWS Hybrid and Edge solutions transform how organizations deliver low-latency applications at the edge. Learn how Local Zones extends Regions and services closer to population centers, fitting use cases from media streaming to real-time gaming and financial trading.

HMC213 | Lightning talk | AWS Local Zones & Outposts: Verifone’s Global Edge Computing Strategy

Tuesday, Dec 2 | 11:30AM – 11:50AM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

Hear from Verifone on how it transformed its global payment ecosystem, managing 35 million terminals with innovative edge computing. Dive into strategies for deploying point-of-sale (POS) software using multi-tier architectures.

HMC402 | Chalk talk | Meet ultra-low latency and high throughput needs with AWS Outposts

Wednesday, Dec 3 | 10:00AM – 11:00AM PST

Mandalay Bay | Level 2 South | Lagoon G

Dive deep into the new category of accelerated networking Amazon EC2 instances on Outposts, purpose-built for modernizing ultra-low latency and high-throughput mission-critical workloads.

Theme 5: Architecture considerations for hybrid cloud

Running applications outside of Regions requires special architectural considerations to address space, power, and networking constraints. We will share best practices and implementation guidance on improving security, resiliency, and availability.

HMC327 | Lightning talk | Nasdaq: Build resilient infrastructure for global financial services

Tuesday, Dec 2 | 11:00AM – 11:20AM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

This session discusses how Nasdaq modernizes its mission-critical capital markets infrastructure while upholding the highest level of resiliency. Learn how Nasdaq integrates Outposts and multi-Region deployments into its core trading and surrounding systems, balancing cloud flexibility with performance and reliability.

HMC328 | Lightning talk | Build resilient and low-latency hybrid telecom infrastructure at scale

Thursday, Dec 4 | 5:00PM – 5:20PM PST

Mandalay Bay | Level 2 South | Oceanside C | Content Hub | Lightning Theater

Discover how Liberty Latin America (LLA) built telecom infrastructure in a hybrid cloud architecture for millions of subscribers. Learn how LLA created a resilient, low-latency network with over 180 VPCs.

HMC314 | Chalk talk | Deploying for resilience: HA/DR strategies for AWS Outposts and AWS Local Zones

Monday, Dec 1 | 1:30PM – 2:30PM PST

MGM | Level 1 | Boulevard 169

In this chalk talk, learn how to plan and implement resilient deployments to deliver high availability and disaster recovery, especially for business-critical or mission-critical workloads.

HMC315 | Chalk talk | Deep dive on AWS hybrid and edge networking architectures

Tuesday, Dec 2 | 1:30PM – 2:30PM PST

MGM | Level 1 | Boulevard 156

This chalk talk covers patterns such as private vs. public connectivity, service link and on-premises connectivity, and designing resilient Multi-AZ architecture across Outposts and Local Zones.

HMC403 | Code talk | Build and optimize edge architecture for resiliency with AI

Wednesday, Dec 3 | 2:30PM – 3:30PM PST

MGM | Level 3 | Chairman’s 356

This live coding session explores how to automate edge infrastructure operations with AI. Discover how to build resilient architectures with the latest Outposts and Local Zones APIs and Strands Agents.

HMC301 | Workshop | Build and operate resilient and performant distributed applications [REPEAT]

HMC301-R: Monday, Dec 1 | 3:00PM – 5:00PM PST | MGM | Grand 117

HMC301-R1: Thursday, Dec 4 | 3:00PM – 5:00PM PST | MGM | Premier 318

This workshop explores how to design and implement applications for multi-geo operations while meeting data residency and performance requirements. You will learn how to design fault-tolerant, latency-sensitive applications across distributed locations with limited hardware resources.

Activities in the Expo

Beyond sessions, join us in the re:Invent Expo (The Venetian, Level 2, Hall B) to meet with AWS experts and learn through interactive demos.

AWS Village (Booth #750)

Connect with AWS experts in the Hybrid Cloud kiosk and AWS Global Infrastructure kiosk to discuss your hybrid cloud and edge computing needs. Watch demos on the following topics and ask questions:

  • Consistent Amazon EKS experience across distributed locations and how it can simplify GPU resource orchestration
  • How to optimize and deploy SLMs locally for AI inference with low latency or data residency needs
  • How to build agentic AI workflows at the edge for manufacturing automation

Stop by the Migration & Modernization area of the AWS Village to see hardware innovations inside the latest generation of Outposts up close and personal.

AWS for Industries Pavillion (Booth #111)

Explore how AWS Hybrid and Edge services unlock new use cases and improve operational efficiency across multiple industries through immersive experiences:

  • AWS for Telecom – Modernizing telecom infrastructure while meeting data sovereignty and performance requirements
  • AWS for Public Sector – Accelerating tactical edge deployments to improve mission outcomes
  • AWS for Automotive – Advancing global R&D of embedded hardware and software

Partner booths

Discover how AWS Hybrid and Edge services and AWS Partner solutions unlock additional use cases through demos:

  • Pure Storage (Booth #1756) – Simplifying cloud migration with on-premises dependencies through Outposts integration with Pure Storage FlashArray
  • Intel (Booth #1010): – Powering physical AI and agentic systems for real-time operations in manufacturing
  • Seagate (Booth #159) – Facilitating data ingestion and pre-processing at the edge

Ready for re:Invent 2025?

We hope this guide to hybrid cloud and edge computing helps you maximize your learning experience at re:Invent. Not able to attend in-person? Register for the virtual-only event offered at no additional cost to livestream keynotes and innovation talks, and access on-demand session content. See you in Las Vegas or virtually!

Introducing guidelines for network scanning

Post Syndicated from Stephen Goodman original https://aws.amazon.com/blogs/security/introducing-guidelines-for-network-scanning/

Amazon Web Services (AWS) is introducing guidelines for network scanning of customer workloads. By following these guidelines, conforming scanners will collect more accurate data, minimize abuse reports, and help improve the security of the internet for everyone.

Network scanning is a practice in modern IT environments that can be used for either legitimate security needs or abused for malicious activity. On the legitimate side, organizations conduct network scans to maintain accurate inventories of their assets, verify security configurations, and identify potential vulnerabilities or outdated software versions that require attention. Security teams, system administrators, and authorized third-party security researchers use scanning in their standard toolkit for collecting security posture data. However, scanning is also performed by threat actors attempting to enumerate systems, discover weaknesses, or gather intelligence for attacks. Distinguishing between legitimate scanning activity and potentially harmful reconnaissance is a constant challenge for security operations.

When software vulnerabilities are found through scanning a given system, it’s particularly important that the scanner is well-intentioned. If a software vulnerability is discovered and attacked by a threat actor, it could allow unauthorized access to an organization’s IT systems. Organizations must effectively manage their software vulnerabilities to protect themselves from ransomware, data theft, operational issues, and regulatory penalties. At the same time, the scale of known vulnerabilities is growing rapidly, at a rate of 21% per year for the past 10 years as reported in the NIST National Vulnerability Database.

With these factors at play, network scanners need to scan and manage the collected security data with care. There are a variety of parties interested in security data, and each group uses the data differently. If security data is discovered and abused by threat actors, then system compromises, ransomware, and denial of service can create disruption and costs for system owners. With the exponential growth of data centers and connected software workloads providing critical services across energy, manufacturing, healthcare, government, education, finance, and transportation sectors, the impact of security data in the wrong hands can have significant real-world consequences.

Multiple parties

Multiple parties have vested interests in security data, including at least the following groups:

  • Organizations want to understand their asset inventories and patch vulnerabilities quickly to protect their assets.
  • Program auditors want evidence that organizations have robust controls in place to manage their infrastructure.
  • Cyber insurance providers want risk evaluations of organizational security posture.
  • Investors performing due diligence want to understand the cyber risk profile of an organization.
  • Security researchers want to identify risks and notify organizations to take action.
  • Threat actors want to exploit unpatched vulnerabilities and weaknesses for unauthorized access.

The sensitive nature of security data creates a complex ecosystem of competing interests, where an organization must maintain different levels of data access for different parties.

Motivation for the guidelines

We’ve described both the legitimate and malicious uses of network scanning, and the different parties that have an interest in the resulting data. We’re introducing these guidelines because we need to protect our networks and our customers; and telling the difference between these parties is challenging. There’s no single standard for the identification of network scanners on the internet. As such, system owners and defenders often don’t know who is scanning their systems. Each system owner is independently responsible for managing identification of these different parties. Network scanners might use unique methods to identify themselves, such as reverse DNS, custom user agents, or dedicated network ranges. In the case of malicious actors, they might attempt to evade identification altogether. This degree of identity variance makes it difficult for system owners to know the motivation of parties performing network scanning.

To address this challenge, we’re introducing behavioral guidelines for network scanning. AWS seeks to provide network security for every customer; our goal is to screen out abusive scanning that doesn’t meet these guidelines. Parties that broadly network scan can follow these guidelines to receive more reliable data from AWS IP space. Organizations running on AWS receive a higher degree of assurance in their risk management.

When network scanning is managed according to these guidelines, it helps system owners strengthen their defenses and improve visibility across their digital ecosystem. For example, Amazon Inspector can detect software vulnerabilities and prioritize remediation efforts while conforming to these guidelines. Similarly, partners in AWS Marketplace use these guidelines to collect internet-wide signals and help organizations understand and manage cyber risk.

“When organizations have clear, data-driven visibility into their own security posture and that of their third parties, they can make faster, smarter decisions to reduce cyber risk across the ecosystem.” – Dave Casion, CTO, Bitsight

Of course, security works better together, so AWS customers can report abusive scanning to our Trust & Safety Center as type Network Activity > Port Scanning and Intrusion Attempts. Each report helps improve the collective protection against malicious use of security data.

The guidelines

To help ensure that legitimate network scanners can clearly differentiate themselves from threat actors, AWS offers the following guidance for scanning customer workloads. This guidance on network scanning complements the policies on penetration testing and vulnerability reporting. AWS reserves the right to limit or block traffic that appears non-compliant with these guidelines. A conforming scanner adheres to the following practices:

Observational

  • Perform no actions that attempt to create, modify, or delete resources or data on discovered endpoints.
  • Respect the integrity of targeted systems. Scans cause no degradation to system function and cause no change in system configuration.
  • Examples of non-mutating scanning include:
    • Initiating and completing a TCP handshake
    • Retrieving the banner from an SSH service

Identifiable

  • Provide transparency by publishing sources of scanning activity.
  • Implement a verifiable process for confirming the authenticity of scanning activities.
  • Examples of identifiable scanning include:
    • Supporting reverse DNS lookups to one of your organization’s public DNS zones for scanning Ips.
    • Publishing scanning IP ranges, organized by types of requests (such as service existence, vulnerability checks).
    • If HTTP scanning, have meaningful content in user agent strings (such as names from your public DNS zones, URL for opt-out)

Cooperative

  • Limit scan rates to minimize impact on target systems.
  • Provide an opt-out mechanism for verified resource owners to request cessation of scanning activity.
  • Honor opt-out requests within a reasonable response period.
  • Examples of cooperative scanning include:
    • Limit scanning to one service transaction per second per destination service.
    • Respect site settings as expressed in robots.txt and security.txt and other such industry standards for expressing site owner intent.

Confidential

  • Maintain secure infrastructure and data handling practices as reflected by industry-standard certifications such as SOC2.
  • Ensure no unauthenticated or unauthorized access to collected scan data.
  • Implement user identification and verification processes.

See the full guidance on AWS.

What’s next?

As more network scanners follow this guidance, system owners will benefit from reduced risk to their confidentiality, integrity, and availability. Legitimate network scanners will send a clear signal of their intention and improve their visibility quality. With the constantly changing state of networking, we expect that this guidance will evolve along with technical controls over time. We look forward to input from customers, system owners, network scanners and others to continue improving security posture across AWS and the internet.

If you have feedback about this post, submit comments in the Comments section below or contact AWS Support.

Stephen Goodman

Stephen Goodman

As a senior manager for Amazon active defense, Stephen leads data-driven programs to protect AWS customers and the internet from threat actors.

Learning email deliverability with Amazon SES

Post Syndicated from Alaa Hammad original https://aws.amazon.com/blogs/messaging-and-targeting/learning-email-deliverability-with-amazon-ses/

Emails landing in spam folders? Facing account suspension warnings? This blog post walks you through seven targeted video tutorials that address the most common Amazon Simple Email Service (Amazon SES) deliverability challenges our customers encounter. Each video in this series provides actionable insights which will allow you to gain knowledge on every aspect of email deliverability, improve your email performance and increase your inbox placement rates.

Why email deliverability matters

Poor deliverability can lead to emails landing in spam folders, increase bounce rates, and can trigger account suspension. Email deliverability directly impacts:

  • Your sender reputation
  • Inbox placement rates
  • Customer engagement
  • Business revenue
  • Compliance with email service provider policies.

Our video series will help you avoid these pitfalls and build a robust email delivery strategy, including onboarding to Amazon SES, choosing your IP, monitoring feedback, reputation, and how to handle unsubscribes, bounces, and complaints.

Onboarding your email solution to Amazon SES

Watch: SES Deliverability learning series: Onboarding your email solution to Amazon SES

What you’ll learn: This video provides a complete migration roadmap for moving your email system to Amazon SES, including domain verification and authentication setup through DKIM, SPF, and DMARC protocols. You’ll learn how to move the account to the production access, acquire and configure dedicated IPs, create and manage configuration sets, and set up Virtual Deliverability Manager for email monitoring. The video also covers the difference between SMTP and API sending methods to explain bulk sender requirements and how to comply with them for successful email delivery.

How this helps with email deliverability: Following SES best practices when onboarding your email system to Amazon SES will help protect your emails from being spoofed, with proactive monitoring using Virtual Deliverability Manager, you can identify issues before they impact delivery, maintaining strong sender reputation and sustained email delivery success.

Choosing the right Amazon SES IP

Watch: SES Deliverability learning series: Choosing the Right Amazon SES IP for Your Email Needs

What you’ll learn: Key differences between shared IPs, dedicated IPs (Managed and Standard), How to choose the right IP environment based on your sending volume and use case, best practices for managing IP reputation, the IP warm-up process and its importance, and shared responsibilities between you and Amazon SES when using dedicated IPs. This video shows you how to rebuild trust with mailbox providers through proper IP management.

How this helps with email deliverability: If your account’s reputation was impacted due to poor sending practices, switching to dedicated IPs can help isolate your reputation and provide better control over your sending environment.

Monitoring email feedback

Watch: Mastering email deliverability: Monitoring email feedback

What you’ll learn: Discover how Amazon SES feedback loops can prevent account pause by alerting you to rising complaint, understanding and analyzing complaint data, utilizing mailbox provider postmaster tools, setting up comprehensive monitoring dashboards, tracking deliverability and engagement metrics, and strategies for emails to reach intended recipients. This video teaches you how to set up systems that alert you to rising complaint rates, bounce rates, and other warning signs before they started to impact your account’s ability to send emails.

How this helps with email deliverability: Proper monitoring is essential for preventing further complaints or bounces that could put your AWS account at risk of termination.

Email reputation with Amazon SES

Watch: Mastering email deliverability: Email Reputation with Amazon SES

What you’ll learn: What email reputation is and why it matters, how email providers evaluate sender reputation, best practices for isolating and managing your reputation, authentication methods to protect your reputation, and effective monitoring techniques for reputation management.

How this helps with email deliverability: Understanding email reputation is important for maintaining a good reputation with mailbox providers.

Handling email unsubscribes

Watch: Mastering email deliverability: Handling email unsubscribes

What you’ll learn: The importance of proper unsubscribe handling, how to remove unsubscribes from your SES mailing lists, the connection between unsubscribes and deliverability rates, strategies to maintain clean email lists, how to identify and prevent mass unsubscribe events, what to do when you see high unsubscribe rates. This video shows you how to properly handle unsubscribes and reduce rates that might trigger spam complaints by mailbox providers or impact your account’s ability to send emails.

How this helps with email deliverability: High unsubscribe rates often indicate content quality issues, sending emails to users who don’t want or expect, list hygiene issue, not having a proper confirmed opt-in, and sending too frequently.

How to handle bounces and boost inbox success

Watch: Mastering Email Deliverability: How to Handle Bounces and Boost Inbox Success

What you’ll learn: Different types of email bounces (soft and hard bounces), What each bounce type indicates about your campaign health, practical strategies to manage and reduce bounce rates, best practices for cleaning your email lists, how to implement effective bounce monitoring systems, steps to identify and address unusual bounce trends, and actionable solutions for common bounce-related challenges. This video provides specific strategies to clean your lists and reduce bounce rates that could impact your account’s ability to send emails, helping you demonstrate improved sending practices with SES.

How this helps with email deliverability: High bounce rates often indicate that a sender is sending unsolicited email to their recipients.

Understanding and handling email complaints

Watch: Mastering Email Deliverability: Understanding and Handling Email Complaints

What you’ll learn: What email complaints are and why they matter, different types of complaints (spam reports, unsubscribe requests), how complaints directly impact your deliverability rates, the correlation between complaint rates and spam folder placement , strategies to keep complaint rates within ideal ranges, practical tips to proactively reduce email complaints, and best practices for content creation and opt-out options. This video helps you understand how to maintain a healthy sender reputation by implementing best practices that align with email service provider guidelines and minimize potential risks to the email system.

How this helps with email deliverability: Mailbox providers may reject emails received from senders based on the complaints they received from their recipients.

Take action today

Don’t let deliverability issues impact your business success. Each video in this series provides practical, actionable guidance that you can implement immediately. Whether you’re dealing with account suspensions issue or simply want to optimize your email performance, these videos will help you establish reliable email delivery that meets mailbox provider requirements.

Ready to get started? Begin with the video that addresses your most pressing challenge, then work through the complete series to learn more about all aspects of email deliverability with Amazon SES.

Have questions about implementing these strategies? The AWS Support team is here to help you optimize your email deliverability and resolve any challenges you may be facing.

About the author

Scaling Generative AI Video Depends on Your Data Egress Strategy

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/scaling-generative-ai-video-depends-on-your-data-egress-strategy/

A decorative image showing several clouds.

The AI and cloud infrastructure industry talks endlessly about GPUs, model size, and compute capacity, but there’s an invisible Achilles heel that can quietly undermine even the most promising AI projects: data egress.

According to a new Dimensional Research survey, 95% of organizations experience unexpected cloud storage fees, often from retrieval, egress, or API transactions. These hidden costs are rarely visible in early budgets, but they can torpedo innovation as workloads scale, especially when video enters the mix. Raw footage, frame-level training data, model checkpoints, and final renders can add up to hundreds of terabytes every week, straining both budgets and infrastructure. 

Read the full report

We surveyed over 400 IT decision makers and one thing stood out. Surprise charges affect almost everyone. Learn what’s driving them—and how to avoid them.

Read the Full Report

Most generative AI video outputs today max out at 480p or 720p resolution. As demand grows for 1080p and 4K, storage and bandwidth requirements will multiply. Without a deliberate egress strategy, that growth becomes a silent tax on innovation. Over time, it restricts experimentation, reduces iteration speed, and undermines cost predictability.

The future of AI video belongs to teams that treat egress strategy as part of their innovation architecture and choose partners that let them move data freely between storage and compute, without penalty.

Inside the generative AI data pipeline

Modern AI systems no longer operate inside a single environment. Data is stored in one place, trained in another, and increasingly delivered at the edge. As workloads scale, the ability to move data efficiently becomes as important as compute capacity.

According to IDC, 88% of cloud buyers now deploy hybrid cloud environments, and 79% already use multiple providers. The Dimensional Research survey found that 99% of organizations struggle with limited flexibility and interoperability, highlighting how closed ecosystems are slowing progress just as multimodal AI demands more open, composable infrastructure.

To understand why egress matters so much for generative AI video, it helps to look at the AI data pipeline, which follows five continuous stages: 

  1. Data ingest and active archive: Collect and store raw images, video, audio, and metadata for future processing.
  2. Data processing: Clean, label, and transform data into usable training sets.
  3. Model experimentation and training: Run GPU-intensive model development and fine-tuning, save checkpoints and weights.
  4. Model deployment and inference: Apply trained models to new video, user queries, or edge devices to generate results.
  5. Monitoring: Track accuracy, latency, and system health to retrain and optimize continuously.
A chart that defines the five continuous stages of the data pipeline, including data ingest and archive, data processing, model experimentation and training, model deployment and inference, and monitoring.

Each stage has distinct storage and compute requirements, but data moves between them constantly. For AI video, those transfers can span regions and providers. When egress is slow or expensive, the entire pipeline backs up, delaying iteration and driving up cost.

When data can’t move, innovation can’t either

Keeping everything under one cloud provider once simplified management. At first glance, it still seems convenient to keep storage, compute, and archive all in one place. Within a single AWS region, egress is free. But as soon as data crosses regions or providers, the model breaks down.

Tiered pricing makes costs hard to forecast. Egress fees penalize movement. Resource contention slows performance, and interoperability gaps lock teams into static configurations. AI video workloads amplify the problem: training, inference, and storage often require different environments optimized for each stage. 

Dimensional Research’s data shows that 55% of organizations note egress costs as the single biggest barrier to switching cloud providers. Many stay with less efficient or more expensive infrastructure simply because the economics of mobility make innovation too costly. Moving just 1 PB of data out of AWS storage in the US East region costs about $53,800 per month—often enough to halt multi-cloud testing entirely.

The true cost, however, is in the experiments that are never run and the innovations that don’t get discovered because of a pricing structure that discourages exploration. 

Freedom of data movement is the new competitive edge

In generative AI, the pace of progress is set by how quickly teams can test, retrain, and redeploy new models. That agility requires data mobility.

As organizations adopt composable AI stacks that mix specialized compute, regional storage, and orchestration tools, success depends on how openly data flows between them. Teams that design for movement can scale faster, adapt to new technologies, and stay resilient as infrastructure changes.

For teams building generative AI video applications, the impact is especially pronounced. A studio fine-tuning a diffusion model might burst to GPU providers with available capacity, render high-resolution outputs, and archive them for reuse, all without rewriting code or paying to move the data each time.

Data mobility has become a measure of competitiveness. The faster teams can move information across environments, the faster they can innovate.

How to build an egress strategy that fuels innovation

A good egress strategy ensures that storage and compute stay aligned as workloads scale. It helps teams anticipate cost, performance, and interoperability issues before they turn into blockers.

Here are a few practical steps to get there:

  1. Map your data flows. Identify where data originates, how it moves between services, and which transfers happen most frequently.
  2. Quantify transfer and API transaction costs. Include both in your total cost of ownership models. Even small fees add up quickly at petabyte scale.
  3. Test portability. Run controlled migrations or bursts to secondary compute providers to expose hidden bottlenecks.
  4. Select for openness. Favor vendors with flat, transparent pricing, free or low-cost egress, and broad S3 compatibility.
  5. Plan for growth. Multimodal models and higher-resolution video outputs will multiply data transfer volumes. Design bandwidth and budget models accordingly.

Beyond controlling costs, the goal is to keep flexibility built into your architecture so your team can use the best tools for each stage of the AI pipeline, without being trapped by pricing friction or closed ecosystems.

The Backblaze difference: Open by design

Storage that supports innovation shouldn’t penalize movement. That’s why we created Backblaze B2 Overdrive to give teams with high-throughput, data-intensive workloads the flexibility they need to innovate. 

Overdrive is the right fit for AI video because of its:

  • Predictable economics: $15/TB/month with unlimited free egress (no penalties for moving data to the compute you need). 
  • Zero transaction fees: API calls don’t become a hidden tax as pipelines scale.
  • S3 compatibility and high throughput: Drop into existing pipelines without rewrites and keep large media workflows moving quickly across training, rendering, inference, and archive.

AI startup Decart put Backblaze B2 through its paces as it developed a real-time generative AI open world model, with millions of hours of training video data and multi-petabyte workloads daily.

What we really needed was a place where we could store an insane amount of data and, at the same time, download it to a few different GPU clusters around the world, and for all that to not cost an insane amount of money. That’s why we chose Backblaze.

—Dean Leitersdorf, Co-Founder and CEO, Decart

With Backblaze’s free egress model, they reduced AI operation costs by 75% while maintaining flexibility across compute environments. 

If you’re scaling generative AI video, Backblaze B2 Overdrive gives you the freedom to put data where it performs best, without egress penalties, transaction surprises, or architectural do-overs.

The post Scaling Generative AI Video Depends on Your Data Egress Strategy appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Security updates for Tuesday

Post Syndicated from daroc original https://lwn.net/Articles/1047950/

Security updates have been issued by AlmaLinux (buildah, firefox, go-rpm-macros, kernel, kernel-rt, podman, and thunderbird), Debian (erlang, python-gevent, and r-cran-gh), Fedora (buildah, chromium, k9s, kubernetes1.33, kubernetes1.34, podman, python-mkdocs-include-markdown-plugin, and webkitgtk), Gentoo (Chromium, Google Chrome, Microsoft Edge. Opera, qtsvg, redict, redis, UDisks, and WebKitGTK+), Mageia (cups-filters and ruby-rack), Oracle (kernel and libssh), Red Hat (.NET 8.0, tigervnc, xorg-x11-server, and xorg-x11-server-Xwayland), SUSE (act, bind, cups-filters, govulncheck-vulndb, grub2, libebml, python39, and tcpreplay), and Ubuntu (linux-raspi, linux-raspi-realtime, openjdk-21, openjdk-25, python3.12, python3.11, python3.10, python3.9, python3.8, python3.7, python3.6, python3.5, python3.4, and runc-app, runc-stable).

The collective thoughts of the interwebz