Tag Archives: announcements

Accelerate Spark on EMR Serverless with larger workers and shuffle-optimized disks

2026-07-29 Karthik Prabhakar

Post Syndicated from Karthik Prabhakar original https://aws.amazon.com/blogs/big-data/accelerate-spark-on-emr-serverless-with-larger-workers-and-shuffle-optimized-disks/

With Amazon EMR Serverless, you can run open source big data frameworks such as Apache Spark and Apache Hive without managing clusters or infrastructure. Customers are increasingly choosing EMR Serverless for their analytics workloads because of the simplicity of a fully managed, serverless experience. As adoption grows, teams want to bring their most demanding jobs to Serverless too. These jobs include large-scale joins, shuffle-heavy ETL, and memory-intensive analytics that previously required carefully sized clusters. Customers migrating these heavyweight workloads from their Spark clusters often need the same compute shapes on EMR Serverless to achieve the same price-performance and make migration easier.

Today, we’re excited to announce a new 32 vCPU / 244 GB worker configuration on Amazon EMR Serverless, giving you the headroom to run your most intensive workloads without leaving the serverless experience.

Overview of larger workers

An EMR Serverless application uses workers to run your Spark tasks, and you can choose a worker size that matches your workload. The new 32 vCPU worker offers a large compute and memory footprint (32 vCPUs and 244 GB of memory) that supports attaching up to 2,000 GB of shuffle-optimized disk. This combination benefits three common workload patterns:

Shuffle-intensive workloads – Wide transformations such as join, groupBy, sortBy, and repartition redistribute large amounts of data across the cluster. Larger workers keep more shuffle data local to each executor and read and write shuffle blocks on higher-throughput disk, reducing remote fetches and shuffle wait time.
I/O-heavy workloads – Queries that scan large datasets or spill intermediate data to disk are limited by disk throughput and IOPS. The shuffle-optimized disk raises the ceiling on both sides. Large workers also improve network bandwidth.
Memory-intensive workloads – Higher per-executor memory (244 GB versus 30 GB) lets more data be cached and processed in memory without spilling, which helps with data skew and caching.

Benchmark setup

We compared the recommended large-worker configuration against common small-worker defaults at identical total compute (192 vCPUs).

Large workers – 6 executors × (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk).
Small workers – 48 executors × (4 vCPU / 30 GB / 200 GB standard disk).

The two configurations differ in both worker shape and disk class. We compare them as paired configurations, since shuffle-optimized disk is the intended disk type for the 32 vCPU worker. The reported gains reflect this combined effect.

Both configurations set spark.dynamicAllocation.enabled=false and spark.scheduler.minRegisteredResourcesRatio=1 to reduce variance from worker launch times. Requiring full registration ensures each query starts only after the cluster is ready. We ran every query with 3 iterations and reported the median. Both benchmarks ran on EMR release emr-7.13.0 in a virtual private cloud (VPC) with an Amazon Simple Storage Service (Amazon S3) gateway endpoint attached to the private subnets.

Benchmark results

This post presents benchmark results comparing the new 32 vCPU workers against the existing 4 vCPU standard workers using the industry-standard TPC-DS and TPC-H benchmarks. Across 126 queries (104 TPC-DS and 22 TPC-H), large workers delivered an average of 29% faster query execution and 29% lower query-attributed cost, with zero regressions and improvements peaking at 45–55% on shuffle-heavy, multi-table join queries. Both the larger executor shape and the shuffle-optimized disk contribute to these gains. The disk advantage is most pronounced on the shuffle and I/O-heavy queries where the largest improvements appear.

The following table summarizes the results across both benchmarks. Large workers won every query on both performance and cost.

Note: The benchmark results in this post are derived from the TPC-DS and TPC-H benchmark specifications. TPC-DS and TPC-H are trademarks of the Transaction Processing Performance Council.

Benchmark	Queries tested	Avg performance improvement	Avg cost improvement
TPC-DS (3 TB)	104	26.7%	27.4%
TPC-H (1 TB)	22	38.5%	37.2%
Combined	126	28.8%	29.1%

TPC-DS 3 TB benchmark

TPC-DS is an industry-standard decision support benchmark that models complex analytical workloads with multi-table joins, subqueries, and aggregations. We ran 104 queries from the TPC-DS v2.4 suite against a 3 TB partitioned Parquet dataset, with 3 iterations per query for statistical confidence.

The test environment was as follows:

Dataset: 3 TB partitioned Parquet (24 TPC-DS tables).
Queries: 104 (full suite minus 4 incompatible with the dataset schema).
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.

The following chart shows the top 10 and bottom 5 queries by performance improvement. All 104 queries show a positive improvement, with the largest gains on shuffle-heavy queries such as q58 (45%), q21 (43%), and q12 (42%).

Bar chart of per-query performance improvement for the top 10 and bottom 5 TPC-DS queries

Performance improvement for the top 10 and bottom 5 TPC-DS queries, 32 vCPU compared to 4 vCPU standard workers

Why large workers are faster for TPC-DS

TPC-DS queries are characterized by complex multi-table joins that generate large shuffle operations. With 6 large executors instead of 48 small ones, the shuffle-optimized disk provides significantly higher random I/O throughput for reading and writing shuffle blocks. In addition, fewer executors mean less network coordination during shuffle. Each executor fetches shuffle data from only 5 remote sources instead of 47. This increases the share of shuffle data read locally, which improves performance.

TPC-H 1 TB benchmark

TPC-H is a decision support benchmark that focuses on ad hoc analytical queries. We ran all 22 TPC-H queries against a 1 TB dataset, with 3 iterations per query. Each query was submitted as a separate start-job-run (a fresh Spark application) to simulate the realistic pattern of independent ad hoc queries arriving without session warmup.

The test environment was as follows:

Dataset: 1 TB partitioned Parquet.
Queries: 22.
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.

Bar chart of per-query performance improvement across the 22 TPC-H queries

Performance improvement for TPC-H queries, 32 vCPU compared to 4 vCPU standard workers

Why large workers are faster for TPC-H

The shuffle-optimized disk accelerates this initial table-scan phase. The subsequent query execution benefits from higher per-executor memory (244 GB versus 30 GB), which lets more data be processed in memory without spilling to disk.

Cost calculation and improvement results

The following table compares the cost of running TPC-DS and TPC-H benchmarks on larger workers (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk) versus smaller workers (4 vCPU / 30 GB / 200 GB standard disk) on EMR Serverless. Both configurations use identical total compute (192 vCPUs).

TPC-DS 3 TB

Metric	Larger Workers (6 × 32 vCPU)	Smaller Workers (48 × 4 vCPU)
Runtime	2,780.8s	3,670.2s
Resource Billed	vCPU = 148.31 \| Memory = 1,130.87 \| Disk = 9,269.47	vCPU = 195.74\| Memory = 1,468.07 \| Disk = 9787.15
Cost	$15.37	$19.87
Total vCPU	192	192
Total Disk	12 TB (shuffle-optimized)	9.6 TB (standard)
Improvement	27.4% lower cost	Baseline

TPC-H 1 TB

Metric	Larger Workers (6 × 32 vCPU)	Smaller Workers (48 × 4 vCPU)
Runtime	940.7s	1,529.5s
Resource Billed	vCPU = 50.17 \| Memory = 382.54 \| Disk = 3,135.60	vCPU = 81.57 \| Memory = 611.81 \| Disk = 4,078.72
Cost	$5.20	$8.28
Total vCPU	192	192
Total Disk	12 TB (shuffle-optimized)	9.6 TB (standard)
Improvement	37.2% lower cost	Baseline

Notes:

Runtime represents the sum of median query execution times across all queries in the benchmark (3 iterations per query, median reported).
Calculated cost is computed using EMR Serverless on-demand pricing: vCPU-hr ($0.052624), Memory GB-hr ($0.0057785), Storage GB-hr ($0.000111).
Illustrative TPC-DS run cost calculation:
- Large worker: (148.31 * $0.052624) + (1,130.87 * $0.0057785) + (9,269.47 * $0.000111) = $15.37.
- Smaller worker: (195.74 * $0.052624) + (1,468.07 * $0.0057785) + (9787.15 * $0.000111) = $19.87.
Cost is proportionally attributed to query execution time, excluding Spark initialization and shutdown overhead.
Both configurations use identical total vCPU (192) with Dynamic Resource Allocation disabled.
The improvement percentage represents the cost reduction achieved by larger workers relative to smaller workers.
Disk Sizing: The larger workers provisioned 25% more disk (12 TB versus 9.6 TB), yet the total cost came out 22.6–37.2% lower. Disk is by far the lowest-priced billing dimension on EMR Serverless ($0.000111 per GB-hour, versus 52x that for memory and 474x for vCPU) and made up only $1.03 of the $15.37 TPC-DS total. Many customers under-provision disk to trim this smallest line item, and it backfires. Shuffles slow down, jobs run longer, and every extra second is billed on the costlier vCPU and memory dimensions. On large workers (8+ vCPUs), shuffle-optimized disks scale IOPS and throughput with capacity, which helps move shuffle data faster. Size disk as a performance lever, not a cost lever.

When to use large workers

To determine whether the 32 vCPU worker with shuffle-optimized disk will benefit your Spark applications, consider the following:

Check the Stages tab of the Spark History Server for your EMR Serverless application and review the Shuffle Read and Shuffle Write columns. The larger the shuffle volume relative to the number of executors, the more a job benefits from keeping shuffle data local on high-throughput disk. Jobs that shuffle tens of gigabytes or more per executor are strong candidates.
Check the Stages tab for the Spill (memory) and Spill (disk) columns and the Executors tab for peak JVM memory. If data is spilling to disk or peak memory is close to the configured executor memory, the higher memory of the large worker can remove the spill and improve performance.

When not to use large workers

Large workers are not the best fit for every workload:

I/O-bound jobs – For workloads whose runtime is dominated by reading and writing data (rather than shuffle or memory pressure), a larger number of smaller workers (for example, 8 or 16 vCPU) with the same disk sizes can deliver better aggregate throughput. Spreading the work across more executors increases read/write parallelism, although a few large workers can leave disk and network bandwidth underused.

Conclusion

In this post, we demonstrated that EMR Serverless 32 vCPU workers deliver performance and cost benefits for shuffle-intensive Spark workloads. Across 126 TPC-DS and TPC-H queries, larger workers achieved an average 29% faster execution and 29% lower cost.

We recommend evaluating the 32 vCPU worker with shuffle-optimized disk for your shuffle-intensive and I/O-heavy EMR Serverless Spark workloads. To get started, use the following configuration:

--conf spark.executor.cores=32
--conf spark.executor.instances=6
--conf spark.executor.memory=220g
--conf spark.emr-serverless.executor.disk=2000G
--conf spark.emr-serverless.executor.disk.type=SHUFFLE_OPTIMIZED

For more information about worker configurations, refer to Worker configurations in the Amazon EMR Serverless User Guide. We also recommend staying up to date with the latest EMR releases to take advantage of ongoing performance improvements.

About the authors

2026 Phase 1a IRAP report is now available on AWS Artifact for Australian customers

2026-07-28 Patrick Chang

Post Syndicated from Patrick Chang original https://aws.amazon.com/blogs/security/2026-phase-1a-irap-report-is-now-available-on-aws-artifact-for-australian-customers/

Amazon Web Services (AWS) is excited to announce that the latest version of Information Security Registered Assessors Program (IRAP) report (Phase 1a – full assessment) is now available through AWS Artifact. An independent Australian Signals Directorate (ASD) certified IRAP assessor completed the IRAP assessment of AWS in June 2026.

The new IRAP report includes four additional AWS services that are now assessed at the PROTECTED level under IRAP. This brings the total number of services assessed at the PROTECTED level to 167.

The four newly assessed services are:

For the full list of services, see the IRAP tab on the AWS Services in Scope by Compliance Program page.

We have developed an IRAP documentation pack to help our Australian customers and their partners plan, architect, and assess risk for their workloads when they use AWS cloud services.

We developed this pack in accordance with the Australian Cyber Security Centre (ACSC) Cloud Security Guidance and Cloud Assessment and Authorisation framework, which addresses guidance within the Australian Government’s Information Security Manual (ISM, September 2025 version), the Department of Home Affairs’ Protective Security Policy Framework (PSPF), and the Digital Transformation Agency’s Secure Cloud Strategy.

The IRAP pack on AWS Artifact also includes newly updated versions of the AWS Consumer Guide and the whitepaper Reference Architectures for ISM PROTECTED Workloads in the AWS Cloud.

Reach out to your AWS representatives to let us know which additional services you want to see in scope for upcoming IRAP assessments. We strive to bring more services into scope at the PROTECTED level under IRAP to support your requirements.

AWS Shield Advanced is embracing the AWS WAF Anti-DDoS managed rule group: What changes and how to prepare

2026-07-27 Eitav Arditti

Post Syndicated from Eitav Arditti original https://aws.amazon.com/blogs/security/aws-shield-advanced-is-embracing-the-aws-waf-anti-ddos-managed-rule-group-what-changes-and-how-to-prepare/

Application-layer distributed denial of service (DDoS) attacks are difficult to detect because they closely resemble legitimate traffic. HTTP request floods are now among the most common vectors targeting web applications, using valid-looking requests that blend in with normal user activity.

In June 2025, AWS launched the AWS WAF Anti-DDoS managed rule group, built specifically for application-layer (L7) DDoS protection. AWS Shield Advanced is adopting it as the default application-layer protection, and in time as the only one. On July 27, AWS Shield Advanced begins adding the Anti-DDoS managed rule group to eligible web access control lists (ACLs) in Count mode. It will not cause any interruption to your traffic alongside your existing L7 automatic mitigation and WAF rules. In this blog post, we provide details regarding the Anti-DDoS managed rule group and when the change is expected to reach your web ACLs. You will understand the phases and steps that you need to take before the finish date, including how the monitoring and metrics will change.

Anti-DDoS managed rule group features

The Anti-DDoS managed rule group builds on what Shield Advanced automatic mitigation already provides. It profiles your traffic, learns what normal traffic looks like for your application, and establishes a baseline in minutes rather than hours. When an attack starts, it reacts within seconds and there are no health checks to configure. The rule group adds a Challenge action to the Block and Count actions you already use. Challenge decisions are driven by the AMR labels that mark the suspicion level of each inspected request. One option is a silent browser challenge, which has a background verification that runs in the visitor’s browser with no interstitial page, so legitimate users are never interrupted while automated traffic is filtered out. You can also exclude workload paths that don’t support Challenge, which fall back to Block mitigations instead. Sensitivity is configurable to Low, Medium, or High, and you set it separately for Block and Challenge. Block and Challenge are tuned independently; meaning you can run Challenge at high sensitivity to catch more suspicious traffic while keeping Block low to avoid dropping legitimate requests or reverse it for a stricter posture.

The rest is about cost and visibility:

It uses less capacity than before. The rule group needs 50 web ACL capacity units (WCUs), down from the 150 the previous protection required, providing you with capacity for the rest of your rules.
The dashboard ships in the AWS Management Console for AWS WAF. It’s there now, showing live DDoS events, match metrics, and the top URIs, geographies, and IP addresses driving traffic.
It labels everything it inspects. Requests get labels for event-detected, graduated suspicion levels, and specific rules. Match on those labels in your own AWS WAF rules when you need logic the rule group doesn’t cover.
You don’t pay for the attack traffic. During active mitigation, blocked DDoS requests drop out of your monthly count. That exclusion covers AWS WAF request fees, Anti-DDoS managed rule group request fees, and Shield Advanced request charges.

AWS Shield Advanced isn’t required to use any of these features. Shield Advanced subscribers get the rule group included with AWS WAF and any customer can turn it on independently. See AWS WAF pricing for more information on costs.

Implementation details

Shield Advanced upgrades application-layer DDoS protection in five phases. The following dates are when AWS will act automatically, not the earliest date when you can act. After the rule group is deployed in Count mode on July 27, 2026, you can begin migrating right away rather than waiting for the October auto-upgrade. There’s no window where protection lapses. Your current automatic mitigation stays active through every phase until the Anti-DDoS managed rule group takes over. That handoff happens in a single operation, with no cutover window and no gap for your traffic flows.

Phase 1: Anti-DDoS managed rule group deployed in Count mode (rolling out July 27–August 7, 2026)

AWS adds the Anti-DDoS managed rule group in Count mode to every web ACL eligible for this rollout. Eligible means any Shield Advanced web ACL with at least one resource using application-layer automatic mitigation that isn’t already running the Anti-DDoS rule group. This is a broader set than the web ACLs eligible for the October auto-upgrade (Phase 3), which applies a stricter test. The deployment rolls out gradually, starting July 27 and expected to finish by August 7, 2026, so different web ACLs might be updated on different days. There’s no impact to your traffic because the rule group watches and labels requests without acting on them while your existing automatic mitigation keeps running. Throughout the evaluation period, you receive DDoS events, metrics, and AWS WAF labels at no additional charge.

Phase 2: Free evaluation period (July 27–September 30, 2026)

The existing automatic mitigation and the Anti-DDoS managed rule group run side by side each detecting independently. Automatic mitigation continues to protect your resources while the rule group operates in Count mode. To compare their detection results, use the DDoSAttackRequests metric, AWS WAF labels, and the Anti-DDoS dashboard. All Anti-DDoS managed rule group charges are waived during this period, including the subscription fee, per-request fees, and WCU consumption costs for the eligible web ACLs from phase 1.

Phase 3: Auto-upgrade begins (October 1, 2026)

For eligible web ACLs, the auto-upgrade mirrors your existing automatic mitigation configuration. The rule group inherits your current setting, so a Block configuration comes up in Block mode and a Count configuration comes up in Count mode in a single, atomic operation. The rule group takes over in the same step that disables automatic mitigation, so protection never drops for an instant. This is a handoff rather than a cutover with no window where your resources are unprotected. If you’d rather not upgrade you can opt out by contacting AWS Support before the auto-upgrade date.

Phase 4: Guided migration (available July 27–December 31, 2026)

You don’t have to wait for the October auto-upgrade to migrate. As soon as the rule group is deployed in Count mode between July 27 and August 7, 2026, you can move to it on your own schedule. This is the path to use for web ACLs that aren’t eligible for the Phase 3 auto-upgrade, meaning mixed-mode web ACLs or ones with resources that don’t have automatic mitigation enabled. Work with your AWS account team and AWS Support at any point in this window to plan and complete the migration. Eligible web ACLs are also upgraded automatically starting October 1 (Phase 3), so guided migration is mainly for the web ACLs the auto-upgrade can’t cover.

Phase 5: Shield Advanced application-layer automatic mitigation sunset (January 1, 2027)

As of January 1, 2027, the Shield Advanced application-layer automatic mitigation feature will no longer be available. Resources that haven’t migrated to the Anti-DDoS managed rule group will lose automatic application-layer DDoS mitigation.

Capability	Shield Advanced application layer automatic mitigation	Anti-DDoS managed rule group (AWSManagedRulesAntiDDoSRuleSet)
Feature type	Shield Advanced automatic mitigation	AWS WAF managed rule group
Detection and mitigation speed	Requires a baseline period; mitigation varies per event	Enhanced detection and faster mitigation
Configuration scope	Per resource (Shield API)	Per web ACL (AWS WAF API)
Mitigation actions	Count, Block	Count, Block, and Challenge
Sensitivity controls	None	Low, Medium, and High for both Block and Challenge
Non-HTML path handling	N/A	URI regex exemptions for Challenge
WCU consumption	150 WCUs	50 WCUs
Health checks	Required (Amazon Route 53 health-based detection)	Not required, provides automatic traffic profiling
Availability	Shield Advanced only	AWS WAF and Shield Advanced (see pricing)

Observability

The existing automatic mitigation and the Anti-DDoS managed rule group use separate Amazon CloudWatch namespaces and metric structures. The rule group gives you three tiers of observability: tier 1 tells you an attack is happening, tier 2 shows which requests it flagged and why, and tier 3 shows what it did about them. You don’t need all three on day 1 because most customer teams start at tier 1 to confirm detection is working, then add the others as they tune.

Tier 1: Event detection alarms

You can detect DDoS events using two CloudWatch metrics, each with its own namespace.

	DDoSDetected (Shield)	DDoSAttackRequests (Anti-DDoS managed rule group)
Namespace	AWS/DDoSProtection	AWS/WAFV2
Requires Shield Advanced	Yes	No
Scope	L3, L4, and L7 events	L7 events only
Value during event	Binary (0 or 1)	Count of requests observed
Value outside event	Reported once daily (keeps metric alive)	Absent (no data points)
Dimensions	ResourceArn	Resource, ResourceType

What this means for your existing alarms:

After the application-layer automatic mitigation feature is sunset, DDoSDetected still fires for infrastructure layer 3 and layer 4 events, so your existing network layer and transport layer alarms remain valid. For the full list, see AWS Shield Advanced metrics.
DDoSAttackRequests is the Anti-DDoS managed rule group equivalent for application-layer event detection. Alarm on Sum >= 1 to detect any event, or set a volume threshold (for example, more than 10,000 requests per minute) for severity-based alerting.
During the evaluation period, both metrics fire independently and you can validate detection parity before migrating your application-layer alarms.
Because DDoSAttackRequests is absent when there are no active DDoS events, set treat-missing-data to missing or notBreaching for alarms on this metric.

Tier 2: Detection labels for custom monitoring

Every request the Anti-DDoS managed rule group evaluates gets a label. Where tier 1 tells you an attack started, tier 2 shows which requests looked suspicious and how confident the rule group was. The labels surface as AWS WAF metrics in the AWS/WAFV2 namespace: AllowedRequests, BlockedRequests, and CountRuleMatch. Each carries the LabelName and LabelNamespace dimensions under the awswaf:managed:aws:anti-ddos: namespace.

event-detected – Requests observed during a detected DDoS event
ddos-request – Requests identified as part of the attack
low-suspicion-ddos-request, medium-suspicion-ddos-request, high-suspicion-ddos-request – Graduated suspicion levels
challengeable-request – Requests eligible for browser challenge

Chart suspicion-level trends on a CloudWatch dashboard to see how an attack builds. Match on the labels in your own AWS WAF rules or dig into them in your AWS WAF logs with CloudWatch Logs Insights or Amazon Athena when you need to understand a specific event after the fact.

Tier 3: Mitigation action metrics

Where tier 2 shows what the rule group flagged, tier 3 shows what it did about those requests during an event. You’ll find these metrics as ChallengeRequests, BlockedRequests, and CountRuleMatch, each scoped by the rule label that produced it.

ChallengeAllDuringEvent – Requests challenged during an active event
ChallengeDDoSRequests – Suspected DDoS requests challenged based on suspicion level
DDoSRequests – Requests blocked (or counted in Count mode)

Watch these during a live event to see whether mitigation is keeping up. If you’re challenging far more requests than you’re blocking, your configuration might be too cautious, and you can raise the sensitivity level after you trust the numbers.

Observability summary

Tier	Automatic mitigation	Anti-DDoS managed rule group
Event alarm	DDoSDetected in AWS/DDoSProtection (binary, L3/L4/L7)	DDoSAttackRequests in AWS/WAFV2 (request count, L7)
Detection labels	None	event-detected, ddos-request, suspicion levels, challengeable-request
Mitigation actions	Not visible (Shield-managed rule group metrics not exposed)	ChallengeAllDuringEvent, ChallengeDDoSRequests, DDoSRequests
Dashboard	Shield console event history	Shield console and Anti-DDoS dashboard in the AWS WAF console
Historical analysis	Shield event history only	AWS WAF logs (CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), Amazon Data Firehose)

Billing

Your Shield Advanced subscription includes the Anti-DDoS managed rule group for up to 50 billion requests per month, counted across your whole organization at the payer account level. For most customers that ceiling is well above normal traffic, so you won’t see a line item here unless you’re operating at very high volume. For the exact rates, see AWS WAF pricing and Shield Advanced pricing.

You aren’t charged for DDoS traffic while the Anti-DDoS managed rule group is actively mitigating, which means Block or Challenge mode rather than Count. This applies to AWS WAF request fees, Anti-DDoS managed rule group request fees, and Shield Advanced request charges. Leaving the rule group in Count mode past the evaluation period costs you the protection without the billing relief, so avoid staying in Count mode longer than you need to validate.

During the evaluation period (July 27 to September 30, 2026), the eligible web ACLs AWS auto-enrolled don’t incur per-request fees or WCU consumption, even when configured in Count mode.

The Anti-DDoS managed rule group works at the web ACL level, so every resource you associate with a web ACL shares that coverage. Before assuming a single resource accounts for the whole cost, look at how many resources sit behind each web ACL. A web ACL fronting 20 resources bills differently from one fronting 2, so check that count first and familiarize yourself with the workload protected by each web ACL.

Adding the Anti-DDoS managed rule group to a web ACL yourself isn’t part of the upgrade path, so standard pricing applies from the moment you enable it. The same is true for any resource that was already running the rule group before the rollout. To get the free evaluation, let the automatic rollout reach your web ACLs rather than adding the rule group ahead of it. There’s no penalty for adding it yourself; you just don’t receive the waiver on that web ACL.

Update your infrastructure as code

If you manage web ACLs with AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), Terraform, or other infrastructure as code (IaC), the auto-upgrade changes your infrastructure configuration outside your templates. Your code is still the source of truth, so you need to do two things. First, change where the protection is declared. Today you enable application-layer automatic mitigation through the Shield API (EnableApplicationLayerAutomaticResponse), configured per protected resource. The Anti-DDoS managed rule group is configured through the AWS WAF API instead (CreateWebACL and UpdateWebACL), as a managed rule group statement inside the web ACL, scoped per web ACL rather than per resource. In IaC terms, you remove the Shield automatic-response block (for example, Terraform’s aws_shield_application_layer_automatic_response) and add the WAF managed rule group statement shown in the following section. Second, pull the upgraded web ACL back into your tooling before your next deploy, or your pipeline will try to revert the change.

For the full statement in Terraform, CloudFormation, and the AWS CDK, plus how to sync state after the auto-upgrade (terraform plan, CloudFormation drift detection, cdk diff), see the iac-webacl-examples helper.

Update your AWS Firewall Manager policy

If you run a Shield Advanced policy in AWS Firewall Manager today, that policy is what enabled application-layer automatic mitigation across your accounts. To keep that protection, add the Anti-DDoS managed rule group to an AWS WAF Firewall Manager policy. Your Shield Advanced policy still handles L3 and L4, while the application-layer piece moves to the AWS WAF policy. The migration is straightforward: add or reuse an AWS WAF Firewall Manager policy, put the Anti-DDoS managed rule group in it, and scope it to the same accounts and resources your Shield Advanced policy covers.

You can’t add the rule group from the Shield console or by editing an account-level web ACL directly, because Firewall Manager owns the web ACLs it creates and overwrites local edits. Instead, add the rule group to the AWS WAF policy and Firewall Manager pushes it to every in-scope account.

You can make this change in the console or as code. If you manage your Firewall Manager policies as code, don’t edit them in the console: add a new AWS WAF policy or update an existing one in your templates with the Anti-DDoS managed rule group included, and deploy it using the following Firewall Manager policies using IaC steps. Otherwise, use the console.

In the console, follow Creating an AWS Firewall Manager policy for AWS WAF to create the policy and reach the Edit policy rules page. Add the Anti-DDoS rule group, listed there as AWS AntiDDoS Protection for Layer 7 attacks (AWSManagedRulesAntiDDoSRuleSet), as a new rule group under First rule groups so it evaluates before your other managed groups, but below any Allow custom rules you use to fast-path known-good traffic.

If you protect CloudFront distributions, make this change in your Global policy, and repeat it in each AWS Regional policy for regional resources. Save the policy, and Firewall Manager rolls the change out to in-scope accounts, which can take a few minutes.

After being added, the rule group appears as the first rule group in the policy, as shown in the following screenshot:

Figure 1: AntiDDoS enabled

Firewall Manager policies using IaC

If you manage Firewall Manager policies as code, make the change in your template instead of the console. The Anti-DDoS managed rule group goes into the AWS WAF policy’s ManagedServiceData, a WAFV2 policy definition carried as a JSON string, added to the first rule groups so it evaluates early. For the ManagedServiceData JSON with CloudFormation, Terraform, and AWS CDK examples, see the firewall-manager-examples helper.

Whichever path you take, scope the policy to the same accounts and resources your Shield Advanced policy already covers, so no resource loses application-layer protection during the move.

Getting started

Between July 27 and August 7, 2026, AWS will add the Anti-DDoS managed rule group in Count mode to Shield Advanced web ACLs that have resources using application layer automatic mitigation but not yet the Anti-DDoS rule group. After it reaches your web ACL, you can evaluate it, and migrate whenever you’re ready, without waiting for the October auto-upgrade.

Review the Anti-DDoS dashboard in the AWS WAF console. The dashboard shows real-time DDoS events, match metrics, and top traffic sources.
Compare event detection side by side. During Count mode, both systems detect independently. Check the DDoSDetected metric in AWS/DDoSProtection alongside DDoSAttackRequests in AWS/WAFV2 to validate detection parity for your resources. You can deploy the CloudWatch comparison dashboard from the AWS Samples repository to view both systems on a single dashboard.
Explore AWS WAF labels. Enable AWS WAF logging and query for labels in the awswaf:managed:aws:anti-ddos: namespace. Look at suspicion levels (low-suspicion-ddos-request, medium-suspicion-ddos-request, high-suspicion-ddos-request), event-detected, and challengeable-request to see per-request visibility into detected events.
Start with Low sensitivity for Block actions during evaluation to minimize false positive risk. Tune up as you gain confidence from the Anti-DDoS dashboard and AWS WAF label data.
Plan your configuration. Review sensitivity levels, URI exemptions for non-HTML paths, and web ACL priority placement. The Anti-DDoS managed rule group should run at the highest priority in your web ACL, or right below any custom rules with the Allow action.
Sync your IaC templates. After the auto-upgrade adds the Anti-DDoS managed rule group to your web ACL, fetch the current state into your IaC tooling (Terraform refresh, CloudFormation drift detection, AWS CDK import) before your next deployment.

Conclusion

The Anti-DDoS managed rule group profiles your traffic within minutes and mitigates within seconds, where the automatic mitigation it builds on established its baseline over hours, and it gives you granular visibility into what it’s doing. The evaluation period exists so you can watch both systems run on your own traffic before anything changes. Spend the first few weeks in Count mode confirming the new detection matches what you see today, then move your alarms over and pick a sensitivity level you’re comfortable with. If you run a web ACL across several resources, or you manage rules through AWS Firewall Manager, contact AWS Support before you start so you don’t have to unwind anything later. The Shield Advanced application-layer automatic mitigation feature retires on January 1, 2027, and anything still relying on it needs to be migrated by then.

Resources

If you have feedback about this post, submit comments in the
Comments section below.

Announcing the Cloud Security Alliance on AWS Compliance Guide

2026-07-27 Juan Rodriguez

Post Syndicated from Juan Rodriguez original https://aws.amazon.com/blogs/security/announcing-the-cloud-security-alliance-on-aws-compliance-guide/

AWS Security Assurance Services is announcing the release of the Cloud Security Alliance (CSA) Compliance Guide on Amazon Web Service (AWS), a new resource that maps the 17 control domains and 207 control objectives of the Cloud Controls Matrix v4.1 (CCM) to AWS services and recommended implementation practices. The guide is intended to help organizations using AWS plan, implement, and evidence the controls relevant to their CCM scope, including those pursuing or maintaining CSA STAR certification.

What is the Cloud Controls Matrix?

The Cloud Security Alliance is a not-for-profit organization dedicated to defining and raising awareness of best practices for cloud security. AWS maintains CSA STAR Level 2 certification, which couples the requirements of ISO/IEC 27001:2022 with the CCM. The CSA STAR documentation and the AWS Consensus Assessments Initiative Questionnaire (CAIQ) are available to AWS customers through AWS Artifact.

The CSA Cloud Controls Matrix is a cybersecurity controls framework developed by the CSA that provides a detailed set of security controls mapped across multiple domains (such as audit and assurance, identity and access management, and encryption and key management) specifically designed to assess and manage security risks in cloud computing environments. The CCM is cloud agnostic, designed to be applicable to any cloud service provider or deployment model, such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS), regardless of the underlying technology or vendor, providing universal security controls that organizations can apply across cloud platforms.

Responsibility models

CCM defines its own Shared Security Responsibility Model (SSRM) with three categories: Cloud service provider (CSP)-owned, customer-owned, and shared (independent or dependent). The guide recommends using the SSRM together with the AWS Shared Responsibility Model. For controls that AWS owns, the guide points to AWS attestations available through AWS Artifact (for example, SOC reports, ISO certificates, and the CSA STAR attestation) as inherited evidence. For controls that customers own or share, the guide describes how to implement and evidence them using AWS services. Using a CSA STAR–certified AWS service doesn’t by itself make a customer workload compliant. Customers remain responsible for configuring services, managing access, protecting data, and implementing additional controls based on their environment, risk assessments, and regulatory obligations. The guide is informational and doesn’t replace the AWS compliance documentation and certifications available through AWS Artifact.

What’s inside the guide

For each control, the guide states applicability, describes how organizations can implement the control on AWS, identifies common pitfalls, and lists examples that can be used as evidence during an assessment.

Related resources

Download available: Here

For further assistance, contact AWS Security Assurance Services. If you have feedback about this post, submit comments in the Comments section below.

AWS Weekly Roundup: Local Zone in Athens, Claude Opus 5 on AWS, Lambda durable execution for .NET, and more (July 27, 2026)

2026-07-27 Daniel Abib

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-july-27-2026/

Last week I had the privilege of spending three days in São Paulo with technical builders from across Latin America, brought together for a regional tech event full of deep-dive sessions, hands-on workshops, and conversations with customers and partners. What struck me most wasn’t any single session, it was the energy of a technical community that so rarely gets to be in the same room. People traded architecture ideas over coffee, sketched out solutions on whiteboards, and left with a longer list of things to try than they arrived with. It’s a good reminder that, for all the tooling we build, the community around it is what makes the technology stick.

That community spirit connects nicely to the week’s biggest infrastructure news, which is all about bringing AWS closer to where builders actually are.

Now, let’s get into this week’s AWS news…

Headlines
AWS Local Zone in Athens, Greece: AWS has opened a new Local Zone in Athens, Greece, the second Local Zone in EMEA with support for Amazon S3 and Amazon EBS Local Snapshots, so you can store and process data within Greece to help meet local data residency requirements. The Athens Local Zone supports Amazon EC2 (C7i, M7i, and R7i instances), Amazon S3 with the One Zone-Infrequent Access storage class, Amazon EBS, and Amazon ECS.

Athens, Greece skyline

AWS Local Zones place AWS infrastructure much closer to large population and industry hubs, enabling applications that require single-digit millisecond latency, such as real-time gaming, media production, and financial services, to run where end users actually are. For builders in Greece, you can now run latency-sensitive workloads locally while connecting seamlessly to the nearest AWS Region for services that don’t require low latency, giving you the flexibility to architect hybrid, latency-optimized applications without managing your own data center infrastructure. To learn more, visit AWS Global Infrastructure and Sustainability Blog post.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

Claude Opus 5 on AWS: You can use Anthropic’s Claude Opus 5, the most advanced Opus model yet, matching Claude Fable 5’s top-tier intelligence in many domains at Opus-tier pricing. Amazon Bedrock offers Claude Opus 5 with zero data retention (ZDR) enabled by default, giving you Opus’ top-tier intelligence while meeting your data governance requirements unlike Claude Fable 5. You have two ways to access Claude Opus 5: Amazon Bedrock and Claude Platform on AWS. To learn more, visit the deep dive blog post.
AWS Lambda durable execution SDK for .NET is now generally available: You can now build resilient, long-running workflows in C# using Lambda durable functions, without implementing custom progress tracking or integrating an external orchestration service. The SDK is a natural fit for multi-step applications like payment processing pipelines, AI agent orchestration, and human-in-the-loop approvals, it checkpoints progress automatically and can pause execution for up to a year. If you’re a .NET developer building serverless workflows, this removes a lot of the plumbing you used to write by hand.
Amazon Bedrock AgentCore now delivers unified observability with traces and logs in a single log group: Amazon Bedrock AgentCore now delivers agent traces and prompts to the same Amazon CloudWatch log group as your agent’s logs. Previously, telemetry was split across destinations, trace spans went to a shared log group while prompts, inputs, and outputs went to a separate one, so debugging a single agent invocation meant searching in multiple places. You can now debug an invocation in one place, and apply fine-grained access control and customer-managed key (CMK) encryption at the individual agent level.
Amazon Connect delivers more natural agentic voice experiences: Amazon Connect now supports more natural, human-sounding agentic voice experiences across 50+ languages, including Portuguese, Spanish, French, Italian, Japanese, Korean, and Thai, with over 100 new voice options and conversational improvements that make AI interactions sound more fluid. Connect’s agentic self-service lets AI agents understand, reason, and take action across voice and digital channels, adapting to a customer’s tone and sentiment. You can now build contact center experiences that feel natural to callers in far more of the languages your customers actually speak.
Amazon SageMaker Unified Studio now supports Amazon OpenSearch: You can now query and analyze your search and log analytics data from Amazon OpenSearch directly alongside other data assets in Amazon SageMaker Unified Studio. With this connection, you can combine operational search data in OpenSearch with data from sources like Amazon Redshift, Amazon S3, and relational databases, all within a single, governed environment. It’s especially useful when you need to correlate analytical and operational workloads, such as joining application logs with transactional data to uncover insights.
Amazon CloudWatch announces coding agent insights: Amazon CloudWatch now gives engineering leaders visibility into how AI coding tools are driving value across their organization. Coding agent insights integrates with the Claude apps gateway for AWS to collect telemetry from Claude Code without additional instrumentation, and also supports agents like Codex and GitHub Copilot. As teams scale AI coding adoption, you can now measure the return on that investment with metrics built on OpenTelemetry, no custom instrumentation required.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

Evaluating AI Agents: A production blueprint with Strands and AgentCore: A practical guide to evaluating AI agents before and after they reach production, using Strands Agents and Amazon Bedrock AgentCore. If you’re moving agents from prototype to production, this post is a great companion to the AgentCore observability update above, it walks through how to measure agent quality systematically rather than by gut feel.
Building multi-region resiliency for AWS CloudFormation custom resource deployment: Learn how to architect CloudFormation custom resources for multi-region resiliency, so your infrastructure-as-code deployments stay reliable even when a single Region has issues.
Introducing Amazon Simple Email Service (SES) pricing plans: Amazon SES now offers pricing plans that give you more predictable costs as your email volume grows. If you send at scale, this could simplify your billing significantly.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

AWS Summits: AWS Summits are free events that bring the cloud and AI community together to connect, learn, and explore the latest technologies. Browse the full calendar to find a Summit near you in the second half of 2026.
AWS Community Days: Community-led conferences where content is planned, sourced, and delivered by community leaders. If you’re in Latin America, don’t miss AWS Community Day Belo Horizonte on August 22, registration is open at awscommunityday.com.br.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Enterprise security at machine speed: AWS Black Hat 2026 preview

2026-07-23 Rahul Sahni

Post Syndicated from Rahul Sahni original https://aws.amazon.com/blogs/security/enterprise-security-at-machine-speed-aws-black-hat-2026-preview/

Black Hat 2026 (Aug 1-6, 2026) brings together over 22,000 security practitioners, researchers, and CISOs who build, break, and defend enterprise infrastructure. They’re security professionals who push the limits of offensive and defensive security and demand proof over promises. As frontier security models like Mythos reshape the enterprise landscape, they need security that operates at the same speed as the events they face. This August, Amazon Web Services (AWS) returns to Las Vegas to meet with our customers and partners to show how we’re delivering enterprise security at machine speed.

At Black Hat USA 2026, connect with AWS through live demos, a practitioner session on autonomous security operations, and an executive roundtable on building durable AI security architectures, plus networking receptions with customers and partners. Here’s where to find us and what you’ll take away.

Experience AWS security innovation in action

Visit us at Booth #1648 to explore five interactive demo pedestals, each aligned to a pillar of our AI-powered security story:

Post-Mythos enterprise security: AWS uses multiple frontier AI reasoning models to find issues, validate exploitability, and autonomously fix exposures, compressing mean-time-to-remediation from days to minutes. Learn how AWS Continuum helps you find risks before you ship, prioritize by real business impact, and remediate at machine speed while you stay in control.
AI-powered investigations: Amazon GuardDuty AI-powered investigations automatically analyze findings and the accounts around them to separate true threats from benign activity at scale. Drawing on log activity, resource configuration, internet reachability, and historical findings, it delivers triage reports that match expert-level accuracy, freeing your team to focus on what matters.
Purpose-built AI workload security: AWS extends the enterprise security controls you already trust to address the unique demands of AI workloads. Amazon Bedrock helps you build and deploy generative AI applications, with hundreds of top foundation models plus built-in safety controls, guardrails, and evaluation tools. Amazon Bedrock AgentCore Identity keeps you in full control of what your AI agents can access.
Full-stack multicloud security: AWS Security Hub Extended delivers full-stack security by integrating AWS services with 21 curated partner solutions across nine categories, from endpoint and identity to AI, with pay-as-you-go pricing, one console, and one bill. All findings flow in OCSF format with zero integration work, and a risk correlation engine traces paths across multicloud so your team fixes root causes rather than chasing symptoms.
AWS Partner solutions: A dedicated partner demo pedestal runs live demonstrations from select AWS Partners, showing how their solutions integrate with AWS to address your most critical security challenges. With over 25 partners rotating throughout Business Hall hours, you get a firsthand view of how these integrations work together to strengthen your security posture.

Theater sessions: Want to go deeper on a specific topic without committing to a full session? During expo hours, our theater hosts 15-minute AWS and partner talks curated around the challenges enterprise security teams face today. Topics range from enabling AI adoption securely, to unifying visibility across clouds, to automating response. Subject matter experts from AWS and our partners lead each session and give you insights you can bring back to your team. Featured sessions include:

From finding to fix at machine speed with AWS Continuum
Securing the agentic AI stack
Architecting defense-in-depth for AI workloads
Network security strategies for the post-Mythos era
One console for unified full-stack security

For the latest theater session schedule, including dates and times, visit AWS at Black Hat 2026.

Engagement Zone: Take a quick break and test your detection skills in a gamified, hands-on challenge. Race the clock to resolve security issues and collect power-ups that speed your mission to secure the environment. Rack up your score, lock in your percentile rank, and see how you stack up against every other player.

Beyond the booth: Sessions and executive roundtable

AWS security experts share insights at speaking sessions and host an executive roundtable at Black Hat USA 2026.

Speaking Session | Machine-Speed Defense: Building an Autonomous Security Operations Loop for the AI Era
Wednesday, August 5, 10:15–10:35 AM PDT
Pulse Stage 2
In a post-Mythos world, organizations need to use AI-powered reasoning to discover, correlate, validate, and remediate security exposures at machine speed. This session provides a blueprint for building an autonomous security operations loop, using business-context graphs and sandbox validation, to unlock machine-speed defense as a durable competitive edge.

Executive Roundtable | Autonomous Defense at Cloud Scale: Critical Choices for Security Leaders Today
Wednesday, August 5, 11:00 AM–12:00 PM PDT
Breakers F
As agentic AI and frontier models reshape the threat landscape, foundational security remains essential, and organizations that layer autonomous workload governance on top can turn AI into a true security force multiplier. This executive roundtable explores how to build a durable, adaptive security harness that accounts for the real economics of AI-powered defense and evolves alongside rapidly advancing models and techniques.

Briefing | ThreatForest: Automated Attack Trees from Source Code
Thursday, August 6, 2:35–3:15 PM PDT
Jasmine, Level 3
Threat modeling is critical, but manual processes can’t keep pace with cloud-focused architectures. This briefing will dive into how ThreatForest uses six specialized AI agents to automatically analyze source code and produce validated attack trees mapped to MITRE ATT&CK with actionable mitigations. Attendees walk away with the open source tool and a reusable multi-agent architecture pattern.

AWS activities and events

Beyond the expo floor, AWS hosts a portfolio of ancillary events built for focused conversations and networking across the security community. Join us at:

Global Startup Program Reception: For the emerging customer and builder community
AWS Customer & Partner Appreciation Event: Welcome reception honoring the customers and partners we’re proud to work alongside

Security LIVE! broadcast

Catch Security LIVE! on-site at Mandalay Bay on Wednesday, August 5 and Thursday, August 6. This infotainment-style broadcast brings AWS and AWS Partners together to solve real security challenges for customers across 20-minute conversational segments, covering everything from data protection and compliance to application and perimeter security. With over 30 segments, Security LIVE! showcases the breadth of the AWS security partner ecosystem. Stop by the set to watch it unfold live.

Join us in Las Vegas

Whether you’re exploring how to secure AI workloads, adopting autonomous remediation as frontier models like Mythos reshape the landscape, or unifying security across domains and clouds, the AWS team at Black Hat USA 2026 is ready to help.

Learn more about AWS Security solutions at AWS Cloud Security.

See you in Las Vegas, August 3–6, 2026.

2026 ISO and CSA STAR certificates are now available with two additional services

2026-07-20 Atulsing Patil

Post Syndicated from Atulsing Patil original https://aws.amazon.com/blogs/security/2026-iso-and-csa-star-certificates-are-now-available-with-two-additional-services/

Amazon Web Services (AWS) successfully completed an onboarding audit with no findings for ISO 9001:2015, 27001:2022, 27017:2015, 27018:2019, 27701:2019, 20000-1:2018, and 22301:2019, and Cloud Security Alliance (CSA) STAR Cloud Controls Matrix (CCM) v4.0. EY Certify Point auditors conducted the audit and reissued the certificates on May 31, 2026. The objective of the audit was to enable AWS to expand their ISO and CSA STAR certifications to include two additional services. The ISO standards cover areas including quality management, information security, cloud security, privacy protection, service management, and business continuity. The certifications demonstrate AWS’s commitment to maintaining robust security controls and protecting customer data across our services.

During this onboarding audit, we added two additional AWS services to the scope since the last certification issued on February 25, 2026.
Following are the two additional services:

For a full list of AWS services that are certified under ISO and CSA Star, see the AWS ISO and CSA STAR Certified page. Customers can also access the certifications in the AWS Management Console through AWS Artifact.

If you have feedback about this post, submit comments in the Comments section below.

Zero Copy access to Apache Iceberg tables in Amazon S3 from Salesforce Data 360 using the Iceberg REST endpoint from AWS Glue Data Catalog

2026-07-15 Avijit Goswami

Post Syndicated from Avijit Goswami original https://aws.amazon.com/blogs/big-data/zero-copy-access-to-apache-iceberg-tables-in-amazon-s3-from-salesforce-data-360-using-the-iceberg-rest-endpoint-from-aws-glue-data-catalog/

Companies increasingly need to query and analyze data across platforms without the cost and complexity of moving it. Salesforce and AWS have collaborated to make this possible by providing Zero Copy access to Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) directly from Salesforce Data 360, using the Iceberg REST endpoint from AWS Glue Data Catalog with data access managed by AWS Lake Formation. This integration helps customers federate their Amazon S3 data lakes with Data 360, preserving data governance, freshness, and business semantics without replication.

Zero Copy file federation plays an important role in activating applications and experiences. By removing the need to physically move or copy data, and connecting to data at the storage level, it addresses key challenges including:

Cost efficiency – Reduce storage duplication costs and minimize the compute resources required for data pipelines.
High scale – Access data with near-native performance at scale through in-Region access.
Enhanced agility – Access and analyze data in real time, accelerating time-to-insight and supporting faster response to evolving business needs.
Streamlined operations – Remove the complexity of building and maintaining intricate data pipelines, clearing up valuable data engineering resources.

In this post, we demonstrate how AWS and Salesforce customers can access their enterprise data lakes on AWS from Data 360 using Zero Copy file federation.

What is Data 360?

Data 360 is the real-time data engine that activates trusted context across the entire Salesforce platform. It connects all your enterprise data — data warehouses, data lakes, third-party signals, and more — to the business context, logic, and governance that already live in Salesforce, without moving or copying it. With Zero Copy federation, your teams and AI agents always operate from a complete, current, and trusted picture of your business in the moment it’s needed. It serves as the essential system of context for Agentforce, enabling agents to reliably get real work done.

What is Apache Iceberg?

Apache Iceberg is a high-performance, open table format for huge analytic datasets that brings the reliability and simplicity of SQL tables to big data. It’s a thriving open source project under the Apache Software Foundation. Data engineers use Apache Iceberg because it’s fast, efficient, and reliable at any scale and keeps records of how datasets change over time. Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more.

Why Amazon S3 for Apache Iceberg data lakes?

Amazon S3 is regarded as the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities, and its ability to integrate with a broad portfolio of AWS and third-party tools for data ingestion and processing. Apache Iceberg was designed and built to interact with Amazon S3, and provides support for many Amazon S3 features as listed in the Iceberg documentation.

What is Zero Copy file federation?

File federation, also termed catalog federation, uses the Data Catalog to communicate with remote catalog systems to discover catalog objects and to authorize access to their data in Amazon S3. When you query a remote Iceberg table, the Data Catalog discovers the latest table information in the remote catalog at query runtime, getting the table’s Amazon S3 location, current schema, and partition information. Your analytics engine then uses this information to access Iceberg data files directly from Amazon S3, and Lake Formation manages access to the table and data by vending scoped credentials to the table data stored in Amazon S3. This approach avoids metadata and data duplication while providing real-time access to remote Iceberg tables through your preferred AWS analytics engines.

Solution overview

Apache Iceberg file federation lets Data 360 directly query data stored in Amazon S3 without copying or moving the data. This Zero Copy approach provides several benefits:

Real-time access to Amazon S3 data from Salesforce.
Reduced data movement and storage costs.
Simplified data architecture.
Improved data freshness.

The following diagram illustrates the architecture of the integration between Data 360 and Amazon S3 using Apache Iceberg file federation.

Key components:

Amazon S3 stores the source data in Apache Iceberg format.
AWS Glue Data Catalog maintains the metadata for Iceberg tables.
AWS Glue Iceberg REST endpoint provides RESTful access to Iceberg tables.
AWS Lake Formation manages metadata and underlying data access for Amazon S3-based data lakes.
Data 360 processes and analyzes the data.
Apache Iceberg connector provides direct access to query Amazon S3 data from Salesforce.

Walkthrough

The following walkthrough shows you how to set up Zero Copy file federation.

Prerequisites

Before you begin, you need the following:

An AWS account.
A data-lake-admin role following the steps in Lake Formation personas and IAM permissions reference, with access to AWS Identity and Access Management (IAM), AWS Glue Data Catalog, AWS Lake Formation, Amazon S3, and Amazon Athena.
A Data 360 instance.
Familiarity with AWS and Data 360.
A user-defined IAM role that Lake Formation will use to register the bucket. To create one, see Requirements for roles used to register locations.

Configure your AWS environment

Set up an Amazon S3 bucket and Iceberg table

Open the Amazon S3 console.
Choose Create bucket to create a bucket.
For Bucket type, choose General purpose, provide a Bucket name, and choose Create bucket.
In the bucket, create two prefixes by choosing Create folder.
Name the prefixes athena_iceberg and athena_results.
Inside the athena_iceberg prefix, create another prefix named customer_iceberg.

Create an Iceberg table using Athena

Open the Amazon Athena console.
Choose Query your data in Athena console, then choose Launch query editor.
In Athena, choose Edit settings.
Set s3://<your-bucket-name>/athena_results/ as the Location of query result, then choose Save. Replace <your-bucket-name> with your bucket name.
Choose Editor to return to the query editor page.
To create the database, copy the following query into the query editor and choose Run. You need to be in the Athena Query Editor to run the following commands.
```
create database iceberg_db;
```

To create the Iceberg table, copy the following query into the query editor, replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table, and choose Run.

CREATE TABLE iceberg_db.churn (
    state string,
    account_length int,
    area_code string,
    phone string,
    intl_plan string,
    vmail_plan string,
    vmail_message int,
    day_mins double,
    day_calls int,
    day_charge double,
    eve_mins double,
    eve_calls int,
    eve_charge double,
    night_mins double,
    night_calls int,
    night_charge double,
    intl_mins double,
    intl_calls int,
    intl_charge double,
    custserv_calls int,
    churn boolean)
LOCATION 's3://<s3 bucket location>/iceberg/churn'
TBLPROPERTIES (
    'table_type'='iceberg',
    'compression_level'='3',
    'format'='PARQUET',
    'write_compression'='ZSTD'
);

Insert some records into the table.

-- Sample data insert for "iceberg_db"."churn"
-- Execute this in the Athena console
INSERT INTO "iceberg_db"."churn" VALUES
('KS', 128, '415', '382-4657', 'no', 'yes', 25, 265.1, 110, 45.07, 197.4, 99, 16.78, 244.7, 91, 11.01, 10.0, 3, 2.70, 1, false),
('OH', 107, '415', '371-7191', 'no', 'yes', 26, 161.6, 123, 27.47, 195.5, 103, 16.62, 254.4, 103, 11.45, 13.7, 3, 3.70, 1, false),
('NJ', 137, '415', '358-1921', 'no', 'no', 0, 243.4, 114, 41.38, 121.2, 110, 10.30, 162.6, 104, 7.32, 12.2, 5, 3.29, 0, false),
('OH', 84, '408', '375-9999', 'yes', 'no', 0, 299.4, 71, 50.90, 61.9, 88, 5.26, 196.9, 89, 8.86, 6.6, 7, 1.78, 2, false),
('OK', 75, '415', '330-6626', 'yes', 'no', 0, 166.7, 113, 28.34, 148.3, 122, 12.61, 186.9, 121, 8.41, 10.1, 3, 2.73, 3, false),
('AL', 118, '510', '391-8027', 'yes', 'no', 0, 223.4, 98, 37.98, 220.6, 101, 18.75, 203.9, 118, 9.18, 6.3, 6, 1.70, 0, false),
('MA', 121, '510', '355-9993', 'no', 'yes', 24, 218.2, 88, 37.09, 348.5, 108, 29.62, 212.6, 118, 9.57, 7.5, 7, 2.03, 3, false),
('MO', 147, '415', '329-9001', 'yes', 'no', 0, 157.0, 79, 26.69, 103.1, 94, 8.76, 211.8, 96, 9.53, 7.1, 4, 1.92, 0, false),
('WV', 141, '415', '330-8173', 'yes', 'yes', 37, 258.6, 84, 43.96, 222.0, 111, 18.87, 326.4, 97, 14.69, 11.2, 5, 3.02, 0, false),
('IN', 65, '415', '329-6603', 'no', 'no', 0, 129.1, 137, 21.95, 228.5, 83, 19.42, 208.8, 111, 9.40, 12.7, 6, 3.43, 4, true),
('RI', 74, '415', '344-9230', 'no', 'no', 0, 187.7, 127, 31.91, 163.4, 148, 13.89, 196.0, 94, 8.82, 9.1, 5, 2.46, 0, false),
('IA', 168, '408', '363-1107', 'no', 'no', 0, 275.8, 90, 46.89, 230.0, 73, 19.55, 191.3, 57, 8.61, 9.9, 3, 2.67, 4, true),
('MT', 95, '510', '394-8006', 'no', 'no', 0, 113.2, 96, 19.24, 269.9, 107, 22.94, 229.1, 87, 10.31, 7.1, 4, 1.92, 1, false),
('NY', 62, '415', '371-5765', 'no', 'no', 0, 236.5, 127, 40.21, 145.3, 101, 12.35, 225.0, 103, 10.13, 12.0, 1, 3.24, 5, true),
('TX', 109, '408', '356-2992', 'no', 'yes', 33, 190.7, 114, 32.42, 218.2, 111, 18.55, 156.5, 122, 7.04, 11.6, 5, 3.13, 1, false),
('CA', 155, '510', '328-8230', 'no', 'no', 0, 197.3, 78, 33.54, 160.2, 86, 13.62, 280.1, 90, 12.60, 8.8, 2, 2.38, 2, false),
('WA', 132, '415', '382-1011', 'yes', 'no', 0, 302.7, 67, 51.46, 212.0, 105, 18.02, 265.5, 82, 11.95, 10.3, 4, 2.78, 3, true),
('FL', 88, '408', '344-5678', 'no', 'yes', 18, 145.3, 95, 24.70, 187.6, 92, 15.95, 198.2, 108, 8.92, 9.4, 3, 2.54, 0, false),
('CO', 201, '510', '367-4321', 'no', 'no', 0, 312.5, 142, 53.13, 178.9, 76, 15.21, 145.7, 95, 6.56, 14.2, 8, 3.83, 6, true),
('GA', 56, '415', '390-2244', 'no', 'yes', 12, 178.4, 101, 30.33, 205.1, 119, 17.43, 230.8, 100, 10.39, 8.0, 2, 2.16, 1, false);

Register the bucket with Lake Formation in Lake Formation mode

To use Lake Formation permissions for access control to the churn table, you must register the location. To do that, complete the following actions:

Open the AWS Lake Formation console.
In the navigation pane under Administration, choose Data lake locations.
Choose Register location and enter the following information:
1. For S3 URI, enter s3://<s3 bucket location>/iceberg/churn. Replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table.
2. For IAM role, choose the user-defined IAM role that you created in the prerequisites.
3. For Permission mode, choose Lake Formation.
Choose Register location.

Enable third-party integration in Lake Formation

From the Lake Formation console, enable full table access for external engines.

Open the AWS Lake Formation console.
On the left pane, expand the Administration section.
Choose Application integration settings and select Allow external engines to access data in Amazon S3 locations with full table access.
Choose Save.

Application integration settings page in the Lake Formation console with full table access enabled for external engines

Set up an IAM user for third-party access

Open the IAM console.

From the left navigation menu, choose Policies, then choose Create policy. Choose JSON and paste the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "lakeformation:GetDataAccess",
            "Resource": "*"
        }
    ]
}

Choose next, provide a name for the policy, and choose Create policy.
From the left navigation menu, choose Users, then choose Create user.
For username, enter data_cloud_user, choose next, and choose Attach policies directly.
Choose AWSGlueServiceRole and the policy that you created in step 3. Choose next and Create user.
Choose the user, then choose Security credentials to create an access key.
Scroll down and choose Create access key, choose Applications running outside AWS, and choose Create access key.
Copy the access key and secret access key, and save them securely. You need these to configure the connector in Data 360.

Set up Lake Formation resource permissions for third-party data access

Open the Lake Formation console.
From the left navigation under Data Catalog, choose Databases, then choose the athena_iceberg_db database.
From the Actions menu, choose Permissions, Grant.
In Principals, choose IAM users and roles, and from the menu choose data_cloud_user, which you just created in IAM.
Scroll down to grant permissions by choosing All tables, then choose Select and Describe permissions for the tables.
Choose Grant to apply the permissions.

Set up Apache Iceberg file federation in Data 360

Create and configure the connection

Navigate to Salesforce Setup. For instructions, see Set Up the AWS Glue Data Catalog Connection.
In Data Cloud, choose Setup, then choose Data Cloud Setup.
Under External Integrations, choose Other Connectors.
Choose New.
On the Source tab, choose AWS Glue Data Catalog, then choose Next.
Complete the following information shown in the following screen:
1. In the Authentication Details section, enter the AWS access key ID and AWS secret access key for the IAM user. Make sure that the IAM user has a policy that grants the user read-only access to AWS Glue Data Catalog. Use Lake Formation to configure storage credential vending. This approach is for AWS Glue Data Catalog to vend temporary credentials at run time so that Data 360 can access the underlying storage bucket.
2. For Catalog URL, enter the URL of AWS Glue Data Catalog. See Connecting to the Data Catalog by using AWS Glue Iceberg REST endpoint.
3. For Catalog ID, enter the 12-digit AWS account ID linked to AWS Glue Data Catalog.
4. For Signing Region, enter the host AWS Region where AWS Glue Data Catalog is located.
5. For Signing Service, enter glue. Data 360 requires the Signing Service, in addition to the AWS access key ID, secret access key, and Signing Region, to sign requests to AWS Glue Data Catalog by using AWS Signature Version 4.
6. Test the connection and check for the success message.
7. Save the connection details.
After the configuration is complete and saved, the new AWS Glue Data Catalog connection shows up with “Active” status in the Connectors screen.

Create and configure the data stream

In Data Cloud, on the Data Streams tab, choose New.
Under Other Sources, choose the AWS Glue Data Catalog source, then choose Next.
From the menus, choose the connection that you just set up, choose a database in your AWS Glue catalog where you have an Iceberg table, choose the table that you want to stream, and choose Next.
Enter the object name and object API name. For more information, see Data Lake Object Naming Standards.
Choose the category to specify the type of data to ingest. For more information, see Category.
Choose a primary key to uniquely identify the incoming records. For more information, see Primary Key.
Choose the source fields you want to ingest, then choose Next. Fields with convertible data types are listed under Supported Fields.
Choose the relevant data space. Choose “Default” if you don’t have any other data space provisioned in your org. For more information, see Data Spaces.
Choose Deploy.
After the setup is complete, the new data stream appears in your Data Cloud environment.
The data stream is ready. You can now go to the Data Explorer in your Data Cloud environment and start viewing the Iceberg tables that reside in your external AWS account.

Best practices and considerations

Use IAM roles with least-privilege access. Grant only the specific permissions each service or user needs.
Implement appropriate Amazon S3 bucket policies. Define bucket-level policies that restrict access by AWS account, VPC endpoint, or IP range.
Monitor access patterns. Enable Amazon S3 server access logging or AWS CloudTrail data events to track who reads from and writes to your table buckets.
Optimize Iceberg table partitioning. Choose partition keys that align with your most common query filters.
Consider data access patterns. Design your table layout around how data is actually queried.
Implement lifecycle policies for Amazon S3 objects. Configure Amazon S3 lifecycle rules to transition older data files to other storage classes.
Use appropriate Iceberg file compaction strategies. Run compaction regularly to merge small files produced by streaming or frequent batch appends.
Monitor data transfer costs. Track cross-Region and internet egress charges using AWS Cost Explorer as applicable.

Clean up

After you finish testing, clean up all the resources in your AWS account that you created (including the Amazon S3 bucket, Athena tables, and other AWS services) to avoid recurring costs.

Conclusion

By implementing Apache Iceberg file federation between Data 360 and Amazon S3, you can create a more efficient and streamlined data architecture. This solution gives you real-time access to Amazon S3 data while using the analytics capabilities of Data 360. As businesses continue to prioritize data-driven decision-making, Zero Copy data sharing plays an important role in unlocking the full potential of customer data across platforms.

To learn more, review the following resources:

About the authors

ICYMI: June 2026 @AWS Security

2026-07-15 Rodolfo Brenes

Post Syndicated from Rodolfo Brenes original https://aws.amazon.com/blogs/security/icymi-june-2026-aws-security/

Read all about the latest AWS security features, compliance updates, and hands-on resources in our new, monthly digest posts. You’ll find expert blog posts, new service capabilities, code samples, and workshops.

AWS Security Blog posts

This month’s AWS Security Blog posts covered identity and access management, threat intelligence, network security, AI-powered security tooling, and multi-account governance. Read on for guidance on restricting console access to expected networks, securing multi-tenant AI agents, preventing data exfiltration, and managing organization-scale migrations.

Identity

Secure multi-tenant AI agents with Amazon Bedrock AgentCore resource-based policies
Authors: Satyen Verma, Satveer Khurpa, Prajit Pabbati, Vijay Kumar Samanthapudi, Zohreh Norouzi | Published: June 2, 2026
Learn to use resource-based policies on Amazon Bedrock AgentCore to grant cross-account access for one tenant while restricting another to VPC-only traffic in a shared multi-tenant AI platform.

Customize federated sign-in with new Amazon Cognito Lambda trigger
Authors: Abrom Douglas | Published: June 4, 2026
Learn to use the new inbound federation AWS Lambda trigger for Amazon Cognito to transform, filter, and enrich user attributes from external identity providers before profile creation in your user pool.

Amazon Cognito unlocks advanced capabilities with next-generation infrastructure
Authors: Howie Li, Georgi Baghdasaryan | Published: June 4, 2026
Amazon Cognito introduced high-throughput performance, customer-managed keys for data encryption at rest, and multi-Region replication for business continuity, built on a new storage infrastructure migrated with zero downtime.

Building secure B2C applications with fine-grained access control using Amazon Cognito and Amazon Verified Permissions
Authors: Sowmya Vemuri | Published: June 5, 2026
Learn to build fine-grained access controls for a Streamlit application using Amazon Cognito for authentication and Amazon Verified Permissions with Cedar policies for authorization.

Restrict AWS Management Console access to expected networks with sign-in resource-based policies and RCPs
Authors: Swara Gandhi, Rishi Tripathy | Published: June 24, 2026
Learn to use sign-in resource-based policies and resource control policies to restrict AWS Management Console sign-in to requests from expected networks, such as corporate VPNs and Amazon VPC endpoints.

Infrastructure security

Gain visibility into DDoS attacks with flow logs in AWS Shield Advanced
Authors: Ken Kitts | Published: June 4, 2026
Learn to configure AWS Shield Advanced attack flow logs to capture traffic metadata during DDoS events, pinpoint sources, verify mitigations, and feed your existing analysis pipelines.

Threat tactic spotlight: Subdomain takeover
Authors: Matt Gurr, Ariam Michael, Geoff Sweet, Luis Pastor | Published: June 16, 2026
Learn to detect and prevent subdomain takeover using AWS Config custom rules to identify dangling DNS CNAME records pointing to deleted resources in globally shared namespaces

Prevent data exfiltration: AWS egress controls for cloud workloads
Authors: Meriem Smache, Maxim Raya | Published: June 22, 2026
Learn to implement layered egress detection and protection using AWS Network Firewall, Amazon Route 53 Resolver DNS Firewall, and data perimeters to help reduce unauthorized data transfer risk.

Detection and incident response

Operationalizing AWS security: A maturity roadmap
Authors: Joseph Sadler | Published: June 8, 2026
A six-phase maturity roadmap for organizations that have already enabled AWS Security Hub and Amazon GuardDuty, covering tuning, notifications, automated remediation, and operational cadence.

Introducing AWS Continuum: Security at machine speed
Authors: Chet Kapoor | Published: June 17, 2026
AWS Continuum for code vulnerabilities is an AI-native platform that addresses the full lifecycle of a code vulnerability at machine speed—from discovery and prioritization through validation and remediation.

Accelerate security investigations with Kiro CLI
Authors: Sibasankar Behera, Marshall Jones | Published: June 18, 2026
Learn to use Kiro CLI to conduct security investigations following the AWS Security Incident Response Guide framework, from triaging Amazon GuardDuty findings through containment and evidence preservation.

What the June 2026 Threat Technique Catalog update means for your AWS environment
Authors: Shannon Brazil, Cydney Stude, Javier Teitelbaum | Published: June 29, 2026
Learn about five new entries and three updates to the Threat Technique Catalog for AWS, covering container security, organization-level trust, and compute hijacking patterns observed by AWS CIRT.

Data protection

Identify unused AWS KMS keys and prevent accidental key deletions
Authors: Andrea Rossi, Poojil Tripathi | Published: June 2, 2026
Learn to use the new AWS KMS GetKeyLastUsage API to audit key activity, identify unused keys, and apply policy controls that prevent accidental deletion of recently used keys.

Governance and compliance

From Monolith to Multi-Account: Pinterest’s AWS Organization Transformation Journey
Authors: Sid Vantair, James Fogel, Jeremy Talis | Published: June 4, 2026
Learn how Pinterest migrated from a single monolithic AWS account to a multi-account architecture, including management account separation, automated account provisioning, and centralized networking.

Build a Multi Account Patch Compliance Dashboard with Kiro Specs
Authors: Justin Thomas | Published: June 9, 2026
Learn to use Kiro’s spec-driven development approach to build a serverless multi-account patch compliance dashboard with AWS Systems Manager Patch Manager and private access via AWS Systems Manager Session Manager.

Transfer AWS accounts between AWS Organizations while preserving AWS Lake Formation permissions
Authors: Alex Torres, Aarthi Srinivasan, Ryan McNamee | Published: June 12, 2026
Learn to migrate member accounts between AWS Organizations without disrupting AWS Lake Formation cross-account permissions by using temporary bridge shares with the new AWS RAM retention parameter.

June Security Bulletins

Investigations of reported security vulnerabilities affecting Amazon and AWS services, software, and products.

AWS Samples

This month brings 10 new AWS samples spanning AI security, identity, infrastructure security, governance, and observability. From securing Amazon Bedrock AgentCore agents with AWS WAF to building graph-based CMDBs for dependency analysis, these repositories help you implement security and governance best practices across your AWS environment.

AI Security

Securing Amazon Bedrock AgentCore Runtime with AWS WAF
Learn to place AWS WAF in front of Amazon Bedrock AgentCore Runtime using two architecture patterns with an Application Load Balancer and VPC endpoints for defense-in-depth protection.

AI Security Posture Management (AI SPM) on AWS
Learn to discover, assess, and protect AI agents running in your environment using AWS-native services across three pillars: observe, govern, and defend, with rules mapped to OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS.

Identity

Implement On-Behalf-Of (OBO) token exchange for Amazon Bedrock AgentCore agents
Learn to implement secure identity propagation from user context through agent chains using Amazon Bedrock AgentCore Identity OBO token exchange without credential sharing.

Infrastructure Security

Monetizing AI traffic with Amazon CloudFront and AWS WAF (x402)
Learn to monetize AI traffic at the edge using AWS WAF native x402 protocol support with Amazon CloudFront, settling payments in a single round-trip with no Lambda@Edge required.

Governance

Turn repeatable processes into SOPs your AI agent works through one auditable step at a time
Learn to use this MCP server to hand standard operating procedures to your AI agent one step at a time, with gated execution and auditable progress tracking.

Bedrock Ops Lens
Learn to deploy an Amazon Bedrock observability dashboard in your own account for per-account, per-model, and per-tag cost attribution, quota tracking, latency monitoring, and model lifecycle management, with an MCP server for IDE access.

AWS Agent Registry sample demo application
Learn to use AWS Agent Registry to publish, review, approve, and discover AI agents, MCP servers, and agent skills through a centralized catalog with governance workflows.

Graph CMDB — an AWS resource relationship graph
Learn to build a graph-based CMDB from AWS Config, AWS IAM, Amazon EKS, AWS CloudTrail, and VPC Flow Logs to visualize resource dependencies, investigate scope of impact, and surface governance findings.

OpenAI Codex through Amazon Bedrock — Usage governance with LiteLLM
Learn to centrally govern, administer, and monitor OpenAI Codex access for engineering teams using Amazon Bedrock as the inference backend with per-user budgets, rate limits, and audit trails via LiteLLM.

Agentic Data Governance — the context ladder
Learn to measure how governance context layers — data dictionaries, semantic metrics, and execution skills — improve data agent accuracy through a four-level ablation ladder on the BIRD benchmark.

Conclusion

June 2026 provides guidance and examples for operationalizing security at organizational scale; from maturity roadmaps and console access restrictions to AI agent registries and posture management platforms. The posts and samples provide patterns for DDoS visibility with flow logs, multi-tenant agent isolation with resource-based policies, egress controls for data exfiltration prevention, and governance frameworks for AI coding assistants. Each resource includes deployment steps or runnable code so you can validate in your own environment before adopting. Subscribe to the AWS Security Blog RSS feed to receive updates as they publish, and revisit this digest monthly for a consolidated view of what changed and what to act on.

If you have feedback about this post, submit comments in the Comments section below.

Amazon SQS turns 20: Two decades of reliable messaging at scale

2026-07-13 Esra Kayabali

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/amazon-sqs-turns-20-two-decades-of-reliable-messaging-at-scale/

On July 13, 2006, we launched Amazon Simple Queue Service (Amazon SQS) as one of the first three services available to customers, alongside Amazon EC2 and Amazon S3. We had learned firsthand that distributed systems need a reliable way to pass messages between components without creating tight dependencies. If one service called another directly and that service was slow or unavailable, failures cascaded through the entire system. Message queuing solved this by letting services communicate asynchronously: a producer could drop a message into a queue and move on, while a consumer picked it up when ready. This approach kept individual service failures from affecting the rest of the system.

When Amazon SQS launched publicly in July 2006, it made this pattern available to every AWS customer. Twenty years later, that core function, decoupling producers from consumers, remains the reason customers use SQS. The scale, performance, and operational controls around it look very different now though.

Jeff Barr covered the first 15 years of SQS milestones in his 15th anniversary post, from the original 8 KB message limit in 2006 through FIFO queues, server-side encryption, and Lambda integration. Over the last five years, we have continued to scale SQS, added stronger security defaults, and introduced new capabilities that address increasingly complex workload patterns.

Key milestones between 2021 and 2026
High throughput mode for FIFO queues (2021): In May 2021, we launched general availability of high throughput mode for FIFO queues, supporting up to 3,000 transactions per second (TPS) per API action, a tenfold increase over the previous limit. We continued raising this ceiling over the following two years: to 6,000 TPS in October 2022, to 9,000 TPS in August 2023, and to 18,000 TPS in October 2023, before reaching 70,000 TPS per API action in select Regions by November 2023.

Server-side encryption with SSE-SQS (2021): In November 2021, we introduced server-side encryption with Amazon SQS-managed encryption keys (SSE-SQS), giving customers an encryption option that required no key management. In October 2022, we made SSE-SQS the default for all newly created queues, so customers no longer needed to explicitly enable it.

Dead-letter queue redrive enhancements (2021): We progressively expanded how customers recover unconsumed messages from dead-letter queues. In December 2021, we added DLQ redrive to source queue directly in the SQS console. In June 2023, we extended this capability to the AWS SDK and CLI through new APIs, including StartMessageMoveTask, CancelMessageMoveTask, and ListMessageMoveTasks. In November 2023, we added redrive support for FIFO queues.

Attribute-based access control, ABAC (2022): In November 2022, we introduced ABAC, giving customers the ability to configure access permissions based on queue tags rather than maintaining static policies as resources scaled.

JSON protocol support (2023): In November 2023, we added support for the JSON protocol in the AWS SDK, reducing end-to-end message processing latency by up to 23% for a 5 KB payload and lowering client-side CPU and memory usage.

Amazon EventBridge Pipes console integration (2023): We added the ability to connect a queue directly to EventBridge Pipes from the SQS console, routing messages to a broad range of AWS service targets without writing custom integration code.

Extended Client Library for Python (2024): We brought the Extended Client Library, previously available for Java, to Python developers, allowing messages up to 2 GB to be sent through SQS by storing the payload in Amazon S3 and passing a reference through the queue.

FIFO in-flight message limit increase (2024): We increased the in-flight message limit for FIFO queues from 20,000 to 120,000 messages, so consumers can process significantly more messages concurrently without being constrained by the previous ceiling.

Fair queues for multi-tenant workloads (2025): We introduced fair queues to mitigate the noisy neighbor problem in multi-tenant standard queues. By including a message group ID when sending messages, customers can prevent a single tenant from delaying message delivery for others, without any changes required on the consumer side.

1 MiB maximum message payload size (2025): We increased the maximum message payload from 256 KiB to 1 MiB for both standard and FIFO queues, helping customers send larger messages without offloading data to external storage. AWS Lambda event source mapping for SQS was updated in parallel to support the new payload size.

The constant underneath the change
Despite two decades of feature additions, the fundamental use case for SQS has not shifted. Customers use it to decouple services, buffer bursts of traffic, and build systems that stay resilient when individual components fail. That same pattern now extends to AI workloads. Customers use SQS queues to buffer requests to large language models, manage inference throughput, and coordinate communication between autonomous AI agents operating as independent services. For an example of this architecture in practice, read Creating asynchronous AI agents with Amazon Bedrock.

To learn more about Amazon SQS, visit the Amazon SQS product page, review the developer guide, or explore recent updates on the AWS Blogs.

— Esra

AWS Weekly Roundup: AWS Builder Center at 1 year, Network Scanning in Security Hub, Loom for AWS, and more (July 13, 2026)

2026-07-13 Esra Kayabali

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-aws-builder-center-at-one-year-network-scanning-in-security-hub-loom-for-aws-and-more-july-13-2026/

AWS Builder Center turned one year old last week. Launched on July 9, 2025, the platform has grown from a community hub with Wishlist voting, community profiles, and a toolbox into a full ecosystem with sandbox environments, workshops, Spaces, and a Builders’ Library. To mark the anniversary, Rick Suttles published a full feature timeline covering everything shipped over the past year: AWS Capabilities by Region (1,500+ services across 37 Regions), Spaces for community-created groups, workshops with category and complexity filters, badges and streaks, article series, view counts, saved items, student status, availability notifications, sign-in with GitHub and Amazon, and sandbox environments.

Jeff Barr published a retrospective summarizing Builder Center’s first year. Since launch, 5,548 authors have published 6,448 articles with more than 10.4 million page views combined. Builders have earned 99,226 badges since the badge system launched in March 2026. Community members have submitted 565 wishes, 10 of which have shipped with another 20 on the near-term roadmap.

The top community article Building an AWS Study Buddy with MCP + Strands Agents SDK by Dineshraj Dhanapathy reached 50,000+ views. Chris Miller’s Migrating an EOL Linux Server to AWS in 8 Hours with Kiro followed at 45,000+, and Yash Aggarwal’s AIdeas: NeuroVoice – Multimodal AI for Early Screening of Neurological Diseases article reached 38,000+.

The week’s headline addition is Sandbox Environments by Rick Suttles. Sandboxes give you a free, pre-provisioned AWS account to complete a workshop exercise. Each environment is active for 8 hours, after which the account and all its resources are automatically de-provisioned. You can have one active sandbox at a time and request one per week. No personal AWS account, credit card, or manual cleanup required.

Last week’s launches
Here’s what else happened this week.

AWS Security Hub introduces Network Scanning – Security Hub introduced Network Scanning, a capability that identifies resources in your environment that are reachable from the public internet. Network Scanning probes your resources from the internet to detect actual reachability, complementing the existing network reachability findings in Security Hub that identify configurations that could make a resource reachable. It discovers public IP addresses, virtual machines, and load balancers across your AWS and Azure environments, identifies reachable ports, and determines what services are running behind them. Each reachable port generates a Security Hub finding with evidence of the port and service discovered. Security Hub Exposures then automatically correlates these findings with other findings and resource configurations to determine broader risk. Existing customers can enable Network Scanning in individual accounts and Regions, or across an organization through a configuration policy. For new customers, Network Scanning is on by default. It is included with Security Hub Essentials at no additional cost.
Security Hub also extends unified security management to Microsoft Azure – Security Hub now monitors Microsoft Azure resources, providing unified posture management, vulnerability management, and security response across both clouds. It automatically discovers Azure VMs, container images, Function Apps, and identities, and evaluates them for misconfigurations, internet exposure, and software vulnerabilities. AWS and Azure findings appear in the same prioritized view with the same formats and automation workflows.
Amazon SageMaker Studio integrates with Hugging Face for one-click model deployment and customization – You can now go from discovering a model on Hugging Face to working with it in SageMaker Studio in a single click. Select any supported model on Hugging Face and choose “Customize on SageMaker AI” or “Deploy on SageMaker AI” to land directly on the corresponding workflow page with the model pre-loaded. New customers receive a Studio environment created in seconds with pre-configured permissions for serverless model customization (including fine-tuning with custom reward functions for reinforcement learning), model evaluation, and deployment to SageMaker or Bedrock endpoints. Verified customers receive default GPU access to G5, G6, and G4dn instances without requesting quota increases, and quota utilization is visible directly inside the Studio environment.
Amazon EKS Auto Mode and Amazon ECS Managed Instances reduce GPU management fees by up to 60% – Beginning July 1, 2026, EKS Auto Mode and ECS Managed Instances reduce management fees for accelerated instance types: G-series fees are down 35%, and P-series and AWS Trainium fees are down 60%. The reductions apply automatically to existing clusters and require no action from customers. Both services include capabilities built for accelerated workloads. EKS Auto Mode provides automatic parallel image pulling on GPU instances with local NVMe storage and accelerator-aware node repair. ECS Managed Instances provides GPU metrics through Amazon CloudWatch Container Insights and automatic health monitoring for GPU hardware failures.
Amazon Aurora DSQL change data capture (CDC) is now generally available – Aurora DSQL CDC streams the results of insert, update, and delete operations as change events to Amazon Kinesis Data Streams. You can use it to synchronize data across microservices, trigger Lambda functions, or deliver changes to S3, Redshift, and OpenSearch Service through Amazon Data Firehose. CDC streaming is designed to have zero impact on database workload performance and requires no infrastructure to manage.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts you may find useful:

Building secure AI agents at scale: Introducing Loom for AWS – Loom is an open-source enterprise platform for building agents with AWS Strands Agents and deploying them on Amazon Bedrock AgentCore Runtime. It provides a unified management UI and backend API with identity provider integration, scope-based authorization, multi-persona navigation, and full lifecycle management for agents, memory, MCP servers, and agent-to-agent integrations. Loom enforces automated resource tagging for cost attribution, implements RBAC and ABAC for multi-tenant security, uses paved-path blueprints for agent deployments, manages identity propagation through delegated actor chains, integrates with AWS Agent Registry for discovery and governance, and supports human-in-the-loop review before sensitive actions. The project is available in AWS Labs on GitHub.
Introducing Claude apps gateway for AWS – The Claude apps gateway is a self-hosted control plane that gives organizations centralized control over access, cost, and policy for Claude Code and Claude Desktop. It connects to any OIDC-compliant identity provider, enforces managed settings on every request, routes inference to Amazon Bedrock or Claude Platform on AWS, and supports per-user and per-group spend caps. The gateway runs as a stateless container in your private network, backed by a PostgreSQL database for short-lived sign-in state. No long-lived secrets are stored on developer machines. Deploy it through Amazon Bedrock to keep data within the AWS security boundary, or through Claude Platform on AWS for the native Claude platform experience.
Introducing OAuth support for AWS MCP Server – You can now connect agents to the AWS MCP Server using browser-based OAuth with the same credentials you use for the AWS Console or CLI. The new sign-in path supports IAM federation, AWS IAM Identity Center, and root or IAM users. AWS Sign-In issues short-lived access tokens and refresh tokens, with automatic token management so developers stay authenticated across restarts. For headless use cases, a non-interactive flow lets applications with existing AWS credentials obtain OAuth access tokens through the create-oauth2-token-with-iam API. New governance controls include OAuth-specific IAM condition keys, token introspection and revocation, dynamic client registration, and CloudTrail audit elements.

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

AWS Summits – Free in-person events for builders and innovators to learn, think big, and make new connections. Coming up: Taipei (July 15), Bogotá (July 30), Jakarta (August 6), Ciudad de México (August 12), Johannesburg (August 19), and Zurich (September 2).
AWS Community Days – Community-led conferences planned and delivered by community leaders. Upcoming events include Yaoundé, Cameroon (July 25), Ahmedabad, India (July 25), Belo Horizonte, Brazil (August 22), Ottawa, Canada (August 22), Tulsa, USA (August 22), and Toronto, Canada (August 29).

Visit the AWS Builder Center to meet other builders, contribute solutions, and find resources that help you keep building.

Wishing everyone a restful and enjoyable summer. Whether you’re building, learning, or recharging, I hope you find time for all three. I’ll be heading to Scandinavia for a few weeks to trade the heat for some cooler weather and longer evenings. Come back next week for more news!

— Esra

New compliance guidance available: HITRUST i1 on AWS

2026-07-13 Abdul Javid

Post Syndicated from Abdul Javid original https://aws.amazon.com/blogs/security/new-compliance-guidance-available-hitrust-i1-on-aws/

We are pleased to announce the publication of a new AWS compliance implementation guidance: HITRUST i1 Compliance on AWS: Customer Implementation Guidance with an Illustrative Healthcare Platform.

Healthcare organizations seeking HITRUST i1 certification increasingly rely on Amazon Web Services (AWS) as their cloud foundation. The HITRUST i1 assessment covers 182 curated controls at the Implemented level and is the most widely required HITRUST certification tier in healthcare vendor contracts and Business Associate Agreements required by health plans, hospital systems, and business associates as a condition of working with them.

This guide is designed to close the gap between understanding what HITRUST i1 requires and knowing how to implement it on AWS. It walks cloud architects, security engineers, compliance leads, and assessment preparation teams through the full lifecycle of an i1 engagement from defining the assessment boundary to implementing controls across each technical domain.

What the guide covers

The guide addresses 11 HITRUST i1 technical control domains, with supporting AWS implementation components relative to these domains. The domains include access control, endpoint protection, configuration management, vulnerability management, network protection, transmission protection, incident management, data protection and privacy, audit logging and monitoring, password management, and business continuity and disaster recovery.

The guidance is grounded in a fictional but realistic connected healthcare platform deployed on AWS Landing Zone Accelerator. The scenario is used to make abstract HITRUST concepts concrete, not to suggest that the same architecture or control choices apply universally. HITRUST i1 scoping is inherently organization-specific. The assessment boundary, applicable controls, and evidence requirements are determined by each organization’s system scope and delivered through the HITRUST MyCSF portal. Readers should treat the guidance as a starting point and work with a HITRUST Authorized External Assessor to validate what applies to their specific environment. This guide doesn’t constitute a compliance certification advisory.

Getting started

You can download the guide here: HITRUST i1 Compliance on AWS: A Customer Implementation Guidance with an Illustrative Healthcare Platform.

AWS HITRUST assurance documentation and the Customer Responsibility Matrix are available through AWS Artifact. For assessment readiness support, visit AWS Security Assurance Services.

If you have feedback about this post, submit comments in the Comments section below.

AWS designated as a critical third party to the UK financial sector

2026-07-10 Michael Jefferson

Post Syndicated from Michael Jefferson original https://aws.amazon.com/blogs/security/aws-designated-as-a-critical-third-party-to-the-uk-financial-sector/

Amazon Web Services EMEA Sarl (AWS) has been designated as a critical third party (CTP) to the UK financial sector by HM Treasury.

The CTP regime came into force on January 1, 2025, and establishes a framework through which the Bank of England, PRA, and FCA (collectively the UK regulators) can set requirements on and have direct oversight of designated CTPs.

AWS supports the objectives of the UK regulators to ensure a robust financial system.

AWS obligations under the regime

The CTP regime is an outcomes-focused framework. Under the regime, AWS will be subject to requirements in relation to its designated Systemic Third-Party Services (STPS). The first step will be a self-assessment of these STPSs against the CTP regime criteria, which AWS will now carry out in line with regulators’ expected timelines.

AWS has actively engaged with the UK authorities as they’ve developed the CTP regime, and we will continue this constructive approach as we work to meet our obligations under it.

Impact on customers

The UK regulators have clarified that the requirements under the regime don’t eliminate, reduce, or replace the accountability of firms, their boards, and senior management for remaining operationally resilient, including when they rely on services provided by third parties.

The regime doesn’t change obligations on financial services customers of designated CTPs. As we manage our obligations, we expect to publish materials that customers can use to inform their own operational resilience planning and third-party risk management.

The AWS commitment to operational resilience

We’re focused on supporting financial services customers in enhancing their operational resilience and providing a range of services and guidance—including the AWS Well-Architected Framework and resources for cloud incident management—to help organizations deliver effective resilience outcomes.

AWS has a team of regulatory and technology experts with expertise in financial services ready to support customers with questions about this regime or operational resilience more broadly. Customers can contact their AWS account team for further information.

Introducing Apache Spark Connect support in AWS Glue interactive sessions

2026-07-07 Zach Mitchell

Post Syndicated from Zach Mitchell original https://aws.amazon.com/blogs/big-data/introducing-apache-spark-connect-support-in-aws-glue-interactive-sessions/

When we built AWS Glue interactive sessions, our goal was to make AWS Glue as interactive as running local Python from a notebook. We mostly succeeded. With a straightforward Python package and a Jupyter notebook, you could execute remotely against the AWS Glue ephemeral Spark backend. The Livy-based approach was ahead of its time, but it had limitations from its REST-based protocol. Running local PySpark unlocked powerful integrated development environment (IDE) features such as debugging and linting, so your environment could understand the code and help you develop Spark applications more quickly. Customers would often split their development work. They used local Spark (or Docker containers) to develop in an IDE on a small amount of data, then switched to AWS Glue interactive sessions to validate scaling and tuning against the full dataset.

With modern PySpark releases came a new protocol: Apache Spark Connect. Spark Connect bridges the gap between these two worlds: you develop in local Python, but execute on AWS Glue against actual data. Today, AWS Glue interactive sessions support Spark Connect natively. You can connect from any environment that supports the PySpark remote() API, including VS Code, PyCharm, Amazon SageMaker Unified Studio notebooks, and standalone Python applications. You don’t need to install specialized kernels or manage cluster infrastructure.

What Spark Connect changes

Spark Connect, introduced in Spark 3.4, decouples the Spark client from the server through a lightweight gRPC protocol. Instead of running your driver program on the cluster, your IDE communicates with a remote Spark server through a thin client layer. This architecture unlocks the key workflow improvement: you develop locally and execute remotely.

Spark Connect architecture diagram showing a thin client communicating with a remote Apache Spark server

Spark Connect architecture — thin client with the full power of Apache Spark

With Spark Connect support in AWS Glue interactive sessions, you get:

IDE freedom – Use VS Code, PyCharm, JupyterLab, or any Python environment. No kernel installation required.
Programmatic access – Build Spark into your Python applications and automation scripts with a standard SparkSession.builder.remote() call.
Serverless execution – AWS Glue provisions and manages the Spark cluster. You pay only for the data processing units (DPUs) consumed while your session is active.
Spark Connect monitoring – The Spark Live UI now includes a dedicated Connect tab showing active Spark Connect sessions and operations alongside the existing Jobs, Stages, and Executors views.

Getting started with SageMaker Unified Studio

Amazon SageMaker Unified Studio provides the most direct path to Spark Connect on AWS Glue. The notebook environment handles session creation, endpoint retrieval, and token refresh automatically, so no connection boilerplate is required.

Prerequisite: You need an Amazon SageMaker Unified Studio project to use this workflow. If you don’t have one, create a project in your SageMaker Unified Studio domain first.

To connect to an AWS Glue Spark Connect session:

A notebook open in SageMaker Unified Studio

A notebook open in SageMaker Unified Studio

Choose the compute icon in the left toolbar to open the Compute environment panel. Expand the Spark section.

Compute environment panel in SageMaker Unified Studio with the Spark section expanded

The Compute environment panel with the Spark dropdown list

Select a Glue Spark connection. Depending on your SageMaker domain configuration, you will see either default.spark or named connections such as project.spark.compatibility. Select the appropriate Glue (Spark) connection and choose Apply.

Notebook cell showing spark.version returns 3.5.6-amzn-1 after connecting to Glue Spark Connect

Connected to Glue Spark Connect — running spark.version returns ‘3.5.6-amzn-1’

After you make your selection, you’re connected. The spark session object is available natively. No imports or configuration are needed. Start running PySpark immediately:

spark.sql("SHOW DATABASES").show()

The session manages itself in the background, including automatic token refresh.

Using the sagemaker_studio SDK

The sagemaker-studio Python package extends the Spark Connect experience beyond SageMaker Unified Studio notebooks into local IDEs, continuous integration and continuous delivery (CI/CD) pipelines, and any Python environment. The sparkutils module handles session initialization and connection configuration in a single call. You get the same streamlined experience as in the notebook, anywhere you run Python:

from sagemaker_studio import sparkutils

# Initialize a Glue Spark Connect session using your project connection
spark = sparkutils.init(connection_name="default.spark")

# Run queries immediately
spark.sql("SHOW DATABASES").show()

You can also use sparkutils.get_spark_options() to retrieve pre-configured Java Database Connectivity (JDBC) options for reading and writing to data sources through your project connections. Supported sources include Amazon Redshift, Amazon Aurora, and Amazon DocumentDB (with MongoDB compatibility):

# Get connection options for a Redshift connection in your project
options = sparkutils.get_spark_options("my_redshift_connection")

# Read from Redshift via Spark Connect
df = spark.read.format("jdbc").options(**options).option("dbtable", "analytics.orders").load()
df.show()

Within SageMaker Unified Studio, the sagemaker-studio SDK is native to the environment. The spark session and sparkutils are available without installation. For local IDE use, install it with pip install sagemaker-studio and configure credentials through an AWS named profile or boto3 session.

How it works

Spark Connect sessions in AWS Glue use a three-step workflow:

Create a session – Call the CreateSession API with SessionType set to SPARK_CONNECT. The session provisions in approximately 30 seconds.
Retrieve the endpoint – Call GetSessionEndpoint to receive a sc:// gRPC endpoint URL and a time-limited authentication token.
Connect with PySpark – Pass the endpoint and token to SparkSession.builder.remote() and start running Spark operations.

Spark Connect protocol flow from the DataFrame API to a logical plan, sent over gRPC and protobuf, with results streamed back over gRPC and Arrow

Spark Connect protocol flow — DataFrame API translated to logical plan, sent via gRPC/protobuf, results streamed back via gRPC/Arrow

Connecting with the low-level API

Some environments don’t have the sagemaker-studio SDK, such as custom containers, AWS Lambda functions, or non-Python toolchains. In these environments, or if you’re not using SageMaker Unified Studio, you can use the AWS SDK (Boto3) to manage sessions directly. The following example demonstrates the full workflow:

import time, boto3, urllib.parse
from pyspark.sql import SparkSession

glue = boto3.client("glue", region_name="us-east-1")

# 1. Create a Spark Connect session
session_id = "my-spark-connect-session"
glue.create_session(
    Id=session_id,
    Role="arn:aws:iam::123456789012:role/GlueServiceRole",
    Command={"Name": "glueetl"},
    GlueVersion="5.1",
    SessionType="SPARK_CONNECT",
    DefaultArguments={"--enable-spark-live-ui": "true"},
)

# 2. Wait for the session to reach READY
while True:
    status = glue.get_session(Id=session_id)["Session"]["Status"]
    if status == "READY":
        break
    time.sleep(5)

# 3. Get the Spark Connect endpoint
sc = glue.get_session_endpoint(SessionId=session_id)["SparkConnect"]
endpoint_url = sc["Url"]
auth_token = sc["AuthToken"]

# 4. Connect with PySpark
encoded_token = urllib.parse.quote(auth_token, safe="")
connection_string = f"{endpoint_url}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
spark = SparkSession.builder.remote(connection_string).getOrCreate()
spark.sql("SELECT 1 + 1 AS result").show()

Monitoring with Spark Live UI

When you enable the Spark Live UI at session creation, you gain access to a real-time dashboard showing:

Jobs and Stages – Track active, completed, and failed jobs with stage-level metrics.
Executors – Monitor memory usage, shuffle data, and executor health.
SQL – Inspect query plans and execution details.
Connect tab – View active Spark Connect sessions and operations (specific to Spark Connect).

Access the dashboard through the GetDashboardUrl API or directly from the AWS Glue console.

import boto3, webbrowser

glue = boto3.client("glue", region_name="us-east-1")
dashboard = glue.get_dashboard_url(
    ResourceId="my-spark-connect-session",
    ResourceType="SESSION",
)
webbrowser.open(dashboard["Url"])

In SageMaker Unified Studio, no API call is needed. Choose Ready in the notebook status bar to open the kernel info popover. From there, open the Spark UI link for the live dashboard or Spark Driver Logs for real-time log output.

Notebook status bar Ready button that opens the Spark UI and Spark Driver Logs links

Image showing “Ready” in the status bar to access Spark UI and Driver Logs directly from the notebook

Token refresh

Authentication tokens expire after 30 minutes. In SageMaker Unified Studio, this is handled automatically. For programmatic use, you can use a background thread to keep the connection alive. The following helper reconnects transparently before the token expires:

import threading, time, boto3, urllib.parse
from pyspark.sql import SparkSession

class GlueSparkConnect:
    """Maintains a SparkSession with automatic token refresh."""

    def __init__(self, session_id, region="us-east-1", refresh_margin=300):
        self.session_id = session_id
        self.glue = boto3.client("glue", region_name=region)
        self.refresh_margin = refresh_margin  # seconds before expiry to refresh
        self._lock = threading.Lock()
        self.spark = self._connect()
        self._start_refresh_loop()

    def _connect(self):
        sc = self.glue.get_session_endpoint(SessionId=self.session_id)["SparkConnect"]
        encoded_token = urllib.parse.quote(sc["AuthToken"], safe="")
        remote_url = f"{sc['Url']}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
        self._token_expiry = sc["AuthTokenExpirationTime"].timestamp()
        return SparkSession.builder.remote(remote_url).getOrCreate()

    def _start_refresh_loop(self):
        def _loop():
            while True:
                sleep_for = max(self._token_expiry - time.time() - self.refresh_margin, 30)
                time.sleep(sleep_for)
                with self._lock:
                    self.spark = self._connect()
        t = threading.Thread(target=_loop, daemon=True)
        t.start()

# Usage
session = GlueSparkConnect("my-spark-connect-session")
session.spark.sql("SELECT 1 + 1 AS result").show()

The background thread sleeps until 5 minutes before token expiry, then transparently reconnects. Because the daemon thread exits when your script ends, there is no cleanup required.

Getting started

To start using Spark Connect with AWS Glue interactive sessions:

Use AWS Glue version 5.1 (Apache Spark 3.5.6).
Install PySpark 3.5.6 locally: pip install pyspark==3.5.6.
Grant your AWS Identity and Access Management (IAM) identity permissions for glue:CreateSession, glue:GetSession, and glue:GetSessionEndpoint.
Create a session with --session-type SPARK_CONNECT and connect from your preferred environment.

VPC note: If you connect to AWS Glue interactive sessions through a virtual private cloud (VPC) endpoint, add the new Spark Connect endpoint (com.amazonaws.{region}.glue.sessions) to your VPC configuration. Existing AWS Glue VPC endpoints don’t cover Spark Connect traffic.

For detailed instructions, see Connecting to a Spark Connect session in the AWS Glue Developer Guide.

About the authors

Amazon Redshift RG: Faster and lower cost, Graviton-powered

2026-07-02 Stefan Gromoll

Post Syndicated from Stefan Gromoll original https://aws.amazon.com/blogs/big-data/amazon-redshift-rg-faster-and-lower-cost-graviton-powered/

Amazon Redshift recently announced the general availability of a new Graviton-powered instance called RG. Built on Amazon’s own Graviton processors, RG delivers:

Up to 2.2x faster performance for data warehouse workloads compared to RA3.
Up to 2.4x faster for Iceberg queries and 1.5x faster for Parquet queries through an integrated vectorized data lake engine.
No per-TB scan charges for data lake queries, eliminating the Amazon Redshift Spectrum cost applied on RA3 clusters.
30 percent lower cost per vCPU compared to RA3.

RG is both faster and cheaper. While cloud vendors typically charge more for faster performance or newer generation hardware, Amazon Redshift delivers better performance at lower cost.

In this post, we describe the innovations that make RG instances so much faster. We also share benchmark results showing that RG delivers up to 4.2x better price-performance than other leading data warehouses.

What makes RG so fast

The new RG instances are built from the ground up to take advantage of Graviton processors. The vectorized engine of Amazon Redshift is optimized with Graviton-based single instruction, multiple data (SIMD) kernels to deliver accelerated, parallelized execution for analytics workloads. Operations like predicate evaluations over Parquet encodings use Graviton vector comparison, table lookup, and vector manipulation intrinsics. To support these increased processing speeds, RG instances use custom-built Nitro SSDs. This lets RG use faster local storage as a caching layer for Amazon Redshift Managed Storage (RMS), data lake scans, and intermediate result sets for computations that can’t fit in memory. RG’s JIT (Just-In-Time) Analyze feature also collects statistics from data lake files automatically as queries run, so the optimizer can produce significantly better query plans. Together, these represent innovation across the entire stack: hardware acceleration with Graviton, vectorized execution with SIMD kernels, high-speed storage with Nitro SSDs, and intelligent query planning with JIT Analyze.

These optimizations, coupled with RG’s purpose-built high-performance vectorized data lake engine, combine to make Amazon Redshift’s new RG instances up to 2.2x faster than RA3 for analytics workloads at 30 percent lower cost.

Purpose-built high-performance vectorized data lake engine

With RA3, data lake queries offloaded scans to a separate compute fleet known as Amazon Redshift Spectrum. Because data lake queries ran on this separate compute, additional overhead was introduced to transfer query metadata and results between RA3 clusters and the Spectrum fleet. Amazon Redshift RG instances include a completely new built-in scan layer designed from the ground up for data lakes. This new scan layer includes a purpose-built I/O subsystem that incorporates smart prefetch capabilities to reduce data latency. The new scan layer is also optimized to process Apache Parquet files, the most commonly used file format for Iceberg, through fast vectorized scans that use SIMD kernels optimized for Graviton. The scan layer includes sophisticated data pruning mechanisms that operate at both partition and file levels, which significantly reduces the volume of data that needs to be scanned. This pruning capability works with the smart prefetch system to create a coordinated approach that maximizes efficiency throughout the entire data retrieval process.

The new purpose-built vectorized data lake engine is up to 2.4x faster than RA3 for Iceberg queries and 1.5x faster than RA3 for Parquet queries.

Because this new vectorized data lake engine integrates directly with the core execution engine of Amazon Redshift, new performance optimizations are possible compared to RA3. With this architecture, data lake queries on RG now benefit from fast local data caching, improved bloom filters, vectorized Parquet scans, and advanced filtering and pruning.

RG also solves a common problem customers face when querying data in the lake: open-format files like Iceberg in Amazon Simple Storage Service (Amazon S3) often lack useful metadata and statistics, which makes it difficult to run a SQL query optimally.

Statistics are metadata about your data, such as distinct value counts, min/max values, distribution patterns, and row counts. The query optimizer uses this information to choose the most efficient way to run a query. For example, when joining two tables, the optimizer needs to know how many unique values each side produces to pick the right join strategy. Without statistics, it has to guess, which often leads to slower joins and unnecessary data movement across nodes. This is where Amazon Redshift’s new feature called JIT (Just-In-Time) Analyze comes in. RG instances automatically fetch and store statistics of your Iceberg files as queries run, so Amazon Redshift can choose query execution strategies that are far more optimized than it could without these statistics.

These improvements make scans of Iceberg and Parquet data much faster than RA3. Removing Amazon Redshift Spectrum compute also means RG instances remove the $5/TB cost for data lake queries, which makes data lake queries cheaper and costs predictable. This is a triple win for data lake price-performance: faster performance, lower compute cost, and no per-TB scan cost.

Faster insights from faster data loads

Amazon Redshift RG’s fast I/O and Graviton-optimized engine result in faster data loads compared to RA3. To measure this improved performance, we ran the data ingestion step of 10TB TPC-DS and TPC-H on equivalently sized RA3 and RG clusters. RG ingested the TPC-DS dataset 2x faster and the TPC-H dataset 1.4x faster, as shown in the following figure.

Bar chart comparing data ingestion time on RA3 and RG, showing RG loads TPC-DS 2x faster and TPC-H 1.4x faster

The new Graviton-based RG instances are up to 2.0x faster for data loads compared to RA3 instances. This means workloads can see the latest data sooner, and users and agents can get up-to-date insights faster. This faster ingestion on RG comes at 30 percent lower cost compared to RA3, resulting in up to 2.9x better price-performance for data loads compared to RA3 instances.

What customers are saying

Amazon Redshift customers are already seeing performance and cost benefits of switching to RG. Southwest Airlines and tombola tested their business-critical workloads, and found they could get better performance and save on cost:

Southwest Airlines

“Amazon Redshift RG instances have the potential to deliver meaningful business impact for Southwest Airlines. Based on initial testing in our development environment, our data warehouse workloads run 50–60% faster, and data lake analytics are 45% faster—enabling teams to get insights sooner, respond to operational conditions faster, and make data‑driven decisions with less latency. These early results are encouraging, and we are excited to validate and scale these improvements in production. All of this comes without per‑terabyte Spectrum scanning charges, delivering 30% lower cost than RA3 at a time when fuel prices continue to pressure industry margins!!”

— Sean Lynch, Vice President, Data and Architecture, Southwest Airlines

tombola

“The new Graviton-based Amazon Redshift RG instances delivered 1.8x–2x faster write throughput and up to 2.2x faster read speeds compared to RA3 across a diverse set of batch and analytical jobs — enabling us to process 40% more within the same window. Compressed ETL cycles, accelerated time-to-insight, and decision-making no longer bottlenecked by the pipeline — together, these translated directly into fresher data reaching our analysts and business teams sooner. What made this even more compelling was a concurrent 30% reduction in compute spend alongside the gains — delivering more for less is a rare outcome, and one worth highlighting. In a volume-heavy gaming industry at tombola, where query latency and cost compound at scale, this has been one of the more impactful platform decisions we’ve made this year.”

— Akshay Srinivasan, Data Engineer, tombola

Qoala

“After migrating our Amazon Redshift cluster from RA3 to the new Graviton-based RG instances, we saw 60–70% faster query processing times across our BI and analytics workloads. As a growing insurtech platform handling millions of policy transactions, faster time-to-insight means our data team can deliver dashboards and reports to the business sooner. We moved to a larger node configuration to accommodate future growth, and the performance gains far exceeded the incremental investment – making this one of the most impactful infrastructure decisions we’ve made this year.”

— Umar Abdul Aziz, VP of Data, Qoala

Performance results

To see how RG stacks up, we ran benchmarks derived from the industry-standard TPC-DS and TPC-H benchmarks at 10TB scale on the new Amazon Redshift RG instances and on leading alternative data warehouses. These benchmarks are designed to run queries of various operational requirements and complexities, such as ad hoc, reporting, iterative online analytical processing (OLAP), and data mining. We sized each data warehouse at approximately the same on-demand cost ($32/hr) and ran three power runs of each benchmark out of the box, with no special tuning or manual customization. The results are shown in the following charts.

Bar chart of TPC-DS 10TB price-performance showing Amazon Redshift RG leading alternative data warehouses

Bar chart of TPC-H 10TB price-performance showing Amazon Redshift RG leading alternative data warehouses

The new RG instance leads, and by a large margin. Better price-performance means better performance and lower cost.

Conclusion

Amazon Redshift RG instances are the next generation of analytics engine, delivering high performance for data warehouse and data lake workloads. Because RG supports all the same workloads and features as RA3, getting started is straightforward. See our migration guide for how to upgrade and start getting better performance at lower cost.

Find the best price-performance for your workloads

The benchmarks used in this post are derived from the industry-standard TPC-DS and TPC-H benchmarks, and have the following characteristics:

We use the schema and data unmodified from TPC-DS and TPC-H.
The queries are generated using the official TPC-DS and TPC-H kits with query parameters generated using the default random seed of the kits. TPC-approved query variants are used for a warehouse if the warehouse doesn’t support the SQL dialect of the default queries.
The test includes the 99 TPC-DS SELECT queries and 22 TPC-H SELECT queries. It doesn’t include maintenance and throughput steps.
Three power runs were run, and the best run is taken for each data warehouse.
Price-performance is calculated as the cost per hour (USD) divided by 3,600 seconds/hour times the benchmark geomean in seconds, which is equivalent to the geomean cost per query. The latest published on-demand pricing is used for all data warehouses.

We call this the Cloud Data Warehouse benchmark, and you can reproduce the preceding benchmark results using the scripts, queries, and data available in our GitHub repository. It’s derived from the TPC-DS benchmarks as described in this post, and as such isn’t comparable to published TPC-DS results, because the results of our tests don’t comply with the official specification.

About the authors

Run log analytics for a fraction of the cost with the new engine for Amazon OpenSearch Service

2026-07-01 Jagadish Kumar

Post Syndicated from Jagadish Kumar original https://aws.amazon.com/blogs/big-data/run-log-analytics-for-a-fraction-of-the-cost-with-the-new-engine-for-amazon-opensearch-service/

Amazon OpenSearch Service is a real-time retrieval engine for AI, search, and analytics at any scale. As log volumes grow 30–40 percent year over year, organizations face rising infrastructure costs and slower analytical queries across their observability data. Teams are forced to choose between retaining the data they need and staying within budget.

We’re introducing a purpose-built log analytics engine for Amazon OpenSearch Service. This new engine delivers up to 4x price performance, 2x faster data ingestion, up to 2x faster analytical queries, and up to 70 percent lower storage costs. You get all of this without sacrificing search capabilities on the same data.

In this post, you learn how to take advantage of these benefits, see how to get started, and review benchmark results at billion-document scale.

How the optimized engine works

The optimized engine is a new engine mode within the same Amazon OpenSearch Service domain. You use the same console, APIs, security model, and networking configuration that you already use with the general-purpose engine.

OpenSearch Service stores all data in Apache Parquet format. For fields configured as searchable, OpenSearch Service also writes the data to the inverted index. Apache Calcite parses and optimizes each query, then routes operations to the engine best suited to execute them: Apache DataFusion for analytical operations on columnar data, or Lucene for search predicates. The two hand off mid-query, so a single query can search log content and aggregate the results without additional roundtrips.

You ingest data through the same REST APIs and client libraries you use today and you don’t need to change your agents or pipelines. The optimized engine supports two query languages: Piped Processing Language (PPL) and SQL. Both execute natively through the vectorized engine. The Domain Specific Language (DSL) query API is not supported on the optimized engine at launch.

Getting started

At launch, the optimized engine is a domain-level setting selected at creation time. You can’t add the optimized engine to an existing domain or enable it on individual indices or fields within a general-purpose domain. To adopt the optimized engine, create a new domain and migrate your ingestion pipelines to it.

Create a new domain in the Amazon OpenSearch Service console and select Observability as your use case. The optimized engine is enabled by default. The console provides a side-by-side comparison of capabilities to help you choose.

Amazon OpenSearch Service console showing the Observability use case selected with a side-by-side comparison of engine capabilities

After your domain is ready, ingest JSON documents through the same Bulk API and client libraries you use today. No changes to your ingestion pipelines or application code are required.

Benefits of the optimized engine for log analytics

The optimized engine for log analytics introduces the following performance and cost improvements:

Up to 4x better price-performance compared to the existing general-purpose engine on internal benchmarks, while retaining full-text search for incident investigation.
Up to 2x faster analytical queries. The engine uses a vectorized query execution path that processes data in columnar batches for fast results across large datasets.
Up to 2x higher ingestion throughput. The append-only columnar write path increases sustained ingestion rates.
Up to 70 percent lower storage with columnar storage for aggregation workloads. You can retain up to 3x more data at the same cost.

To demonstrate these improvements, we benchmarked observability workloads at billion-document scale. In the following sections, we explore the benchmark methodology, test environment, and results. We recommend testing the optimized engine with your own workload to validate the gains for your use case.

Benchmark methodology

We used the Telemetry Generator for OpenTelemetry to generate synthetic traces and logs at scale, producing three observability datasets: OTEL traces, OTEL logs, and web server access logs. We stored the generated data as bulk-format NDJSON in Amazon Simple Storage Service (Amazon S3). We then ingested it through a pipeline on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. The pipeline reads chunks from Amazon S3, transforms timestamps, and writes to the OpenSearch Bulk API, simulating a production observability flow.

We benchmarked on two OpenSearch Service domains running OpenSearch 3.5, each with 9 data nodes in a 3-Availability Zone configuration:

Configuration	Optimized Engine	Standard Lucene
Instance type	9x or2.4xlarge.search	9x r8g.4xlarge.search
Leader nodes	3x m7g.large.search	3x m7g.large.search
EBS	2,500 GB gp3, 7,500 IOPS, 500 MB/s per node	2,500 GB gp3, 7,500 IOPS, 500 MB/s per node
Engine mode	OPTIMIZED	General Purpose (`best_compression`)

We ingested three data sets totaling 24.4 billion documents and 9.5 TB of raw JSON. All indices used 9 primary shards, 1 replica, and Index State Management (ISM)-managed rollover at 50 GB per primary shard. The Lucene baseline used best_compression (zstd) codec with _source enabled, representing the default customer configuration.

The ingestion pipeline ran on 90 Fargate tasks (16 vCPU, 120 GB RAM each, 48 writer threads per task, bulk size of 3,000 documents) in the same virtual private cloud (VPC) as the OpenSearch Service domains.

Results

Ingestion throughput

The optimized engine’s append-only columnar storage writes segments in bulk-optimized batches without per-document stored field overhead.

Metric	Optimized Engine	Lucene Baseline
Peak throughput	1.78M docs/sec	~647K docs/sec
Cluster CPU at peak	62%	72%
Write rejections	0	0
Total documents ingested	24.4 billion	15.7 billion

The optimized engine sustained 1.78 million documents per second at matched concurrency, approximately 2x the throughput of the Lucene baseline, while consuming less CPU. Both domains ran with zero write rejections. For teams ingesting terabytes per day, the throughput advantage translates to fewer nodes for the same volume, or longer retention on the same infrastructure.

Storage compression

The columnar Parquet format compresses observability data through dictionary encoding of repeated fields, tight packing of numeric columns, and elimination of per-document JSON overhead.

Measured across 24.4 billion documents:

Dataset	Documents	Source	Optimized Engine	Lucene (default)	Compression vs. source	Savings vs. Lucene
Web logs	8.76B	2,360 GB	254 GB	614 GB	89%	59%
OTEL logs	8.20B	3,720 GB	815 GB	1,549 GB	78%	47%
OTEL traces	7.43B	4,131 GB	841 GB	1,790 GB	80%	53%
Total	24.4B	9,539 GB	1,910 GB	3,953 GB	80%	52%

The optimized engine stores the same data at 5x compression versus raw JSON (80 percent savings). Against the default Lucene configuration (_source enabled, what most domains run), the optimized engine uses roughly half the storage. The optimized engine derives _source from Parquet columns on read, eliminating the need to store the raw JSON blob while still allowing document retrieval.

Analytical query performance

We measured query latency on a typical observability dashboard pattern: analytical aggregations scoped to a 15-minute time window over billions of log events. The optimized engine uses row-group pruning on the @timestamp column to skip data outside the query window, reading only the relevant subset.

Query pattern	Dataset	Optimized Engine	Lucene baseline	Speedup
Error count by service	OTEL logs	717 ms	2.8 s	3.9x
Log volume by host	OTEL logs	252 ms	17.6 s	70x
5xx errors by service and method	OTEL logs	171 ms	885 ms	5.2x
Top services by error	OTEL traces	635 ms	569 ms	~1x
Point lookup (single traceId)	OTEL traces	394 ms	783 ms	2x

All queries scoped to a 15-minute window. Index sizes: 8.2 billion OTEL log events, 7.4 billion OTEL trace spans.

The optimized engine completes time-filtered analytical queries in 171 ms to 717 ms over billions of documents. The advantage is most pronounced on unfiltered aggregations (log volume by host: 70x) where the columnar engine reads only the columns needed. On queries where the Lucene inverted index provides strong predicate selectivity (top services by error on traces), performance is comparable between the two engines.

Search and point lookups

The optimized engine retains the Lucene inverted index alongside columnar storage. When the query planner recognizes a selective lookup (such as retrieving a single trace by ID), the planner routes the query to the inverted index rather than scanning columnar data. In our benchmark, a single traceId lookup across 7.4 billion spans returned in 165 ms.

This means a real investigation can use both engines in sequence: broad aggregations to localize the problem, then a point lookup to pull the offending trace, all from the same domain.

Now available

The optimized engine for Amazon OpenSearch Service is generally available today in all commercial AWS Regions (Regions other than the AWS GovCloud (US) Regions and the China Regions) where OpenSearch Optimized Instances are available.

Pricing follows standard Amazon OpenSearch Service rates for instances and storage, with no additional premium for the optimized engine. For more information, see Amazon OpenSearch Service Pricing.

To learn more about configuring and using the optimized engine, see Optimized for Log Analytics in the Amazon OpenSearch Service documentation. For an overview of the service, visit Amazon OpenSearch Service Log Analytics.

Give it a try and send feedback to AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.

About the authors

Scale analytics with Amazon Redshift multi-warehouse enhancements

2026-06-29 Raza Hafeez

Post Syndicated from Raza Hafeez original https://aws.amazon.com/blogs/big-data/scale-analytics-with-amazon-redshift-multi-warehouse-enhancements/

Onboard analytics workloads at scale with Amazon Redshift’s improved remote table data definition language (DDL), materialized view improvements, and concurrency scaling enhancements for zero-ETL and auto-copy.

As organizations scale their analytics capabilities, they need the ability to add workloads without disrupting production operation or being constrained by the resources of a single data warehouse. In this post, we introduce new capabilities of Amazon Redshift that enhance our multi-warehouse and scaling capabilities: remote materialized view (MV) operations, remote table DDL support, and concurrency scaling enhancements for zero-ETL and S3 event integration. These features help you build more scalable, performant decentralized analytics architectures on Amazon Redshift.

Let us review how these new features enable you to run analytics at scale.

New remote materialized view operations

Amazon Redshift now classifies CREATE MATERIALIZED VIEW as a user workload, enabling concurrency scaling to execute MV logic on additional warehouses during resource contention. This ensures queries consistently benefit from MV performance advantages, even under heavy load.
Amazon Redshift now supports creating MVs on remote data shares, allowing customers who share data across Redshift warehouses to leverage MV performance benefits on both local and shared data.
Consumer warehouses can now refresh MVs created on a producer and create MVs on top of data-shared MVs, enabling full MV parity across producer and consumer warehouses in a data sharing architecture.

New remote table DDL operations

ALTER TABLE ALTER DISTSTYLE operations now work on remote warehouses through concurrency scaling and data sharing. You can dynamically optimize data distribution across distributed environments, improving query performance and resource utilization without requiring data migration. This is especially valuable for data engineers fine-tuning performance across multiple warehouses and administrators adapting to changing query patterns.
ALTER TABLE APPEND operations now extend to remote warehouses through concurrency scaling and data sharing. This consolidates data across distributed environments, so you can efficiently combine tables without complex data movement or extract, transform, and load (ETL) processes. Organizations managing dynamic table operations across multiple environments can maintain data consistency while reducing operational overhead.

Concurrency scaling improvements

Amazon Redshift’s enhanced zero-ETL feature now supports concurrency scaling for automated data ingestion from applications and operational sources.
Amazon Redshift’s enhanced auto-copy feature now supports concurrency scaling for automated data ingestion from S3.
Amazon Redshift now extends concurrency scaling to support COPY queries from Amazon S3. You can now scale data ingestion automatically with concurrency scaling for batch workloads.

With these new concurrency scaling capabilities, you can maintain consistent data freshness without compromising existing warehouse performance. This eliminates the traditional trade-off between analytics and data loading. Apart from turning on concurrency scaling, no additional changes are required to take advantage of these features.

Customer use cases

This section covers two industry use cases: the first for a financial services customer and the second for a gaming industry customer.

Financial services use case

The following is a sample architecture for a large financial services customer with global operations. This customer uses a multi-warehouse architecture built on Amazon Redshift.

Financial services multi-warehouse architecture using STG, DWH, ETL, and USR Amazon Redshift warehouses

The staging (STG) warehouse serves as a raw zone for data from various sources, like the bronze layer of a medallion architecture. This warehouse also cleanses and standardizes the raw data to the silver layer and makes it available for further processing. The STG warehouse uses MVs to process millions of nested JSON messages and extract attributes into scalar columnar Amazon Redshift tables.

CREATE MATERIALIZED VIEW rawdb.fsi.customer_orders_raw
distkey(c_custkey) sortkey(c_custkey) AS (
    SELECT c_custkey,
        o.o_orderstatus,
        o.o_totalprice,
        o_idx
    FROM customer_orders_lineitem c,
        c.c_orders o AT o_idx
);
REFRESH MATERIALIZED VIEW rawdb.fsi.customer_orders_raw;

The DWH warehouse serves as the primary Amazon Redshift instance and gold layer, providing data to consuming applications like Business Objects and Tableau. The zero-ETL concurrency scaling improvements provide consistent data freshness even when zero-ETL ingestion spikes occur alongside heavy DWH workloads. The DWH MVs provide fast access to aggregated data for Tableau extracts and Business Objects live reports. The DWH warehouse takes advantage of concurrency scaling when multiple MVs need to be refreshed on the DWH instance.

CREATE MATERIALIZED VIEW bodb.final.customer_churn_tbl
AS (
    SELECT state,
        account_length,
        area_code,
        total_charge/account_length AS average_daily_spend,
        cust_serv_calls/account_length AS average_daily_cases,
        churn
    FROM custdb.final.customer_activity_all
);
REFRESH MATERIALIZED VIEW bodb.final.customer_churn_tbl;

The ETL01/02 warehouses serve as dedicated compute environments for running project-specific ETL jobs, while the USR01/02 warehouses handle user workloads such as ad-hoc analysis or model building from dbt. When new objects are required by user workloads, they are created and maintained on the remote producer warehouse (DWH).

ALTER TABLE salesdb.final.sales_report_all
ALTER DISTKEY sales_id;

ALTER TABLE APPEND salesdb.final.sales_report_all
FROM stagingdb.sales.sales_2026_02;

Gaming industry use case

A leading gaming company has built their entire analytics infrastructure on AWS, with their analytics team managing data streaming from games, data warehousing, and business intelligence tools. They standardized Amazon Redshift across the organization, migrating off Vertica running on Amazon Elastic Compute Cloud (Amazon EC2). After overcoming early challenges with cluster resize operations, the team became strong advocates for Amazon Redshift and now runs their primary production cluster on 32 ra3.16xlarge nodes.

As their data ingestion pipeline grew, query workloads began competing with data ingestion processes, creating performance bottlenecks. Rather than scaling up their primary cluster, they implemented a workload isolation strategy using Amazon Redshift data sharing. The customer launched a second 16-node ra3.4xlarge cluster as a data share consumer, with the primary cluster serving as the producer. This architecture allowed them to migrate consumption workloads to the consumer cluster while the producer focused on data ingestion, effectively supporting growth without increasing the primary cluster size.

Gaming company architecture with a producer Amazon Redshift cluster sharing data to a consumer cluster

Recognizing the advantages of this distributed architecture, the gaming company expanded their approach by migrating workloads to Amazon Redshift Serverless, further using the data sharing model for workload isolation. Amazon Redshift’s remote materialized view capability allowed the gaming company to create materialized views directly on the data shared by the producer cluster. Each consumer cluster could now build materialized views optimized for its specific workload patterns. This created pre-aggregated datasets, custom join strategies, and workload-specific data distributions, without impacting the producer cluster’s performance or requiring data duplication. The producer warehouse maintains data distribution and sorting strategies designed for generic enterprise needs, providing consistent data quality across all consumers. Meanwhile, consumer warehouses used remote materialized views to fine-tune query performance for their distinct analytical requirements, whether supporting real-time player analytics, business intelligence dashboards, or ad-hoc data science workloads. This distributed approach to data consumption optimization proved essential for the gaming company. It delivered fast query performance across diverse analytical workloads while maintaining a single source of truth in the producer cluster and avoiding the operational overhead of managing redundant data copies.

Best practices

To get the most out of these new capabilities, consider the following best practices:

Enable concurrency scaling on your Amazon Redshift clusters and Serverless workgroups to allow ETLs and user queries to run even faster, providing consistent report and dashboard performance.
Set up usage limits for concurrency scaling on both Amazon Redshift provisioned clusters and Serverless workgroups by configuring an appropriate MaxRPU setting. This helps you avoid unexpected additional costs. For more information, see the Amazon Redshift usage limits documentation.
Use remote MVs to offload resource-intensive MV creation and refresh operations from your primary warehouse to remote data share clusters.

Conclusion

In this post, we walked through the new MV refresh features, remote table DDL capabilities, and expanded concurrency scaling support for zero-ETL and S3 auto-copy. These features help you move beyond the constraints of a single warehouse. They are particularly valuable for organizations managing distributed data architectures that require dynamic table management across multiple environments while maintaining data consistency and adapting quickly to changing workloads. To get started, make sure you are running the latest Amazon Redshift version. Then visit the Amazon Redshift documentation to learn more about concurrency scaling, data sharing, and materialized views.

About the authors

Detecting fraud patterns across Snowflake and AWS using SageMaker Data Agent

2026-06-22 Akash Gupta

Post Syndicated from Akash Gupta original https://aws.amazon.com/blogs/big-data/detecting-fraud-patterns-across-snowflake-and-aws-using-sagemaker-data-agent/

Financial services organizations increasingly run analytical workloads across multiple systems. For example, customers typically store transaction records in Snowflake for its concurrency handling during peak volumes, while they store risk scores, customer profiles, and behavioral signals on AWS. To bridge that divide, practitioners have had to stitch together manual exports, custom extract, transform, and load (ETL) code, and external business intelligence (BI) tools to query both sources, cache expensive aggregations, and visualize results.

Amazon SageMaker Data Agent now closes these gaps with three new capabilities in Amazon SageMaker Unified Studio notebooks: SQL analytics on Snowflake data sources, materialized view management, and interactive charting. Practitioners can use them together to query Snowflake alongside AWS data, pre-compute and schedule repeated aggregations, and create interactive visualizations from natural language prompts in a single notebook, without writing boilerplate code or switching tools.

In this post, we describe the challenges these capabilities address, introduce each one, and walk through a fraud analytics scenario that demonstrates them working together in an end-to-end investigation workflow.

Challenges with fraud detection

Fraud analytics teams working in SageMaker Unified Studio notebooks encounter several recurring friction points that slow their path from alert to insight:

Querying across AWS and third-party warehouses. Customers store transaction data in Snowflake and maintain risk scores and customer profiles on AWS. SageMaker Data Agent supported SQL generation for AWS-native engines: Amazon Athena, Amazon Redshift, Apache Spark, and DuckDB. However, it didn’t yet generate Snowflake-dialect SQL. This created a gap for customers working with data distributed across both AWS services and Snowflake. Analysts had to write Snowflake SQL manually and export results as CSV files to join with AWS data. The process consumed 1–2 hours before any actual investigation could begin.
Rich visualization requires coding expertise. When analysts want to plot query results, they must write Python code using packages like matplotlib, seaborn, or plotly. They must choose the right chart type, format axes, handle data transformations, and debug rendering issues. For fraud teams whose expertise is in investigation rather than data visualization code, each chart becomes a detour: either learn the package interface, ask an engineer for help, or export to an external BI tool. This slows the exploratory cycle that fraud investigations depend on, where every new angle (time-of-day patterns, category breakdowns, geographic clusters) ideally takes seconds, not minutes of code iteration.
Expensive repeated queries with no caching. Fraud signal queries flag transactions that exceed a customer’s historical average and compute risk-score distributions by merchant category. These queries re-scan entire tables on each execution. A team running the same aggregation every morning over millions of rows pays the full compute cost each time, with no mechanism to pre-compute results or schedule automatic refreshes. For fraud teams, this means investigations start with a 30-minute wait for queries that ran identically yesterday.

These three friction points (accessing data across platforms, visualizing it interactively, and operationalizing repeated analyses) are what the new Data Agent capabilities address together.

What’s new in Data Agent

Snowflake connectivity

SageMaker Data Agent can now connect to Snowflake data warehouses through connections registered in Amazon SageMaker Unified Studio. The agent discovers available Snowflake databases, browses schemas progressively (databases → schemas → tables → columns), and generates Snowflake-dialect SQL, including Snowflake-specific syntax like FLATTEN, VARIANT column access, and semi-structured data handling. Analysts query Snowflake tables alongside AWS data sources from a single notebook conversation, and the agent handles dialect differences automatically: Snowflake SQL for extraction, Spark SQL for Amazon Simple Storage Service (Amazon S3) Tables operations, with no manual translation required.

Materialized view management

Data Agent now creates and manages materialized views through natural language prompts. Analysts describe the aggregation they want, for example, “create a materialized view that flags transactions where risk_score is above 0.7, refreshed every 6 hours,” and the agent generates the Spark SQL DDL, including SCHEDULE REFRESH syntax. Materialized views store pre-computed results in Apache Iceberg format for fast repeated access, turning expensive full-table scans into sub-second queries. Supported operations include create, refresh, drop, describe, and scheduled refresh. When asked, Data Agent can also analyze notebook query patterns and recommend which queries would benefit from materialization.

Interactive charting

Instead of generating matplotlib code that produces static images, Data Agent now creates native interactive chart cells powered by Vega-Lite. Supported chart types include bar, line, scatter, pie, area, heatmap, and more. Charts render inline in the notebook with hover tooltips, zoom, and filtering. Analysts can reconfigure them through the sidebar or by typing inline instructions like “change this to a heatmap showing volume by hour and category.” This removes the cycle of modifying Python plotting code or exporting to an external BI tool every time the analysis needs a different view.

Detecting fraud patterns across Snowflake and AWS: a walkthrough

Solution overview

In this section, we walk through how these three capabilities work together in a realistic fraud investigation. A fraud analytics lead at a mid-size fintech processes a high volume of card transactions daily. Customers store transaction data in Snowflake and maintain customer risk profiles on AWS.

This morning, the real-time alerting system flagged an unusual spike in declined transactions from a cluster of new accounts, all purchasing high-value electronics. The analyst suspects a fraud ring using synthetic identities, fabricated customer profiles that pass initial verification but share telltale patterns like similar device fingerprints or overlapping IP ranges. The analyst has three goals:

Confirm the fraud ring hypothesis. Determine whether the flagged accounts share device fingerprints, IP ranges, or behavioral patterns indicating coordinated fraud.
Quantify the exposure. Calculate total fraudulent transaction volume and identify all affected accounts, not only the ones that triggered today’s alert.
Set up ongoing monitoring. Create a reusable, auto-refreshing query so the team catches the next ring faster.

The analyst wants to do all of this without leaving the SageMaker notebook, without writing boilerplate data-engineering code, and within a single morning standup cycle so the investigations team can be briefed by noon.

How Data Agent approaches this analysis

Data Agent is context-aware. It discovers your actual table names, column schemas, and data source connections through Amazon SageMaker Unified Studio rather than requiring you to specify them manually. It generates SQL in the correct dialect for each source (Snowflake SQL for Snowflake, Spark SQL for S3 Tables) and operates within your existing AWS Identity and Access Management (IAM) permissions boundaries.

You interact with Data Agent in two modes: the Agent Panel for multi-step investigations like the example walkthrough that follows, where each prompt builds on previous context, and inline interactions for quick adjustments like “change this to a heatmap” directly on a chart cell.

Prerequisites

Before starting this walkthrough, verify that you have:

An Amazon SageMaker Unified Studio domain with a project configured.
A Snowflake account with a warehouse and USAGE grants on the database and schemas you want to query.
A Snowflake connection registered in your SageMaker Unified Studio project.
An S3 Tables catalog in your project containing customer data (or equivalent AWS-hosted tables for joining with Snowflake data).
A notebook open in SageMaker Unified Studio with Data Agent available in the chat panel.

Step 1: Explore Snowflake transaction data

What the analyst wants: Before investigating the fraud ring, the analyst must understand what data is available in Snowflake and verify recent transactions are accessible. The schema isn’t memorized (the payments team manages these tables), so Data Agent needs to discover the structure.

In the SageMaker notebook Agent Panel, the analyst types:

“Show me a preview of transactions over $500 for the last 24 hours. I’m looking for repeated high-value purchases that might indicate synthetic identity fraud.”

What Data Agent does for you: Data Agent discovers the Snowflake connection through SageMaker Unified Studio, browses the available databases, and locates PAYMENTS_DB → CARD_TRANSACTIONS schema → transactions table. It surfaces the column structure (transaction_id, customer_id, amount, merchant_category, transaction_timestamp, device_fingerprint, ip_address) so the analyst can confirm the right data is available without writing a single DESCRIBE TABLE statement.

Data Agent then generates a Snowflake-dialect SQL query to preview the last 24 hours of high-value transactions (amount > $500), returning hundreds of results. The preview immediately reveals what was suspected: alongside legitimate high-value purchases (mortgage payments, business supplies), there are clusters of electronics purchases at similar price points from different customer_id values but the same device_fingerprint, a classic synthetic identity pattern.

Data Agent querying Snowflake transaction data and generating equivalent code in the cell

Figure 1: Data Agent querying Snowflake transaction data and generating equivalent code in the cell.

Notebook cell results showing high-value Snowflake transactions

Figure 2: Displaying results when the notebook cell runs.

Step 2: Land Snowflake data into S3 Tables and join with risk profiles

What the analyst wants: Pulling historical high-value transactions into S3 Tables makes this data available for downstream analysis, including the materialized view that will cross-reference risk profiles automatically.

“Load the last 90 days of transactions where amount is greater than 500 into S3 Tables.”

What Data Agent does for you: Data Agent queries Snowflake to extract a large volume of high-value transactions from the last 90 days, converts the result to a PySpark DataFrame, creates an Apache Iceberg table at payments.fraud_analytics.high_value_transactions, and writes all the rows. Data Agent stores the transaction data (transaction_id, customer_id, amount, merchant_category, transaction_timestamp, device_fingerprint, ip_address) as Iceberg in S3 Tables, allowing you to query it entirely on AWS.

Data Agent handles the cross-source complexity: Snowflake-dialect SQL for extraction, automatic schema inference for the Iceberg table, and PySpark for the write. The analyst didn’t write a single line of ETL code.

Prompt sent to Data Agent to land Snowflake transactions into an S3 Tables

Figure 3: Sending a prompt to land Snowflake transactions into an S3 Tables catalog.

Generated PySpark code that reads transaction data from Snowflake

Figure 4: Reading data from Snowflake using code Data Agent generated.

Generated cell creating an S3 Tables Iceberg table populated with Snowflake data

Figure 5: Data Agent creating a new cell to create an S3 Tables Iceberg table and populate it with the Snowflake data.

Step 3: Create a materialized view for ongoing fraud monitoring

What the analyst wants: The pattern is confirmed, but re-running this expensive join across two tables every morning isn’t sustainable. A pre-computed view that automatically refreshes and surfaces transactions from high-risk customers means tomorrow’s investigation starts with answers instead of queries (goal #3, ongoing monitoring).

“Create a materialized view called mv_fraud_signals that joins high_value_transactions with customer_risk_profiles, flagging transactions where risk_score is above 0.7. Refresh it every 6 hours.”

What Data Agent does for you: Data Agent browses the S3 Tables catalog to discover both tables and their schemas, generates the Spark SQL DDL with SCHEDULE REFRESH EVERY 6 HOURS, and creates an INNER JOIN on customer_id with a risk_score > 0.7 filter. The resulting materialized view contains only the high-risk subset of transactions, and subsequent queries against it return significantly faster compared to a full table scan.

Data Agent can also recommend materialized views when asked. If the analyst prompts “analyze my notebook and suggest which queries would benefit from materialized views,” Data Agent examines query patterns and suggests candidates. This is useful when a team runs the same expensive aggregations repeatedly without realizing a materialized view would help.

New cell created by Data Agent to create the mv_fraud_signals materialized view

Figure 6: Data Agent creates a new cell to create the materialized view.

Generated query against the newly created materialized view

Figure 7: Data Agent adds code to query the newly created materialized view.

Step 4: Visualize fraud patterns with interactive charting

What the analyst wants: The data is ready, but the investigations team needs a clear visual story by noon to see which merchant categories are targeted and what time of day the fraud occurs, so they can build detection rules. The team needs interactive charts that can be explored on the fly, not static matplotlib images that need regenerating every time someone asks “what about category X?”

“Show me a scatter plot of flagged transactions: amount vs risk_score, colored by merchant_category.”

What Data Agent does for you: Data Agent queries the materialized view, generates a Vega-Lite specification, and renders an interactive scatter plot directly in the notebook cell, with no matplotlib code and no BI tool export. Hovering over any point reveals the transaction details. A dense cluster immediately stands out: Electronics & Computers transactions with risk scores between 0.75–0.95, all in the $950–$1,000 range.

Generated scatter plot of flagged transactions colored by merchant category

Detail view of the scatter plot highlighting the Electronics cluster

Figures 8, 9, and 10: Data Agent creates a scatter plot showing a dense cluster of Electronics transactions in the $950–$1,000 range with risk scores between 0.75–.95.

The analyst follows up with a second prompt to explore temporal patterns:

“Change this to a heatmap showing transaction volume by hour of day and merchant category.”

What Data Agent does for you: Data Agent generates a new heatmap visualization from the same materialized view. The heatmap reveals that Business Supplies and Mortgage Payments maintain steady transaction volumes throughout the day. However, Electronics shows a distinctly uneven temporal distribution, with noticeable volume dips during early morning hours (midnight to 5 AM) and late evening. This variability, absent in legitimate purchase categories, is a signal the detection rules team can act on immediately.

Heatmap of transaction volume by hour and merchant category

Detail view of the heatmap showing off-hours dips in the Electronics row

Figures 11 and 12: Data Agent creates a heat map to show transaction volume by hour of day and merchant category, revealing uneven temporal distribution in high-risk categories.

From insight to action

This investigation, from Snowflake connection to visual evidence, streamlined a workflow that previously required significant time across multiple tools. The analyst shares the notebook link with the investigations team, who confirm a fraud ring of dozens of synthetic identities responsible for significant fraudulent purchases. The temporal pattern, uneven Electronics transaction distribution with off-hours variability, is added to the company’s real-time detection rules that same afternoon.

The materialized view continues refreshing every 6 hours. The next morning, it flags three new accounts matching the same pattern, caught within hours of their first transaction instead of days.

Why SageMaker Data Agent for fraud analytics

This walkthrough demonstrates three new capabilities working together:

SQL analytics on Snowflake data sources removed the CSV export and manual ETL that consumed half of the investigation time.
Materialized view management turned a one-time query into persistent, auto-refreshing monitoring, transforming reactive investigations into proactive detection.
Interactive charting kept the entire analysis in the notebook, removing the BI tool context switch and making the inline exploration that revealed the Electronics temporal anomaly possible.

For the team, the combined effect is a reduction in time-to-insight, allowing faster fraud pattern analysis. This means daily fraud pattern reviews instead of weekly, and an investigation workflow that’s reproducible. The notebook itself serves as documentation for compliance and audit purposes.

Cleanup

The walkthrough creates notebook cells, SQL queries, and materialized views in your SageMaker Unified Studio session. To remove the generated cells, delete them from your notebook or delete the notebook itself.

If you created resources specifically for this walkthrough, remove the following to avoid ongoing charges:

Materialized view. In the notebook Agent Panel, prompt: “Drop the materialized view mv_fraud_signals.” This removes the Iceberg table from S3 Tables and cancels the scheduled refresh. Alternatively, run the Spark SQL statement DROP MATERIALIZED VIEW payments.fraud_analytics.mv_fraud_signals directly.
Landed Iceberg tables. Drop any tables created during the data landing step (for example, payments.fraud_analytics.high_value_transactions) by prompting Data Agent or running DROP TABLE in a Spark SQL cell. This removes the data from S3 Tables and the underlying Amazon Simple Storage Service (Amazon S3) storage.
SageMaker Unified Studio domain. If you created a domain solely for this walkthrough, delete it to stop incurring charges. Refer to the SageMaker Unified Studio administration guide for deletion steps.
Amazon S3 storage. Verify that dropping the materialized view and Iceberg tables removed the associated S3 objects. If residual Iceberg metadata files remain in your S3 Tables bucket, delete them manually.
Snowflake compute. No persistent Snowflake resources are created. Queries use your existing warehouse. Review your Snowflake query history to estimate the compute credits consumed during the walkthrough.

Conclusion

In this post, we walked through three new capabilities in Amazon SageMaker Data Agent for notebooks: Snowflake connectivity, materialized views, and native interactive charting. Using a fraud analytics scenario, we demonstrated how these features work together. We connected to a Snowflake warehouse to explore transaction data, landed results into S3 Tables and joined them with AWS-hosted risk profiles, created a materialized view for ongoing fraud monitoring, and visualized patterns with interactive charts that revealed temporal anomalies in Electronics transactions linked to dozens of synthetic identities.

These capabilities are available now in Amazon SageMaker Unified Studio. To get started, open a notebook in your SageMaker Unified Studio domain and begin a conversation with Data Agent in the chat panel.

To learn more, see the following resources:

Amazon SageMaker Data Agent documentation.
Accelerate context-aware data analysis and ML workflows with Amazon SageMaker Data Agent (related blog post).
Amazon SageMaker Data Agent introduces charting capabilities and support for materialized views (What’s New post).

About the authors

Introducing Private Networking for Amazon MQ for RabbitMQ

2026-06-19 Jean-Sébastien Dominique

Post Syndicated from Jean-Sébastien Dominique original https://aws.amazon.com/blogs/big-data/introducing-private-networking-for-amazon-mq-for-rabbitmq/

With Private Networking for Amazon MQ for RabbitMQ, your brokers can establish outbound connections to private resources in your VPC without exposing those resources publicly. This post explains how the feature works and walks you through setting it up.

Amazon MQ for RabbitMQ brokers could previously only reach external destinations over the public internet. If you used a private Lightweight Directory Access Protocol (LDAP) server for broker authentication, you had to expose that server publicly. If you wanted to federate messages between private brokers, you needed workarounds like Network Load Balancers with IP allowlisting, as described in Implementing Federation on Amazon MQ for RabbitMQ Private Brokers. Private Networking removes those constraints.

You can connect your broker to private identity providers, other Amazon MQ for RabbitMQ brokers, or self-hosted RabbitMQ brokers running in private subnets. Combined with cross-Region networking services like AWS Transit Gateway, you can extend these connections across AWS Regions and accounts, with traffic staying on the AWS private network.

How it works

Private Networking connects your broker to private destinations using three AWS services: Amazon VPC Lattice, AWS Resource Access Manager (AWS RAM), and AWS PrivateLink.

You create a VPC Lattice resource gateway in a VPC that can reach your private destination. You then create a VPC Lattice resource configuration that defines the destination, such as an IP address or Domain Name System (DNS) name. You add the resource configuration to a RAM resource share and associate the resource share with your broker through the UpdateBroker API operation. After rebooting the broker, the network path is active and your broker can reach the private destination.

The broker does not need to be private. A publicly accessible broker works the same way.

What you can connect to

Private Networking supports three use cases.

Private identity providers

If you use an LDAP server or other identity provider for RabbitMQ authentication, you no longer need to expose it publicly. Create a resource configuration pointing to your identity provider, associate it with your broker, and use the DNS name returned by the DescribeSharedResources API operation in place of the public endpoint. Follow the existing guidance for setting up an identity provider, substituting the private DNS name.

Self-hosted RabbitMQ brokers

You can use Shovel or Federation to connect your Amazon MQ for RabbitMQ broker to a self-hosted RabbitMQ broker running in a private subnet. Create a resource configuration pointing to the self-hosted broker and use the DNS name from the DescribeSharedResources API operation in your Shovel or Federation configuration.

This pattern is useful for hybrid cloud architectures where you run RabbitMQ on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), or on-premises infrastructure and want to exchange messages with Amazon MQ without exposing either side publicly.

Other Amazon MQ for RabbitMQ brokers

You can federate or shovel messages between two Amazon MQ for RabbitMQ brokers using Private Networking. Create a resource configuration pointing to the destination broker’s endpoint and specify that same endpoint as the custom domain name on the resource configuration. This helps to verify that the DNS name resolves correctly and Transport Layer Security (TLS) peer verification succeeds.

This extends to brokers in different AWS Regions and different AWS accounts. By combining Private Networking with cross-Region networking services like AWS Transit Gateway or VPC peering, you can build a fully private federation or shovel path between brokers, with no public endpoints required.

DNS names and custom domains

Each resource configuration can include a custom domain name. If you add a verified domain, that domain resolves to the private destination. If you do not add a verified domain, Amazon MQ provides a DNS name for the broker’s private connection. Retrieve this DNS name with the DescribeSharedResources API operation.

If you specify an unverified domain on a resource configuration, it is ignored. The broker’s private connection receives a private DNS name instead, which you can retrieve with the DescribeSharedResources API operation.

For more details on custom domain names and domain verification with VPC Lattice, see Custom domain names for VPC Lattice resources.

TLS peer verification in RabbitMQ 4

Note: If you are running RabbitMQ 4, review this section before configuring Shovel or Federation connections.

RabbitMQ 4 enforces TLS certificate peer verification by default for Shovel and Federation connections. RabbitMQ 3 does not enforce this by default. When using Private Networking, the DNS name that Amazon MQ assigns to the private connection will not match the TLS certificate of the destination, which causes peer verification to fail.

The recommended approach is to specify the destination broker’s endpoint (for example, b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111.mq.us-east-1.on.aws) as the custom domain name on the resource configuration. This exception only applies to Amazon MQ for RabbitMQ broker endpoints. You cannot use an unverified domain for self-hosted brokers. Specifying the Amazon MQ endpoint causes the DNS name to match the destination’s TLS certificate, and peer verification succeeds. This approach works regardless of your RabbitMQ version and avoids the issue entirely.

Getting started

To get started with Private Networking for Amazon MQ for RabbitMQ, follow these steps.

Prerequisites

Before you begin, verify you have the following:

An AWS account.
The AWS Command Line Interface (AWS CLI) installed and configured.
AWS Identity and Access Management (IAM) permissions to manage Amazon MQ, VPC Lattice, and AWS RAM resources.
An existing VPC with connectivity to your private destination.

Walkthrough

After you have the prerequisites, follow these steps:

Create an Amazon MQ for RabbitMQ broker if you do not already have one.
Create a VPC Lattice resource gateway in a VPC that can reach your private destination. Make sure the resource gateway’s security group allows outbound traffic to your destination on the required port (for example, port 5671 for AMQPS (AMQP over TLS) or port 636 for LDAPS (LDAP over TLS)). The resource gateway must share at least one Availability Zone with the broker. Cluster brokers cover multiple Availability Zones, so this is satisfied. For single-instance brokers, verify the Availability Zone overlap.
Create a VPC Lattice resource configuration pointing to your private destination (IP address or DNS name). If you’re connecting to another Amazon MQ broker, specify the destination broker’s endpoint as the custom domain name on the resource configuration, as shown in the following figure.Figure 1: VPC Lattice resource configuration showing the custom domain name field and resource definition populated with the Amazon MQ broker endpoint.
Add the resource configuration to a RAM resource share. The resource share must allow external principals, as shown in the following figure.Figure 2: RAM resource share configuration with the Allow external principals option selected.
Associate the resource share with your broker by editing the broker and adding the resource share. You can also do this using the update-broker command with the AWS CLI. You must pass the entire list of resource share ARNs you want on the broker. This is a put operation, not an add or remove operation.
```
aws mq update-broker \
  --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \
  --resource-share-arns arn:aws:ram:us-east-1:111122223333:resource-share/a1b2c3d4-5678-90ab-cdef-EXAMPLE22222
```
The associated RAM resource share appears as shown in the following figure.

Figure 3: Network settings view with associated RAM resource shares.

Select the resource share in the Associated RAM resource shares section. The network status of each shared resource is displayed in the Shared resources section, as shown in the following figure.

Figure 4: RAM resource share selection showing the network status of each shared resource.
Reboot the broker from the AWS Management Console or the AWS CLI to create the network path:
```
aws mq reboot-broker --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
```
Retrieve the DNS names for your RabbitMQ configuration. This operation also surfaces issues encountered during setup:
```
aws mq describe-shared-resources --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
```
Use the DNS name returned in the output in your Shovel, Federation, or identity provider configuration. Adding new resource configurations to an existing RAM resource share does not automatically update the broker. You must call update-broker and reboot the broker for the new resource configurations to take effect.

Cleaning up

Private Networking uses VPC Lattice and PrivateLink resources that incur ongoing charges. If you no longer need the private connection:

Call update-broker with the resource share removed from the list (or an empty list to remove all), then reboot the broker.
After the broker reboot completes and the resources are no longer in use, delete the VPC Lattice resource configuration and resource gateway.
Optionally, remove the Amazon MQ account principal from the RAM resource share. This principal may still be in use if other brokers are associated with the same resource share, so only remove it if no other brokers depend on it.
If you created a new Amazon MQ for RabbitMQ broker for this walkthrough and no longer need it, delete the broker from the Amazon MQ console or with the delete-broker command.

Operational behavior: Resource access and reboots

Removing a VPC Lattice resource configuration from a RAM resource share while the broker is actively using it revokes access immediately, with no reboot required. Removing a principal from a RAM resource share has the same effect: brokers associated through that principal lose access to the resources in the share immediately. These are intentional security behaviors managed by RAM and VPC Lattice.

Adding new resource configurations to an existing resource share does not take effect automatically. You must call update-broker and reboot the broker for the new resource configurations to take effect. This is by design. It helps verify that changes to a resource share only reach the broker when someone with broker management permissions explicitly triggers the update, providing clear security separation between share management and broker management.

Private Networking is available for Amazon MQ for RabbitMQ brokers in all the AWS Regions where Amazon VPC Lattice is available. Amazon MQ for ActiveMQ brokers do not support this feature.

Pricing

Private Networking uses Amazon VPC Lattice and AWS PrivateLink. Data processing and data transfer charges apply to traffic sent through the private connection. There is an Amazon MQ pricing of $0.01 per GB of data processed through the resource endpoint. For details, see the Amazon MQ pricing page, VPC Lattice pricing page and AWS PrivateLink pricing page.

Conclusion

In this post, we explained how Private Networking for Amazon MQ for RabbitMQ works and walked through the setup process. Whether you’re securing a private identity provider, federating messages between brokers, or connecting to self-hosted RabbitMQ, your broker can now reach private destinations without exposing them publicly.

To learn more, see the Amazon MQ Private Networking documentation.

If you have questions or feedback, leave a comment on this post.

About the authors

Spring 2026 SOC 1 and 2 reports are now available in OSCAL format

2026-06-18 Thomas Fischer

Post Syndicated from Thomas Fischer original https://aws.amazon.com/blogs/security/spring-2026-soc-1-and-2-reports-are-now-available-in-oscal-format/

Amazon Web Services (AWS) is excited to release the Spring 2026 System and Organization Controls (SOC) 1 and 2 reports in machine-readable OSCAL format alongside the PDF version of the reports. The reports cover 188 services over the 12-month period from April 1, 2025 to March 31, 2026, giving customers a full year of assurance. These reports demonstrate our continuous commitment to adhering to the heightened expectations of cloud service providers.

AWS is the first major cloud provider to offer key compliance reports to customers in the National Institute of Standards and Technology’s (NIST) Open Security Controls Assessment Language (OSCAL), as of June 2026. OSCAL is an open source, machine-readable (JSON) format for security information. The SOC 1 and SOC 2 report package in OSCAL format is now available as a distinct package in AWS Artifact, marking a milestone toward open, standards-based compliance automation. This machine-readable version of the SOC report package enables workflow automation to reduce manual processing time and modernize security and compliance processes. Your use cases for this content are innovative, and we want to hear about them through the contact information found in the OSCAL report package.

You can download the Spring 2026 SOC 1 and 2 reports in OSCAL format through AWS Artifact, a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact. The SOC 3 report can be found on the AWS SOC Compliance page and in AWS Artifact.

AWS strives to continuously bring services into the scope of its compliance programs to help customers meet their architectural and regulatory needs. You can view the current list of services in scope on our Services in Scope page. As an AWS customer, you can reach out to your AWS account team if you have any questions or feedback about SOC compliance.

To learn more about AWS compliance and security programs, see AWS Compliance Programs.

If you have feedback about this post, submit comments in the Comments section below.