Tag Archives: announcements

Announcing the Cloud Security Alliance on AWS Compliance Guide

Post Syndicated from Juan Rodriguez original https://aws.amazon.com/blogs/security/announcing-the-cloud-security-alliance-on-aws-compliance-guide/

AWS Security Assurance Services is announcing the release of the Cloud Security Alliance (CSA) Compliance Guide on Amazon Web Service (AWS), a new resource that maps the 17 control domains and 207 control objectives of the Cloud Controls Matrix v4.1 (CCM) to AWS services and recommended implementation practices. The guide is intended to help organizations using AWS plan, implement, and evidence the controls relevant to their CCM scope, including those pursuing or maintaining CSA STAR certification.

What is the Cloud Controls Matrix?

The Cloud Security Alliance is a not-for-profit organization dedicated to defining and raising awareness of best practices for cloud security. AWS maintains CSA STAR Level 2 certification, which couples the requirements of ISO/IEC 27001:2022 with the CCM. The CSA STAR documentation and the AWS Consensus Assessments Initiative Questionnaire (CAIQ) are available to AWS customers through AWS Artifact.

The CSA Cloud Controls Matrix is a cybersecurity controls framework developed by the CSA that provides a detailed set of security controls mapped across multiple domains (such as audit and assurance, identity and access management, and encryption and key management) specifically designed to assess and manage security risks in cloud computing environments. The CCM is cloud agnostic, designed to be applicable to any cloud service provider or deployment model, such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS), regardless of the underlying technology or vendor, providing universal security controls that organizations can apply across cloud platforms.

Responsibility models

CCM defines its own Shared Security Responsibility Model (SSRM) with three categories: Cloud service provider (CSP)-owned, customer-owned, and shared (independent or dependent). The guide recommends using the SSRM together with the AWS Shared Responsibility Model. For controls that AWS owns, the guide points to AWS attestations available through AWS Artifact (for example, SOC reports, ISO certificates, and the CSA STAR attestation) as inherited evidence. For controls that customers own or share, the guide describes how to implement and evidence them using AWS services. Using a CSA STAR–certified AWS service doesn’t by itself make a customer workload compliant. Customers remain responsible for configuring services, managing access, protecting data, and implementing additional controls based on their environment, risk assessments, and regulatory obligations. The guide is informational and doesn’t replace the AWS compliance documentation and certifications available through AWS Artifact.

What’s inside the guide

For each control, the guide states applicability, describes how organizations can implement the control on AWS, identifies common pitfalls, and lists examples that can be used as evidence during an assessment.

Related resources

Download available: Here

For further assistance, contact AWS Security Assurance Services. If you have feedback about this post, submit comments in the Comments section below.


Juan Rodriguez

Juan is a Security Assurance Consultant at AWS, where he works with Strategic Services and customers to assess and secure cloud environments against frameworks including CMMC, FedRAMP, GovRAMP, and NIST based practices. He holds his CMMC Certified Professional and AWS Certified Security – Specialty certifications. Juan pairs technical expertise with a research-driven mindset to help organizations strengthen and architect their security posture and align with federal and industry standards.

Max Schiessl

Max Schiessl

Max is an Assurance Consultant with AWS Security Assurance Services (SAS), focusing on multiple international cybersecurity and compliance frameworks. He combines a decade of experience as an IT auditor and certified penetration tester and has contributed to the development and implementation of cybersecurity regulations at the European national level. He works closely with customers, partners, and AWS teams to help organizations achieve and maintain their compliance goals in the cloud.

Meg Winn

Meg Winn

Meg is a Sr. Security Assurance Consultant at Amazon Web Services with over a decade of experience in cybersecurity governance, risk and compliance, cloud security strategy, and compliance engineering. She holds multiple industry and AWS certifications and advises enterprise clients on achieving and sustaining audit readiness across multiple standards. Meg partners with executive stakeholders in highly regulated industries to prepare and guide clients through successful audit outcomes at scale.

Satish Uppalapati

Satish is an Associate Assurance Consultant with AWS Security Assurance Services (SAS) and has more than 8 years of experience in IT risk, governance, and regulatory assurance. He works with AWS customers to align cloud environments with multiple frameworks. Satish helps organizations build security and governance programs that meet regulatory objectives while supporting business operations. He also focuses on advancing governance for AI systems, including emerging standards.

Ted Tanner

Ted Tanner

Ted is a Principal Assurance Consultant and PCI DSS Qualified Security Assessor with AWS Security Assurance Services, and has more than 25 years of IT and security experience. He leverages this to provide AWS customers with guidance on compliance and security, and build and optimize their cloud compliance programs.

AWS Weekly Roundup: Local Zone in Athens, Claude Opus 5 on AWS, Lambda durable execution for .NET, and more (July 27, 2026)

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-july-27-2026/

Last week I had the privilege of spending three days in São Paulo with technical builders from across Latin America, brought together for a regional tech event full of deep-dive sessions, hands-on workshops, and conversations with customers and partners. What struck me most wasn’t any single session, it was the energy of a technical community that so rarely gets to be in the same room. People traded architecture ideas over coffee, sketched out solutions on whiteboards, and left with a longer list of things to try than they arrived with. It’s a good reminder that, for all the tooling we build, the community around it is what makes the technology stick.

That community spirit connects nicely to the week’s biggest infrastructure news, which is all about bringing AWS closer to where builders actually are.

Now, let’s get into this week’s AWS news…

Headlines
AWS Local Zone in Athens, Greece: AWS has opened a new Local Zone in Athens, Greece, the second Local Zone in EMEA with support for Amazon S3 and Amazon EBS Local Snapshots, so you can store and process data within Greece to help meet local data residency requirements. The Athens Local Zone supports Amazon EC2 (C7i, M7i, and R7i instances), Amazon S3 with the One Zone-Infrequent Access storage class, Amazon EBS, and Amazon ECS.

Athens, Greece skyline

AWS Local Zones place AWS infrastructure much closer to large population and industry hubs, enabling applications that require single-digit millisecond latency, such as real-time gaming, media production, and financial services, to run where end users actually are. For builders in Greece, you can now run latency-sensitive workloads locally while connecting seamlessly to the nearest AWS Region for services that don’t require low latency, giving you the flexibility to architect hybrid, latency-optimized applications without managing your own data center infrastructure. To learn more, visit AWS Global Infrastructure and Sustainability Blog post.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Claude Opus 5 on AWS: You can use Anthropic’s Claude Opus 5, the most advanced Opus model yet, matching Claude Fable 5’s top-tier intelligence in many domains at Opus-tier pricing. Amazon Bedrock offers Claude Opus 5 with zero data retention (ZDR) enabled by default, giving you Opus’ top-tier intelligence while meeting your data governance requirements unlike Claude Fable 5. You have two ways to access Claude Opus 5: Amazon Bedrock and Claude Platform on AWS. To learn more, visit the deep dive blog post.
  • AWS Lambda durable execution SDK for .NET is now generally available: You can now build resilient, long-running workflows in C# using Lambda durable functions, without implementing custom progress tracking or integrating an external orchestration service. The SDK is a natural fit for multi-step applications like payment processing pipelines, AI agent orchestration, and human-in-the-loop approvals, it checkpoints progress automatically and can pause execution for up to a year. If you’re a .NET developer building serverless workflows, this removes a lot of the plumbing you used to write by hand.
  • Amazon Bedrock AgentCore now delivers unified observability with traces and logs in a single log group: Amazon Bedrock AgentCore now delivers agent traces and prompts to the same Amazon CloudWatch log group as your agent’s logs. Previously, telemetry was split across destinations, trace spans went to a shared log group while prompts, inputs, and outputs went to a separate one, so debugging a single agent invocation meant searching in multiple places. You can now debug an invocation in one place, and apply fine-grained access control and customer-managed key (CMK) encryption at the individual agent level.
  • Amazon Connect delivers more natural agentic voice experiences: Amazon Connect now supports more natural, human-sounding agentic voice experiences across 50+ languages, including Portuguese, Spanish, French, Italian, Japanese, Korean, and Thai, with over 100 new voice options and conversational improvements that make AI interactions sound more fluid. Connect’s agentic self-service lets AI agents understand, reason, and take action across voice and digital channels, adapting to a customer’s tone and sentiment. You can now build contact center experiences that feel natural to callers in far more of the languages your customers actually speak.
  • Amazon SageMaker Unified Studio now supports Amazon OpenSearch: You can now query and analyze your search and log analytics data from Amazon OpenSearch directly alongside other data assets in Amazon SageMaker Unified Studio. With this connection, you can combine operational search data in OpenSearch with data from sources like Amazon Redshift, Amazon S3, and relational databases, all within a single, governed environment. It’s especially useful when you need to correlate analytical and operational workloads, such as joining application logs with transactional data to uncover insights.
  • Amazon CloudWatch announces coding agent insights: Amazon CloudWatch now gives engineering leaders visibility into how AI coding tools are driving value across their organization. Coding agent insights integrates with the Claude apps gateway for AWS to collect telemetry from Claude Code without additional instrumentation, and also supports agents like Codex and GitHub Copilot. As teams scale AI coding adoption, you can now measure the return on that investment with metrics built on OpenTelemetry, no custom instrumentation required.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Summits: AWS Summits are free events that bring the cloud and AI community together to connect, learn, and explore the latest technologies. Browse the full calendar to find a Summit near you in the second half of 2026.
  • AWS Community Days: Community-led conferences where content is planned, sourced, and delivered by community leaders. If you’re in Latin America, don’t miss AWS Community Day Belo Horizonte on August 22, registration is open at awscommunityday.com.br.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Enterprise security at machine speed: AWS Black Hat 2026 preview

Post Syndicated from Rahul Sahni original https://aws.amazon.com/blogs/security/enterprise-security-at-machine-speed-aws-black-hat-2026-preview/

Black Hat 2026 (Aug 1-6, 2026) brings together over 22,000 security practitioners, researchers, and CISOs who build, break, and defend enterprise infrastructure. They’re security professionals who push the limits of offensive and defensive security and demand proof over promises. As frontier security models like Mythos reshape the enterprise landscape, they need security that operates at the same speed as the events they face. This August, Amazon Web Services (AWS) returns to Las Vegas to meet with our customers and partners to show how we’re delivering enterprise security at machine speed.

At Black Hat USA 2026, connect with AWS through live demos, a practitioner session on autonomous security operations, and an executive roundtable on building durable AI security architectures, plus networking receptions with customers and partners. Here’s where to find us and what you’ll take away.

Experience AWS security innovation in action

Visit us at Booth #1648 to explore five interactive demo pedestals, each aligned to a pillar of our AI-powered security story:

  • Post-Mythos enterprise security: AWS uses multiple frontier AI reasoning models to find issues, validate exploitability, and autonomously fix exposures, compressing mean-time-to-remediation from days to minutes. Learn how AWS Continuum helps you find risks before you ship, prioritize by real business impact, and remediate at machine speed while you stay in control.
  • AI-powered investigations: Amazon GuardDuty AI-powered investigations automatically analyze findings and the accounts around them to separate true threats from benign activity at scale. Drawing on log activity, resource configuration, internet reachability, and historical findings, it delivers triage reports that match expert-level accuracy, freeing your team to focus on what matters.
  • Purpose-built AI workload security: AWS extends the enterprise security controls you already trust to address the unique demands of AI workloads. Amazon Bedrock helps you build and deploy generative AI applications, with hundreds of top foundation models plus built-in safety controls, guardrails, and evaluation tools. Amazon Bedrock AgentCore Identity keeps you in full control of what your AI agents can access.
  • Full-stack multicloud security: AWS Security Hub Extended delivers full-stack security by integrating AWS services with 21 curated partner solutions across nine categories, from endpoint and identity to AI, with pay-as-you-go pricing, one console, and one bill. All findings flow in OCSF format with zero integration work, and a risk correlation engine traces paths across multicloud so your team fixes root causes rather than chasing symptoms.
  • AWS Partner solutions: A dedicated partner demo pedestal runs live demonstrations from select AWS Partners, showing how their solutions integrate with AWS to address your most critical security challenges. With over 25 partners rotating throughout Business Hall hours, you get a firsthand view of how these integrations work together to strengthen your security posture.

Theater sessions: Want to go deeper on a specific topic without committing to a full session? During expo hours, our theater hosts 15-minute AWS and partner talks curated around the challenges enterprise security teams face today. Topics range from enabling AI adoption securely, to unifying visibility across clouds, to automating response. Subject matter experts from AWS and our partners lead each session and give you insights you can bring back to your team. Featured sessions include:

  • From finding to fix at machine speed with AWS Continuum
  • Securing the agentic AI stack
  • Architecting defense-in-depth for AI workloads
  • Network security strategies for the post-Mythos era
  • One console for unified full-stack security

For the latest theater session schedule, including dates and times, visit AWS at Black Hat 2026.

Engagement Zone: Take a quick break and test your detection skills in a gamified, hands-on challenge. Race the clock to resolve security issues and collect power-ups that speed your mission to secure the environment. Rack up your score, lock in your percentile rank, and see how you stack up against every other player.

Beyond the booth: Sessions and executive roundtable

AWS security experts share insights at speaking sessions and host an executive roundtable at Black Hat USA 2026.

Speaking Session | Machine-Speed Defense: Building an Autonomous Security Operations Loop for the AI Era
Wednesday, August 5, 10:15–10:35 AM PDT
Pulse Stage 2
In a post-Mythos world, organizations need to use AI-powered reasoning to discover, correlate, validate, and remediate security exposures at machine speed. This session provides a blueprint for building an autonomous security operations loop, using business-context graphs and sandbox validation, to unlock machine-speed defense as a durable competitive edge.

Executive Roundtable | Autonomous Defense at Cloud Scale: Critical Choices for Security Leaders Today
Wednesday, August 5, 11:00 AM–12:00 PM PDT
Breakers F
As agentic AI and frontier models reshape the threat landscape, foundational security remains essential, and organizations that layer autonomous workload governance on top can turn AI into a true security force multiplier. This executive roundtable explores how to build a durable, adaptive security harness that accounts for the real economics of AI-powered defense and evolves alongside rapidly advancing models and techniques.

Briefing | ThreatForest: Automated Attack Trees from Source Code
Thursday, August 6, 2:35–3:15 PM PDT
Jasmine, Level 3
Threat modeling is critical, but manual processes can’t keep pace with cloud-focused architectures. This briefing will dive into how ThreatForest uses six specialized AI agents to automatically analyze source code and produce validated attack trees mapped to MITRE ATT&CK with actionable mitigations. Attendees walk away with the open source tool and a reusable multi-agent architecture pattern.

AWS activities and events

Beyond the expo floor, AWS hosts a portfolio of ancillary events built for focused conversations and networking across the security community. Join us at:

Security LIVE! broadcast

Catch Security LIVE! on-site at Mandalay Bay on Wednesday, August 5 and Thursday, August 6. This infotainment-style broadcast brings AWS and AWS Partners together to solve real security challenges for customers across 20-minute conversational segments, covering everything from data protection and compliance to application and perimeter security. With over 30 segments, Security LIVE! showcases the breadth of the AWS security partner ecosystem. Stop by the set to watch it unfold live.

Join us in Las Vegas

Whether you’re exploring how to secure AI workloads, adopting autonomous remediation as frontier models like Mythos reshape the landscape, or unifying security across domains and clouds, the AWS team at Black Hat USA 2026 is ready to help.

Learn more about AWS Security solutions at AWS Cloud Security.

See you in Las Vegas, August 3–6, 2026.


Rahul Sahni

Rahul Sahni

Rahul is a Senior Product Marketing Manager at AWS Security. An avid Amazonian, he embodies the company’s Leadership Principle of Learn and Be Curious in both his professional and personal life, whether diving into emerging security topics or experimenting with new dishes from around the world.

2026 ISO and CSA STAR certificates are now available with two additional services

Post Syndicated from Atulsing Patil original https://aws.amazon.com/blogs/security/2026-iso-and-csa-star-certificates-are-now-available-with-two-additional-services/

Amazon Web Services (AWS) successfully completed an onboarding audit with no findings for ISO 9001:2015, 27001:2022, 27017:2015, 27018:2019, 27701:2019, 20000-1:2018, and 22301:2019, and Cloud Security Alliance (CSA) STAR Cloud Controls Matrix (CCM) v4.0. EY Certify Point auditors conducted the audit and reissued the certificates on May 31, 2026. The objective of the audit was to enable AWS to expand their ISO and CSA STAR certifications to include two additional services. The ISO standards cover areas including quality management, information security, cloud security, privacy protection, service management, and business continuity. The certifications demonstrate AWS’s commitment to maintaining robust security controls and protecting customer data across our services.

During this onboarding audit, we added two additional AWS services to the scope since the last certification issued on February 25, 2026.
Following are the two additional services:

For a full list of AWS services that are certified under ISO and CSA Star, see the AWS ISO and CSA STAR Certified page. Customers can also access the certifications in the AWS Management Console through AWS Artifact.

If you have feedback about this post, submit comments in the Comments section below.


Atulsing Patil

Atulsing is a Compliance Program Manager at AWS. He has 29 years of consulting experience in information technology and information security management. Atulsing holds a Master of Science in Electronics degree and professional certifications such as CCSP, CISSP, CISM, ISO 42001 Lead Auditor, ISO 27001 Lead Auditor, HITRUST CSF, Archer Certified Consultant, and AWS CCP.

Shalini Mishra

Shalini Mishra

Shalini is a Compliance Program Manager at AWS. She has 5+ years of experience leading end-to-end compliance programs across ISO, SOC, and cloud security frameworks, with deep expertise in third-party risk management and enterprise governance, driving measurable improvements in security posture and audit readiness. Shalini holds a Master of Science degree in Information Systems, CRISC and ISO 27001 lead auditor certification.

Zero Copy access to Apache Iceberg tables in Amazon S3 from Salesforce Data 360 using the Iceberg REST endpoint from AWS Glue Data Catalog

Post Syndicated from Avijit Goswami original https://aws.amazon.com/blogs/big-data/zero-copy-access-to-apache-iceberg-tables-in-amazon-s3-from-salesforce-data-360-using-the-iceberg-rest-endpoint-from-aws-glue-data-catalog/

Companies increasingly need to query and analyze data across platforms without the cost and complexity of moving it. Salesforce and AWS have collaborated to make this possible by providing Zero Copy access to Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3) directly from Salesforce Data 360, using the Iceberg REST endpoint from AWS Glue Data Catalog with data access managed by AWS Lake Formation. This integration helps customers federate their Amazon S3 data lakes with Data 360, preserving data governance, freshness, and business semantics without replication.

Zero Copy file federation plays an important role in activating applications and experiences. By removing the need to physically move or copy data, and connecting to data at the storage level, it addresses key challenges including:

  • Cost efficiency – Reduce storage duplication costs and minimize the compute resources required for data pipelines.
  • High scale – Access data with near-native performance at scale through in-Region access.
  • Enhanced agility – Access and analyze data in real time, accelerating time-to-insight and supporting faster response to evolving business needs.
  • Streamlined operations – Remove the complexity of building and maintaining intricate data pipelines, clearing up valuable data engineering resources.

In this post, we demonstrate how AWS and Salesforce customers can access their enterprise data lakes on AWS from Data 360 using Zero Copy file federation.

What is Data 360?

Data 360 is the real-time data engine that activates trusted context across the entire Salesforce platform. It connects all your enterprise data — data warehouses, data lakes, third-party signals, and more — to the business context, logic, and governance that already live in Salesforce, without moving or copying it. With Zero Copy federation, your teams and AI agents always operate from a complete, current, and trusted picture of your business in the moment it’s needed. It serves as the essential system of context for Agentforce, enabling agents to reliably get real work done.

What is Apache Iceberg?

Apache Iceberg is a high-performance, open table format for huge analytic datasets that brings the reliability and simplicity of SQL tables to big data. It’s a thriving open source project under the Apache Software Foundation. Data engineers use Apache Iceberg because it’s fast, efficient, and reliable at any scale and keeps records of how datasets change over time. Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more.

Why Amazon S3 for Apache Iceberg data lakes?

Amazon S3 is regarded as the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities, and its ability to integrate with a broad portfolio of AWS and third-party tools for data ingestion and processing. Apache Iceberg was designed and built to interact with Amazon S3, and provides support for many Amazon S3 features as listed in the Iceberg documentation.

What is Zero Copy file federation?

File federation, also termed catalog federation, uses the Data Catalog to communicate with remote catalog systems to discover catalog objects and to authorize access to their data in Amazon S3. When you query a remote Iceberg table, the Data Catalog discovers the latest table information in the remote catalog at query runtime, getting the table’s Amazon S3 location, current schema, and partition information. Your analytics engine then uses this information to access Iceberg data files directly from Amazon S3, and Lake Formation manages access to the table and data by vending scoped credentials to the table data stored in Amazon S3. This approach avoids metadata and data duplication while providing real-time access to remote Iceberg tables through your preferred AWS analytics engines.

Solution overview

Apache Iceberg file federation lets Data 360 directly query data stored in Amazon S3 without copying or moving the data. This Zero Copy approach provides several benefits:

  • Real-time access to Amazon S3 data from Salesforce.
  • Reduced data movement and storage costs.
  • Simplified data architecture.
  • Improved data freshness.

The following diagram illustrates the architecture of the integration between Data 360 and Amazon S3 using Apache Iceberg file federation.

Key components:

  1. Amazon S3 stores the source data in Apache Iceberg format.
  2. AWS Glue Data Catalog maintains the metadata for Iceberg tables.
  3. AWS Glue Iceberg REST endpoint provides RESTful access to Iceberg tables.
  4. AWS Lake Formation manages metadata and underlying data access for Amazon S3-based data lakes.
  5. Data 360 processes and analyzes the data.
  6. Apache Iceberg connector provides direct access to query Amazon S3 data from Salesforce.

Walkthrough

The following walkthrough shows you how to set up Zero Copy file federation.

Prerequisites

Before you begin, you need the following:

Configure your AWS environment

Set up an Amazon S3 bucket and Iceberg table

Sign in as the data lake admin and complete the following steps:

  1. Open the Amazon S3 console.
  2. Choose Create bucket to create a bucket.
  3. For Bucket type, choose General purpose, provide a Bucket name, and choose Create bucket.
  4. In the bucket, create two prefixes by choosing Create folder.
  5. Name the prefixes athena_iceberg and athena_results.
  6. Inside the athena_iceberg prefix, create another prefix named customer_iceberg.

Create an Iceberg table using Athena

  1. Open the Amazon Athena console.
  2. Choose Query your data in Athena console, then choose Launch query editor.
  3. In Athena, choose Edit settings.
  4. Set s3://<your-bucket-name>/athena_results/ as the Location of query result, then choose Save. Replace <your-bucket-name> with your bucket name.
  5. Choose Editor to return to the query editor page.
  6. To create the database, copy the following query into the query editor and choose Run. You need to be in the Athena Query Editor to run the following commands.
    create database iceberg_db;

  7. To create the Iceberg table, copy the following query into the query editor, replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table, and choose Run.
    CREATE TABLE iceberg_db.churn (
        state string,
        account_length int,
        area_code string,
        phone string,
        intl_plan string,
        vmail_plan string,
        vmail_message int,
        day_mins double,
        day_calls int,
        day_charge double,
        eve_mins double,
        eve_calls int,
        eve_charge double,
        night_mins double,
        night_calls int,
        night_charge double,
        intl_mins double,
        intl_calls int,
        intl_charge double,
        custserv_calls int,
        churn boolean)
    LOCATION 's3://<s3 bucket location>/iceberg/churn'
    TBLPROPERTIES (
        'table_type'='iceberg',
        'compression_level'='3',
        'format'='PARQUET',
        'write_compression'='ZSTD'
    );

  8. Insert some records into the table.
    -- Sample data insert for "iceberg_db"."churn"
    -- Execute this in the Athena console
    INSERT INTO "iceberg_db"."churn" VALUES
    ('KS', 128, '415', '382-4657', 'no', 'yes', 25, 265.1, 110, 45.07, 197.4, 99, 16.78, 244.7, 91, 11.01, 10.0, 3, 2.70, 1, false),
    ('OH', 107, '415', '371-7191', 'no', 'yes', 26, 161.6, 123, 27.47, 195.5, 103, 16.62, 254.4, 103, 11.45, 13.7, 3, 3.70, 1, false),
    ('NJ', 137, '415', '358-1921', 'no', 'no', 0, 243.4, 114, 41.38, 121.2, 110, 10.30, 162.6, 104, 7.32, 12.2, 5, 3.29, 0, false),
    ('OH', 84, '408', '375-9999', 'yes', 'no', 0, 299.4, 71, 50.90, 61.9, 88, 5.26, 196.9, 89, 8.86, 6.6, 7, 1.78, 2, false),
    ('OK', 75, '415', '330-6626', 'yes', 'no', 0, 166.7, 113, 28.34, 148.3, 122, 12.61, 186.9, 121, 8.41, 10.1, 3, 2.73, 3, false),
    ('AL', 118, '510', '391-8027', 'yes', 'no', 0, 223.4, 98, 37.98, 220.6, 101, 18.75, 203.9, 118, 9.18, 6.3, 6, 1.70, 0, false),
    ('MA', 121, '510', '355-9993', 'no', 'yes', 24, 218.2, 88, 37.09, 348.5, 108, 29.62, 212.6, 118, 9.57, 7.5, 7, 2.03, 3, false),
    ('MO', 147, '415', '329-9001', 'yes', 'no', 0, 157.0, 79, 26.69, 103.1, 94, 8.76, 211.8, 96, 9.53, 7.1, 4, 1.92, 0, false),
    ('WV', 141, '415', '330-8173', 'yes', 'yes', 37, 258.6, 84, 43.96, 222.0, 111, 18.87, 326.4, 97, 14.69, 11.2, 5, 3.02, 0, false),
    ('IN', 65, '415', '329-6603', 'no', 'no', 0, 129.1, 137, 21.95, 228.5, 83, 19.42, 208.8, 111, 9.40, 12.7, 6, 3.43, 4, true),
    ('RI', 74, '415', '344-9230', 'no', 'no', 0, 187.7, 127, 31.91, 163.4, 148, 13.89, 196.0, 94, 8.82, 9.1, 5, 2.46, 0, false),
    ('IA', 168, '408', '363-1107', 'no', 'no', 0, 275.8, 90, 46.89, 230.0, 73, 19.55, 191.3, 57, 8.61, 9.9, 3, 2.67, 4, true),
    ('MT', 95, '510', '394-8006', 'no', 'no', 0, 113.2, 96, 19.24, 269.9, 107, 22.94, 229.1, 87, 10.31, 7.1, 4, 1.92, 1, false),
    ('NY', 62, '415', '371-5765', 'no', 'no', 0, 236.5, 127, 40.21, 145.3, 101, 12.35, 225.0, 103, 10.13, 12.0, 1, 3.24, 5, true),
    ('TX', 109, '408', '356-2992', 'no', 'yes', 33, 190.7, 114, 32.42, 218.2, 111, 18.55, 156.5, 122, 7.04, 11.6, 5, 3.13, 1, false),
    ('CA', 155, '510', '328-8230', 'no', 'no', 0, 197.3, 78, 33.54, 160.2, 86, 13.62, 280.1, 90, 12.60, 8.8, 2, 2.38, 2, false),
    ('WA', 132, '415', '382-1011', 'yes', 'no', 0, 302.7, 67, 51.46, 212.0, 105, 18.02, 265.5, 82, 11.95, 10.3, 4, 2.78, 3, true),
    ('FL', 88, '408', '344-5678', 'no', 'yes', 18, 145.3, 95, 24.70, 187.6, 92, 15.95, 198.2, 108, 8.92, 9.4, 3, 2.54, 0, false),
    ('CO', 201, '510', '367-4321', 'no', 'no', 0, 312.5, 142, 53.13, 178.9, 76, 15.21, 145.7, 95, 6.56, 14.2, 8, 3.83, 6, true),
    ('GA', 56, '415', '390-2244', 'no', 'yes', 12, 178.4, 101, 30.33, 205.1, 119, 17.43, 230.8, 100, 10.39, 8.0, 2, 2.16, 1, false);

Register the bucket with Lake Formation in Lake Formation mode

To use Lake Formation permissions for access control to the churn table, you must register the location. To do that, complete the following actions:

  1. Open the AWS Lake Formation console.
  2. In the navigation pane under Administration, choose Data lake locations.
  3. Choose Register location and enter the following information:
    1. For S3 URI, enter s3://<s3 bucket location>/iceberg/churn. Replace <s3 bucket location> with your Amazon S3 bucket location hosting the Iceberg table.
    2. For IAM role, choose the user-defined IAM role that you created in the prerequisites.
    3. For Permission mode, choose Lake Formation.
  4. Choose Register location.

Enable third-party integration in Lake Formation

From the Lake Formation console, enable full table access for external engines.

  1. Open the AWS Lake Formation console.
  2. On the left pane, expand the Administration section.
  3. Choose Application integration settings and select Allow external engines to access data in Amazon S3 locations with full table access.
  4. Choose Save.

Application integration settings page in the Lake Formation console with full table access enabled for external engines

Set up an IAM user for third-party access

  1. Open the IAM console.
  2. From the left navigation menu, choose Policies, then choose Create policy. Choose JSON and paste the following policy:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": "lakeformation:GetDataAccess",
                "Resource": "*"
            }
        ]
    }

  3. Choose next, provide a name for the policy, and choose Create policy.
  4. From the left navigation menu, choose Users, then choose Create user.
  5. For username, enter data_cloud_user, choose next, and choose Attach policies directly.
  6. Choose AWSGlueServiceRole and the policy that you created in step 3. Choose next and Create user.
  7. Choose the user, then choose Security credentials to create an access key.
  8. Scroll down and choose Create access key, choose Applications running outside AWS, and choose Create access key.
  9. Copy the access key and secret access key, and save them securely. You need these to configure the connector in Data 360.

Set up Lake Formation resource permissions for third-party data access

  1. Open the Lake Formation console.
  2. From the left navigation under Data Catalog, choose Databases, then choose the athena_iceberg_db database.
  3. From the Actions menu, choose Permissions, Grant.
  4. In Principals, choose IAM users and roles, and from the menu choose data_cloud_user, which you just created in IAM.
  5. Scroll down to grant permissions by choosing All tables, then choose Select and Describe permissions for the tables.
  6. Choose Grant to apply the permissions.

Set up Apache Iceberg file federation in Data 360

Create and configure the connection

  1. Navigate to Salesforce Setup. For instructions, see Set Up the AWS Glue Data Catalog Connection.
  2. In Data Cloud, choose Setup, then choose Data Cloud Setup.
    Data Cloud Setup page in Salesforce showing options to configure connections
  3. Under External Integrations, choose Other Connectors.
  4. Choose New.
  5. On the Source tab, choose AWS Glue Data Catalog, then choose Next.
    New connector page in Salesforce Data Cloud with AWS Glue Data Catalog selected as the source
  6. Complete the following information shown in the following screen:
    1. In the Authentication Details section, enter the AWS access key ID and AWS secret access key for the IAM user. Make sure that the IAM user has a policy that grants the user read-only access to AWS Glue Data Catalog. Use Lake Formation to configure storage credential vending. This approach is for AWS Glue Data Catalog to vend temporary credentials at run time so that Data 360 can access the underlying storage bucket.
    2. For Catalog URL, enter the URL of AWS Glue Data Catalog. See Connecting to the Data Catalog by using AWS Glue Iceberg REST endpoint.
    3. For Catalog ID, enter the 12-digit AWS account ID linked to AWS Glue Data Catalog.
    4. For Signing Region, enter the host AWS Region where AWS Glue Data Catalog is located.
    5. For Signing Service, enter glue. Data 360 requires the Signing Service, in addition to the AWS access key ID, secret access key, and Signing Region, to sign requests to AWS Glue Data Catalog by using AWS Signature Version 4.
    6. Test the connection and check for the success message.
    7. Save the connection details.

    AWS Glue Data Catalog connection configuration form in Salesforce Data Cloud showing authentication details, catalog URL, catalog ID, signing Region, and signing service fields

  7. After the configuration is complete and saved, the new AWS Glue Data Catalog connection shows up with “Active” status in the Connectors screen.Connectors screen in Salesforce Data Cloud showing the new AWS Glue Data Catalog connection with Active status

Create and configure the data stream

  1. In Data Cloud, on the Data Streams tab, choose New.
  2. Under Other Sources, choose the AWS Glue Data Catalog source, then choose Next.
  3. From the menus, choose the connection that you just set up, choose a database in your AWS Glue catalog where you have an Iceberg table, choose the table that you want to stream, and choose Next.Data stream configuration page showing AWS Glue Data Catalog source with connection, database, and Iceberg table selection menus
  4. Enter the object name and object API name. For more information, see Data Lake Object Naming Standards.
  5. Choose the category to specify the type of data to ingest. For more information, see Category.
  6. Choose a primary key to uniquely identify the incoming records. For more information, see Primary Key.
  7. Choose the source fields you want to ingest, then choose Next. Fields with convertible data types are listed under Supported Fields.Source fields selection page showing supported fields for the Iceberg table data stream
  8. Choose the relevant data space. Choose “Default” if you don’t have any other data space provisioned in your org. For more information, see Data Spaces.
  9. Choose Deploy.
    Data stream deployment confirmation page in Salesforce Data Cloud
  10. After the setup is complete, the new data stream appears in your Data Cloud environment.
    Data Cloud environment showing the newly deployed data stream for the Iceberg table
  11. The data stream is ready. You can now go to the Data Explorer in your Data Cloud environment and start viewing the Iceberg tables that reside in your external AWS account.Data Explorer in Salesforce Data Cloud showing Iceberg tables from the external AWS account

Best practices and considerations

  • Use IAM roles with least-privilege access. Grant only the specific permissions each service or user needs.
  • Implement appropriate Amazon S3 bucket policies. Define bucket-level policies that restrict access by AWS account, VPC endpoint, or IP range.
  • Monitor access patterns. Enable Amazon S3 server access logging or AWS CloudTrail data events to track who reads from and writes to your table buckets.
  • Optimize Iceberg table partitioning. Choose partition keys that align with your most common query filters.
  • Consider data access patterns. Design your table layout around how data is actually queried.
  • Implement lifecycle policies for Amazon S3 objects. Configure Amazon S3 lifecycle rules to transition older data files to other storage classes.
  • Use appropriate Iceberg file compaction strategies. Run compaction regularly to merge small files produced by streaming or frequent batch appends.
  • Monitor data transfer costs. Track cross-Region and internet egress charges using AWS Cost Explorer as applicable.

Clean up

After you finish testing, clean up all the resources in your AWS account that you created (including the Amazon S3 bucket, Athena tables, and other AWS services) to avoid recurring costs.

Conclusion

By implementing Apache Iceberg file federation between Data 360 and Amazon S3, you can create a more efficient and streamlined data architecture. This solution gives you real-time access to Amazon S3 data while using the analytics capabilities of Data 360. As businesses continue to prioritize data-driven decision-making, Zero Copy data sharing plays an important role in unlocking the full potential of customer data across platforms.

To learn more, review the following resources:


About the authors

Avijit Goswami

Avijit Goswami

Avijit is a principal specialist solutions architect at AWS specializing in data and analytics. He helps customers design and implement robust data lake solutions. Outside the office, you can find Avijit exploring new trails, discovering new destinations, cheering on his favorite teams, enjoying music, or testing out new recipes in the kitchen.

Srividya Parthasarathy

Srividya Parthasarathy

Srividya is a Senior Big Data Architect on the AWS Lake Formation team. She works with the product team and customers to build robust features and solutions for their analytical data platform. She enjoys building data mesh solutions and sharing them with the community.

Pratik Das

Pratik Das

Pratik is a Senior Product Manager with AWS Lake Formation. He is passionate about all things data and works with customers to understand their requirements and build delightful experiences. He has a background in building data-driven solutions and machine learning systems

Bill Tarr

Bill Tarr

From software builder to architecture, Bill has 20+ years of experience shaping best-in-class SaaS technology strategies for organizations from startup to enterprise. He’s also an AWS SaaS community leader and a producer of the “Building SaaS on AWS” show on twitch.com/aws, as well as an experienced public speaker with experience at top tier AWS events such as re:Invent, and publisher of SaaS best practices.

ICYMI: June 2026 @AWS Security

Post Syndicated from Rodolfo Brenes original https://aws.amazon.com/blogs/security/icymi-june-2026-aws-security/

Read all about the latest AWS security features, compliance updates, and hands-on resources in our new, monthly digest posts. You’ll find expert blog posts, new service capabilities, code samples, and workshops.

AWS Security Blog posts

This month’s AWS Security Blog posts covered identity and access management, threat intelligence, network security, AI-powered security tooling, and multi-account governance. Read on for guidance on restricting console access to expected networks, securing multi-tenant AI agents, preventing data exfiltration, and managing organization-scale migrations.

Identity

Secure multi-tenant AI agents with Amazon Bedrock AgentCore resource-based policies
Authors: Satyen Verma, Satveer Khurpa, Prajit Pabbati, Vijay Kumar Samanthapudi, Zohreh Norouzi | Published: June 2, 2026
Learn to use resource-based policies on Amazon Bedrock AgentCore to grant cross-account access for one tenant while restricting another to VPC-only traffic in a shared multi-tenant AI platform.

Customize federated sign-in with new Amazon Cognito Lambda trigger
Authors: Abrom Douglas | Published: June 4, 2026
Learn to use the new inbound federation AWS Lambda trigger for Amazon Cognito to transform, filter, and enrich user attributes from external identity providers before profile creation in your user pool.

Amazon Cognito unlocks advanced capabilities with next-generation infrastructure
Authors: Howie Li, Georgi Baghdasaryan | Published: June 4, 2026
Amazon Cognito introduced high-throughput performance, customer-managed keys for data encryption at rest, and multi-Region replication for business continuity, built on a new storage infrastructure migrated with zero downtime.

Building secure B2C applications with fine-grained access control using Amazon Cognito and Amazon Verified Permissions
Authors: Sowmya Vemuri | Published: June 5, 2026
Learn to build fine-grained access controls for a Streamlit application using Amazon Cognito for authentication and Amazon Verified Permissions with Cedar policies for authorization.

Restrict AWS Management Console access to expected networks with sign-in resource-based policies and RCPs
Authors: Swara Gandhi, Rishi Tripathy | Published: June 24, 2026
Learn to use sign-in resource-based policies and resource control policies to restrict AWS Management Console sign-in to requests from expected networks, such as corporate VPNs and Amazon VPC endpoints.

Infrastructure security

Gain visibility into DDoS attacks with flow logs in AWS Shield Advanced
Authors: Ken Kitts | Published: June 4, 2026
Learn to configure AWS Shield Advanced attack flow logs to capture traffic metadata during DDoS events, pinpoint sources, verify mitigations, and feed your existing analysis pipelines.

Threat tactic spotlight: Subdomain takeover
Authors: Matt Gurr, Ariam Michael, Geoff Sweet, Luis Pastor | Published: June 16, 2026
Learn to detect and prevent subdomain takeover using AWS Config custom rules to identify dangling DNS CNAME records pointing to deleted resources in globally shared namespaces

Prevent data exfiltration: AWS egress controls for cloud workloads
Authors: Meriem Smache, Maxim Raya | Published: June 22, 2026
Learn to implement layered egress detection and protection using AWS Network Firewall, Amazon Route 53 Resolver DNS Firewall, and data perimeters to help reduce unauthorized data transfer risk.

Detection and incident response

Operationalizing AWS security: A maturity roadmap
Authors: Joseph Sadler | Published: June 8, 2026
A six-phase maturity roadmap for organizations that have already enabled AWS Security Hub and Amazon GuardDuty, covering tuning, notifications, automated remediation, and operational cadence.

Introducing AWS Continuum: Security at machine speed
Authors: Chet Kapoor | Published: June 17, 2026
AWS Continuum for code vulnerabilities is an AI-native platform that addresses the full lifecycle of a code vulnerability at machine speed—from discovery and prioritization through validation and remediation.

Accelerate security investigations with Kiro CLI
Authors: Sibasankar Behera, Marshall Jones | Published: June 18, 2026
Learn to use Kiro CLI to conduct security investigations following the AWS Security Incident Response Guide framework, from triaging Amazon GuardDuty findings through containment and evidence preservation.

What the June 2026 Threat Technique Catalog update means for your AWS environment
Authors: Shannon Brazil, Cydney Stude, Javier Teitelbaum | Published: June 29, 2026
Learn about five new entries and three updates to the Threat Technique Catalog for AWS, covering container security, organization-level trust, and compute hijacking patterns observed by AWS CIRT.

Data protection

Identify unused AWS KMS keys and prevent accidental key deletions
Authors: Andrea Rossi, Poojil Tripathi | Published: June 2, 2026
Learn to use the new AWS KMS GetKeyLastUsage API to audit key activity, identify unused keys, and apply policy controls that prevent accidental deletion of recently used keys.

Governance and compliance

From Monolith to Multi-Account: Pinterest’s AWS Organization Transformation Journey
Authors: Sid Vantair, James Fogel, Jeremy Talis | Published: June 4, 2026
Learn how Pinterest migrated from a single monolithic AWS account to a multi-account architecture, including management account separation, automated account provisioning, and centralized networking.

Build a Multi Account Patch Compliance Dashboard with Kiro Specs
Authors: Justin Thomas | Published: June 9, 2026
Learn to use Kiro’s spec-driven development approach to build a serverless multi-account patch compliance dashboard with AWS Systems Manager Patch Manager and private access via AWS Systems Manager Session Manager.

Transfer AWS accounts between AWS Organizations while preserving AWS Lake Formation permissions
Authors: Alex Torres, Aarthi Srinivasan, Ryan McNamee | Published: June 12, 2026
Learn to migrate member accounts between AWS Organizations without disrupting AWS Lake Formation cross-account permissions by using temporary bridge shares with the new AWS RAM retention parameter.

June Security Bulletins

Investigations of reported security vulnerabilities affecting Amazon and AWS services, software, and products.

AWS Samples

This month brings 10 new AWS samples spanning AI security, identity, infrastructure security, governance, and observability. From securing Amazon Bedrock AgentCore agents with AWS WAF to building graph-based CMDBs for dependency analysis, these repositories help you implement security and governance best practices across your AWS environment.

AI Security

Securing Amazon Bedrock AgentCore Runtime with AWS WAF
Learn to place AWS WAF in front of Amazon Bedrock AgentCore Runtime using two architecture patterns with an Application Load Balancer and VPC endpoints for defense-in-depth protection.

AI Security Posture Management (AI SPM) on AWS
Learn to discover, assess, and protect AI agents running in your environment using AWS-native services across three pillars: observe, govern, and defend, with rules mapped to OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS.

Identity

Implement On-Behalf-Of (OBO) token exchange for Amazon Bedrock AgentCore agents
Learn to implement secure identity propagation from user context through agent chains using Amazon Bedrock AgentCore Identity OBO token exchange without credential sharing.

Infrastructure Security

Monetizing AI traffic with Amazon CloudFront and AWS WAF (x402)
Learn to monetize AI traffic at the edge using AWS WAF native x402 protocol support with Amazon CloudFront, settling payments in a single round-trip with no Lambda@Edge required.

Governance

Turn repeatable processes into SOPs your AI agent works through one auditable step at a time
Learn to use this MCP server to hand standard operating procedures to your AI agent one step at a time, with gated execution and auditable progress tracking.

Bedrock Ops Lens
Learn to deploy an Amazon Bedrock observability dashboard in your own account for per-account, per-model, and per-tag cost attribution, quota tracking, latency monitoring, and model lifecycle management, with an MCP server for IDE access.

AWS Agent Registry sample demo application
Learn to use AWS Agent Registry to publish, review, approve, and discover AI agents, MCP servers, and agent skills through a centralized catalog with governance workflows.

Graph CMDB — an AWS resource relationship graph
Learn to build a graph-based CMDB from AWS Config, AWS IAM, Amazon EKS, AWS CloudTrail, and VPC Flow Logs to visualize resource dependencies, investigate scope of impact, and surface governance findings.

OpenAI Codex through Amazon Bedrock — Usage governance with LiteLLM
Learn to centrally govern, administer, and monitor OpenAI Codex access for engineering teams using Amazon Bedrock as the inference backend with per-user budgets, rate limits, and audit trails via LiteLLM.

Agentic Data Governance — the context ladder
Learn to measure how governance context layers — data dictionaries, semantic metrics, and execution skills — improve data agent accuracy through a four-level ablation ladder on the BIRD benchmark.

Conclusion

June 2026 provides guidance and examples for operationalizing security at organizational scale; from maturity roadmaps and console access restrictions to AI agent registries and posture management platforms. The posts and samples provide patterns for DDoS visibility with flow logs, multi-tenant agent isolation with resource-based policies, egress controls for data exfiltration prevention, and governance frameworks for AI coding assistants. Each resource includes deployment steps or runnable code so you can validate in your own environment before adopting. Subscribe to the AWS Security Blog RSS feed to receive updates as they publish, and revisit this digest monthly for a consolidated view of what changed and what to act on.

If you have feedback about this post, submit comments in the Comments section below.


Rodolfo Brenes

Rodolfo Brenes

Rodolfo is a Principal Solutions Architect focused on Cloud Governance and Compliance. With over 18 years of experience, he currently leads a technical field community in AWS helping customers scale and improve their security and governance frameworks. Besides work, Rodolfo enjoys video games, playing with his four cats, and won’t say no to a good outdoor adventure.

Anna Brinkmann

Anna has 18 years of experience in the technical content space and has spent the last 6 years managing the AWS Security Blog. Outside of work, she enjoys spending time with her family.

Amazon SQS turns 20: Two decades of reliable messaging at scale

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/amazon-sqs-turns-20-two-decades-of-reliable-messaging-at-scale/

On July 13, 2006, we launched Amazon Simple Queue Service (Amazon SQS) as one of the first three services available to customers, alongside Amazon EC2 and Amazon S3. We had learned firsthand that distributed systems need a reliable way to pass messages between components without creating tight dependencies. If one service called another directly and that service was slow or unavailable, failures cascaded through the entire system. Message queuing solved this by letting services communicate asynchronously: a producer could drop a message into a queue and move on, while a consumer picked it up when ready. This approach kept individual service failures from affecting the rest of the system.

When Amazon SQS launched publicly in July 2006, it made this pattern available to every AWS customer. Twenty years later, that core function, decoupling producers from consumers, remains the reason customers use SQS. The scale, performance, and operational controls around it look very different now though.

Jeff Barr covered the first 15 years of SQS milestones in his 15th anniversary post, from the original 8 KB message limit in 2006 through FIFO queues, server-side encryption, and Lambda integration. Over the last five years, we have continued to scale SQS, added stronger security defaults, and introduced new capabilities that address increasingly complex workload patterns.

Key milestones between 2021 and 2026
High throughput mode for FIFO queues (2021): In May 2021, we launched general availability of high throughput mode for FIFO queues, supporting up to 3,000 transactions per second (TPS) per API action, a tenfold increase over the previous limit. We continued raising this ceiling over the following two years: to 6,000 TPS in October 2022, to 9,000 TPS in August 2023, and to 18,000 TPS in October 2023, before reaching 70,000 TPS per API action in select Regions by November 2023.

Server-side encryption with SSE-SQS (2021): In November 2021, we introduced server-side encryption with Amazon SQS-managed encryption keys (SSE-SQS), giving customers an encryption option that required no key management. In October 2022, we made SSE-SQS the default for all newly created queues, so customers no longer needed to explicitly enable it.

Dead-letter queue redrive enhancements (2021): We progressively expanded how customers recover unconsumed messages from dead-letter queues. In December 2021, we added DLQ redrive to source queue directly in the SQS console. In June 2023, we extended this capability to the AWS SDK and CLI through new APIs, including StartMessageMoveTask, CancelMessageMoveTask, and ListMessageMoveTasks. In November 2023, we added redrive support for FIFO queues.

Attribute-based access control, ABAC (2022): In November 2022, we introduced ABAC, giving customers the ability to configure access permissions based on queue tags rather than maintaining static policies as resources scaled.

JSON protocol support (2023): In November 2023, we added support for the JSON protocol in the AWS SDK, reducing end-to-end message processing latency by up to 23% for a 5 KB payload and lowering client-side CPU and memory usage.

Amazon EventBridge Pipes console integration (2023): We added the ability to connect a queue directly to EventBridge Pipes from the SQS console, routing messages to a broad range of AWS service targets without writing custom integration code.

Extended Client Library for Python (2024): We brought the Extended Client Library, previously available for Java, to Python developers, allowing messages up to 2 GB to be sent through SQS by storing the payload in Amazon S3 and passing a reference through the queue.

FIFO in-flight message limit increase (2024): We increased the in-flight message limit for FIFO queues from 20,000 to 120,000 messages, so consumers can process significantly more messages concurrently without being constrained by the previous ceiling.

Fair queues for multi-tenant workloads (2025): We introduced fair queues to mitigate the noisy neighbor problem in multi-tenant standard queues. By including a message group ID when sending messages, customers can prevent a single tenant from delaying message delivery for others, without any changes required on the consumer side.

1 MiB maximum message payload size (2025): We increased the maximum message payload from 256 KiB to 1 MiB for both standard and FIFO queues, helping customers send larger messages without offloading data to external storage. AWS Lambda event source mapping for SQS was updated in parallel to support the new payload size.

The constant underneath the change
Despite two decades of feature additions, the fundamental use case for SQS has not shifted. Customers use it to decouple services, buffer bursts of traffic, and build systems that stay resilient when individual components fail. That same pattern now extends to AI workloads. Customers use SQS queues to buffer requests to large language models, manage inference throughput, and coordinate communication between autonomous AI agents operating as independent services. For an example of this architecture in practice, read Creating asynchronous AI agents with Amazon Bedrock.

To learn more about Amazon SQS, visit the Amazon SQS product page, review the developer guide, or explore recent updates on the AWS Blogs.

— Esra

AWS Weekly Roundup: AWS Builder Center at 1 year, Network Scanning in Security Hub, Loom for AWS, and more (July 13, 2026)

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-aws-builder-center-at-one-year-network-scanning-in-security-hub-loom-for-aws-and-more-july-13-2026/

AWS Builder Center turned one year old last week. Launched on July 9, 2025, the platform has grown from a community hub with Wishlist voting, community profiles, and a toolbox into a full ecosystem with sandbox environments, workshops, Spaces, and a Builders’ Library. To mark the anniversary, Rick Suttles published a full feature timeline covering everything shipped over the past year: AWS Capabilities by Region (1,500+ services across 37 Regions), Spaces for community-created groups, workshops with category and complexity filters, badges and streaks, article series, view counts, saved items, student status, availability notifications, sign-in with GitHub and Amazon, and sandbox environments.

Jeff Barr published a retrospective summarizing Builder Center’s first year. Since launch, 5,548 authors have published 6,448 articles with more than 10.4 million page views combined. Builders have earned 99,226 badges since the badge system launched in March 2026. Community members have submitted 565 wishes, 10 of which have shipped with another 20 on the near-term roadmap.

The top community article Building an AWS Study Buddy with MCP + Strands Agents SDK by Dineshraj Dhanapathy reached 50,000+ views. Chris Miller’s Migrating an EOL Linux Server to AWS in 8 Hours with Kiro followed at 45,000+, and Yash Aggarwal’s AIdeas: NeuroVoice – Multimodal AI for Early Screening of Neurological Diseases article reached 38,000+.

The week’s headline addition is Sandbox Environments by Rick Suttles. Sandboxes give you a free, pre-provisioned AWS account to complete a workshop exercise. Each environment is active for 8 hours, after which the account and all its resources are automatically de-provisioned. You can have one active sandbox at a time and request one per week. No personal AWS account, credit card, or manual cleanup required.

Last week’s launches
Here’s what else happened this week.

  • AWS Security Hub introduces Network Scanning – Security Hub introduced Network Scanning, a capability that identifies resources in your environment that are reachable from the public internet. Network Scanning probes your resources from the internet to detect actual reachability, complementing the existing network reachability findings in Security Hub that identify configurations that could make a resource reachable. It discovers public IP addresses, virtual machines, and load balancers across your AWS and Azure environments, identifies reachable ports, and determines what services are running behind them. Each reachable port generates a Security Hub finding with evidence of the port and service discovered. Security Hub Exposures then automatically correlates these findings with other findings and resource configurations to determine broader risk. Existing customers can enable Network Scanning in individual accounts and Regions, or across an organization through a configuration policy. For new customers, Network Scanning is on by default. It is included with Security Hub Essentials at no additional cost.
  • Security Hub also extends unified security management to Microsoft Azure – Security Hub now monitors Microsoft Azure resources, providing unified posture management, vulnerability management, and security response across both clouds. It automatically discovers Azure VMs, container images, Function Apps, and identities, and evaluates them for misconfigurations, internet exposure, and software vulnerabilities. AWS and Azure findings appear in the same prioritized view with the same formats and automation workflows.
  • Amazon SageMaker Studio integrates with Hugging Face for one-click model deployment and customization – You can now go from discovering a model on Hugging Face to working with it in SageMaker Studio in a single click. Select any supported model on Hugging Face and choose “Customize on SageMaker AI” or “Deploy on SageMaker AI” to land directly on the corresponding workflow page with the model pre-loaded. New customers receive a Studio environment created in seconds with pre-configured permissions for serverless model customization (including fine-tuning with custom reward functions for reinforcement learning), model evaluation, and deployment to SageMaker or Bedrock endpoints. Verified customers receive default GPU access to G5, G6, and G4dn instances without requesting quota increases, and quota utilization is visible directly inside the Studio environment.
  • Amazon EKS Auto Mode and Amazon ECS Managed Instances reduce GPU management fees by up to 60% – Beginning July 1, 2026, EKS Auto Mode and ECS Managed Instances reduce management fees for accelerated instance types: G-series fees are down 35%, and P-series and AWS Trainium fees are down 60%. The reductions apply automatically to existing clusters and require no action from customers. Both services include capabilities built for accelerated workloads. EKS Auto Mode provides automatic parallel image pulling on GPU instances with local NVMe storage and accelerator-aware node repair. ECS Managed Instances provides GPU metrics through Amazon CloudWatch Container Insights and automatic health monitoring for GPU hardware failures.
  • Amazon Aurora DSQL change data capture (CDC) is now generally available – Aurora DSQL CDC streams the results of insert, update, and delete operations as change events to Amazon Kinesis Data Streams. You can use it to synchronize data across microservices, trigger Lambda functions, or deliver changes to S3, Redshift, and OpenSearch Service through Amazon Data Firehose. CDC streaming is designed to have zero impact on database workload performance and requires no infrastructure to manage.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts you may find useful:

  • Building secure AI agents at scale: Introducing Loom for AWS – Loom is an open-source enterprise platform for building agents with AWS Strands Agents and deploying them on Amazon Bedrock AgentCore Runtime. It provides a unified management UI and backend API with identity provider integration, scope-based authorization, multi-persona navigation, and full lifecycle management for agents, memory, MCP servers, and agent-to-agent integrations. Loom enforces automated resource tagging for cost attribution, implements RBAC and ABAC for multi-tenant security, uses paved-path blueprints for agent deployments, manages identity propagation through delegated actor chains, integrates with AWS Agent Registry for discovery and governance, and supports human-in-the-loop review before sensitive actions. The project is available in AWS Labs on GitHub.
  • Introducing Claude apps gateway for AWS – The Claude apps gateway is a self-hosted control plane that gives organizations centralized control over access, cost, and policy for Claude Code and Claude Desktop. It connects to any OIDC-compliant identity provider, enforces managed settings on every request, routes inference to Amazon Bedrock or Claude Platform on AWS, and supports per-user and per-group spend caps. The gateway runs as a stateless container in your private network, backed by a PostgreSQL database for short-lived sign-in state. No long-lived secrets are stored on developer machines. Deploy it through Amazon Bedrock to keep data within the AWS security boundary, or through Claude Platform on AWS for the native Claude platform experience.
  • Introducing OAuth support for AWS MCP Server – You can now connect agents to the AWS MCP Server using browser-based OAuth with the same credentials you use for the AWS Console or CLI. The new sign-in path supports IAM federation, AWS IAM Identity Center, and root or IAM users. AWS Sign-In issues short-lived access tokens and refresh tokens, with automatic token management so developers stay authenticated across restarts. For headless use cases, a non-interactive flow lets applications with existing AWS credentials obtain OAuth access tokens through the create-oauth2-token-with-iam API. New governance controls include OAuth-specific IAM condition keys, token introspection and revocation, dynamic client registration, and CloudTrail audit elements.

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

Visit the AWS Builder Center to meet other builders, contribute solutions, and find resources that help you keep building.

Wishing everyone a restful and enjoyable summer. Whether you’re building, learning, or recharging, I hope you find time for all three. I’ll be heading to Scandinavia for a few weeks to trade the heat for some cooler weather and longer evenings. Come back next week for more news!

— Esra

New compliance guidance available: HITRUST i1 on AWS

Post Syndicated from Abdul Javid original https://aws.amazon.com/blogs/security/new-compliance-guidance-available-hitrust-i1-on-aws/

We are pleased to announce the publication of a new AWS compliance implementation guidance: HITRUST i1 Compliance on AWS: Customer Implementation Guidance with an Illustrative Healthcare Platform.

Healthcare organizations seeking HITRUST i1 certification increasingly rely on Amazon Web Services (AWS) as their cloud foundation. The HITRUST i1 assessment covers 182 curated controls at the Implemented level and is the most widely required HITRUST certification tier in healthcare vendor contracts and Business Associate Agreements required by health plans, hospital systems, and business associates as a condition of working with them.

This guide is designed to close the gap between understanding what HITRUST i1 requires and knowing how to implement it on AWS. It walks cloud architects, security engineers, compliance leads, and assessment preparation teams through the full lifecycle of an i1 engagement from defining the assessment boundary to implementing controls across each technical domain.

What the guide covers

The guide addresses 11 HITRUST i1 technical control domains, with supporting AWS implementation components relative to these domains. The domains include access control, endpoint protection, configuration management, vulnerability management, network protection, transmission protection, incident management, data protection and privacy, audit logging and monitoring, password management, and business continuity and disaster recovery.

The guidance is grounded in a fictional but realistic connected healthcare platform deployed on AWS Landing Zone Accelerator. The scenario is used to make abstract HITRUST concepts concrete, not to suggest that the same architecture or control choices apply universally. HITRUST i1 scoping is inherently organization-specific. The assessment boundary, applicable controls, and evidence requirements are determined by each organization’s system scope and delivered through the HITRUST MyCSF portal. Readers should treat the guidance as a starting point and work with a HITRUST Authorized External Assessor to validate what applies to their specific environment. This guide doesn’t constitute a compliance certification advisory.

Getting started

You can download the guide here: HITRUST i1 Compliance on AWS: A Customer Implementation Guidance with an Illustrative Healthcare Platform.

AWS HITRUST assurance documentation and the Customer Responsibility Matrix are available through AWS Artifact. For assessment readiness support, visit AWS Security Assurance Services.

If you have feedback about this post, submit comments in the Comments section below.


Abdul Javid

Abdul is a Senior Security Assurance Consultant at AWS Security Assurance Services. He holds HITRUST certifications and has led HITRUST r2 and i1 engagements across multiple healthcare technology companies. Abdul holds multiple security and auditing certifications and supports customers building responsible AI governance programs on AWS. He has over 25 years of experience and holds certifications across AWS, CMMC, PCI DSS, PMI, ISC2, and ISACA.

Shreya Singh

Shreya Singh

Shreya is a Security Assurance Consultant at AWS with more than eight years of experience in governance, risk, compliance, and cloud security. She holds the CISA and HITRUST Certified CSF Practitioner (CCSFP) certifications and supports healthcare and technology organizations with HITRUST, HIPAA, SOC 2, risk management, and audit readiness initiatives. She holds a Master of Engineering in Cybersecurity from the University of Maryland, College Park.

AWS designated as a critical third party to the UK financial sector

Post Syndicated from Michael Jefferson original https://aws.amazon.com/blogs/security/aws-designated-as-a-critical-third-party-to-the-uk-financial-sector/

Amazon Web Services EMEA Sarl (AWS) has been designated as a critical third party (CTP) to the UK financial sector by HM Treasury.

The CTP regime came into force on January 1, 2025, and establishes a framework through which the Bank of England, PRA, and FCA (collectively the UK regulators) can set requirements on and have direct oversight of designated CTPs.

AWS supports the objectives of the UK regulators to ensure a robust financial system.

AWS obligations under the regime

The CTP regime is an outcomes-focused framework. Under the regime, AWS will be subject to requirements in relation to its designated Systemic Third-Party Services (STPS). The first step will be a self-assessment of these STPSs against the CTP regime criteria, which AWS will now carry out in line with regulators’ expected timelines.

AWS has actively engaged with the UK authorities as they’ve developed the CTP regime, and we will continue this constructive approach as we work to meet our obligations under it.

Impact on customers

The UK regulators have clarified that the requirements under the regime don’t eliminate, reduce, or replace the accountability of firms, their boards, and senior management for remaining operationally resilient, including when they rely on services provided by third parties.

The regime doesn’t change obligations on financial services customers of designated CTPs. As we manage our obligations, we expect to publish materials that customers can use to inform their own operational resilience planning and third-party risk management.

The AWS commitment to operational resilience

We’re focused on supporting financial services customers in enhancing their operational resilience and providing a range of services and guidance—including the AWS Well-Architected Framework and resources for cloud incident management—to help organizations deliver effective resilience outcomes.

AWS has a team of regulatory and technology experts with expertise in financial services ready to support customers with questions about this regime or operational resilience more broadly. Customers can contact their AWS account team for further information.


Michael Jefferson

Michael Jefferson

Michael is Head of Financial Services Public Policy for EMEA. He leads on policy and engagement for issues relating to adoption and use of cloud and technology across the finance sector. Before joining AWS, he led on capital markets policy at the Investment Association and prior to that at UK Finance. He previously was head of Public Policy EMEA at Nomura and spent the early part of his career as a UK civil servant working on international trade and business issues.

Introducing Apache Spark Connect support in AWS Glue interactive sessions

Post Syndicated from Zach Mitchell original https://aws.amazon.com/blogs/big-data/introducing-apache-spark-connect-support-in-aws-glue-interactive-sessions/

When we built AWS Glue interactive sessions, our goal was to make AWS Glue as interactive as running local Python from a notebook. We mostly succeeded. With a straightforward Python package and a Jupyter notebook, you could execute remotely against the AWS Glue ephemeral Spark backend. The Livy-based approach was ahead of its time, but it had limitations from its REST-based protocol. Running local PySpark unlocked powerful integrated development environment (IDE) features such as debugging and linting, so your environment could understand the code and help you develop Spark applications more quickly. Customers would often split their development work. They used local Spark (or Docker containers) to develop in an IDE on a small amount of data, then switched to AWS Glue interactive sessions to validate scaling and tuning against the full dataset.

With modern PySpark releases came a new protocol: Apache Spark Connect. Spark Connect bridges the gap between these two worlds: you develop in local Python, but execute on AWS Glue against actual data. Today, AWS Glue interactive sessions support Spark Connect natively. You can connect from any environment that supports the PySpark remote() API, including VS Code, PyCharm, Amazon SageMaker Unified Studio notebooks, and standalone Python applications. You don’t need to install specialized kernels or manage cluster infrastructure.

What Spark Connect changes

Spark Connect, introduced in Spark 3.4, decouples the Spark client from the server through a lightweight gRPC protocol. Instead of running your driver program on the cluster, your IDE communicates with a remote Spark server through a thin client layer. This architecture unlocks the key workflow improvement: you develop locally and execute remotely.

Spark Connect architecture diagram showing a thin client communicating with a remote Apache Spark server

Spark Connect architecture — thin client with the full power of Apache Spark

With Spark Connect support in AWS Glue interactive sessions, you get:

  • IDE freedom – Use VS Code, PyCharm, JupyterLab, or any Python environment. No kernel installation required.
  • Programmatic access – Build Spark into your Python applications and automation scripts with a standard SparkSession.builder.remote() call.
  • Serverless execution – AWS Glue provisions and manages the Spark cluster. You pay only for the data processing units (DPUs) consumed while your session is active.
  • Spark Connect monitoring – The Spark Live UI now includes a dedicated Connect tab showing active Spark Connect sessions and operations alongside the existing Jobs, Stages, and Executors views.

Getting started with SageMaker Unified Studio

Amazon SageMaker Unified Studio provides the most direct path to Spark Connect on AWS Glue. The notebook environment handles session creation, endpoint retrieval, and token refresh automatically, so no connection boilerplate is required.

Prerequisite: You need an Amazon SageMaker Unified Studio project to use this workflow. If you don’t have one, create a project in your SageMaker Unified Studio domain first.

To connect to an AWS Glue Spark Connect session:

  1. Sign in to SageMaker Unified Studio, choose your project, and create or open a Notebook.

A notebook open in SageMaker Unified Studio

A notebook open in SageMaker Unified Studio

  1. Choose the compute icon in the left toolbar to open the Compute environment panel. Expand the Spark section.

Compute environment panel in SageMaker Unified Studio with the Spark section expanded

The Compute environment panel with the Spark dropdown list

  1. Select a Glue Spark connection. Depending on your SageMaker domain configuration, you will see either default.spark or named connections such as project.spark.compatibility. Select the appropriate Glue (Spark) connection and choose Apply.

Notebook cell showing spark.version returns 3.5.6-amzn-1 after connecting to Glue Spark Connect

Connected to Glue Spark Connect — running spark.version returns ‘3.5.6-amzn-1’

After you make your selection, you’re connected. The spark session object is available natively. No imports or configuration are needed. Start running PySpark immediately:

spark.sql("SHOW DATABASES").show()

The session manages itself in the background, including automatic token refresh.

Using the sagemaker_studio SDK

The sagemaker-studio Python package extends the Spark Connect experience beyond SageMaker Unified Studio notebooks into local IDEs, continuous integration and continuous delivery (CI/CD) pipelines, and any Python environment. The sparkutils module handles session initialization and connection configuration in a single call. You get the same streamlined experience as in the notebook, anywhere you run Python:

from sagemaker_studio import sparkutils

# Initialize a Glue Spark Connect session using your project connection
spark = sparkutils.init(connection_name="default.spark")

# Run queries immediately
spark.sql("SHOW DATABASES").show()

You can also use sparkutils.get_spark_options() to retrieve pre-configured Java Database Connectivity (JDBC) options for reading and writing to data sources through your project connections. Supported sources include Amazon Redshift, Amazon Aurora, and Amazon DocumentDB (with MongoDB compatibility):

# Get connection options for a Redshift connection in your project
options = sparkutils.get_spark_options("my_redshift_connection")

# Read from Redshift via Spark Connect
df = spark.read.format("jdbc").options(**options).option("dbtable", "analytics.orders").load()
df.show()

Within SageMaker Unified Studio, the sagemaker-studio SDK is native to the environment. The spark session and sparkutils are available without installation. For local IDE use, install it with pip install sagemaker-studio and configure credentials through an AWS named profile or boto3 session.

How it works

Spark Connect sessions in AWS Glue use a three-step workflow:

  1. Create a session – Call the CreateSession API with SessionType set to SPARK_CONNECT. The session provisions in approximately 30 seconds.
  2. Retrieve the endpoint – Call GetSessionEndpoint to receive a sc:// gRPC endpoint URL and a time-limited authentication token.
  3. Connect with PySpark – Pass the endpoint and token to SparkSession.builder.remote() and start running Spark operations.

Spark Connect protocol flow from the DataFrame API to a logical plan, sent over gRPC and protobuf, with results streamed back over gRPC and Arrow

Spark Connect protocol flow — DataFrame API translated to logical plan, sent via gRPC/protobuf, results streamed back via gRPC/Arrow

Connecting with the low-level API

Some environments don’t have the sagemaker-studio SDK, such as custom containers, AWS Lambda functions, or non-Python toolchains. In these environments, or if you’re not using SageMaker Unified Studio, you can use the AWS SDK (Boto3) to manage sessions directly. The following example demonstrates the full workflow:

import time, boto3, urllib.parse
from pyspark.sql import SparkSession

glue = boto3.client("glue", region_name="us-east-1")

# 1. Create a Spark Connect session
session_id = "my-spark-connect-session"
glue.create_session(
    Id=session_id,
    Role="arn:aws:iam::123456789012:role/GlueServiceRole",
    Command={"Name": "glueetl"},
    GlueVersion="5.1",
    SessionType="SPARK_CONNECT",
    DefaultArguments={"--enable-spark-live-ui": "true"},
)

# 2. Wait for the session to reach READY
while True:
    status = glue.get_session(Id=session_id)["Session"]["Status"]
    if status == "READY":
        break
    time.sleep(5)

# 3. Get the Spark Connect endpoint
sc = glue.get_session_endpoint(SessionId=session_id)["SparkConnect"]
endpoint_url = sc["Url"]
auth_token = sc["AuthToken"]

# 4. Connect with PySpark
encoded_token = urllib.parse.quote(auth_token, safe="")
connection_string = f"{endpoint_url}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
spark = SparkSession.builder.remote(connection_string).getOrCreate()
spark.sql("SELECT 1 + 1 AS result").show()

Monitoring with Spark Live UI

When you enable the Spark Live UI at session creation, you gain access to a real-time dashboard showing:

  • Jobs and Stages – Track active, completed, and failed jobs with stage-level metrics.
  • Executors – Monitor memory usage, shuffle data, and executor health.
  • SQL – Inspect query plans and execution details.
  • Connect tab – View active Spark Connect sessions and operations (specific to Spark Connect).

Access the dashboard through the GetDashboardUrl API or directly from the AWS Glue console.

import boto3, webbrowser

glue = boto3.client("glue", region_name="us-east-1")
dashboard = glue.get_dashboard_url(
    ResourceId="my-spark-connect-session",
    ResourceType="SESSION",
)
webbrowser.open(dashboard["Url"])

In SageMaker Unified Studio, no API call is needed. Choose Ready in the notebook status bar to open the kernel info popover. From there, open the Spark UI link for the live dashboard or Spark Driver Logs for real-time log output.

Notebook status bar Ready button that opens the Spark UI and Spark Driver Logs links

Image showing “Ready” in the status bar to access Spark UI and Driver Logs directly from the notebook

Token refresh

Authentication tokens expire after 30 minutes. In SageMaker Unified Studio, this is handled automatically. For programmatic use, you can use a background thread to keep the connection alive. The following helper reconnects transparently before the token expires:

import threading, time, boto3, urllib.parse
from pyspark.sql import SparkSession

class GlueSparkConnect:
    """Maintains a SparkSession with automatic token refresh."""

    def __init__(self, session_id, region="us-east-1", refresh_margin=300):
        self.session_id = session_id
        self.glue = boto3.client("glue", region_name=region)
        self.refresh_margin = refresh_margin  # seconds before expiry to refresh
        self._lock = threading.Lock()
        self.spark = self._connect()
        self._start_refresh_loop()

    def _connect(self):
        sc = self.glue.get_session_endpoint(SessionId=self.session_id)["SparkConnect"]
        encoded_token = urllib.parse.quote(sc["AuthToken"], safe="")
        remote_url = f"{sc['Url']}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
        self._token_expiry = sc["AuthTokenExpirationTime"].timestamp()
        return SparkSession.builder.remote(remote_url).getOrCreate()

    def _start_refresh_loop(self):
        def _loop():
            while True:
                sleep_for = max(self._token_expiry - time.time() - self.refresh_margin, 30)
                time.sleep(sleep_for)
                with self._lock:
                    self.spark = self._connect()
        t = threading.Thread(target=_loop, daemon=True)
        t.start()

# Usage
session = GlueSparkConnect("my-spark-connect-session")
session.spark.sql("SELECT 1 + 1 AS result").show()

The background thread sleeps until 5 minutes before token expiry, then transparently reconnects. Because the daemon thread exits when your script ends, there is no cleanup required.

Getting started

To start using Spark Connect with AWS Glue interactive sessions:

  1. Use AWS Glue version 5.1 (Apache Spark 3.5.6).
  2. Install PySpark 3.5.6 locally: pip install pyspark==3.5.6.
  3. Grant your AWS Identity and Access Management (IAM) identity permissions for glue:CreateSession, glue:GetSession, and glue:GetSessionEndpoint.
  4. Create a session with --session-type SPARK_CONNECT and connect from your preferred environment.

VPC note: If you connect to AWS Glue interactive sessions through a virtual private cloud (VPC) endpoint, add the new Spark Connect endpoint (com.amazonaws.{region}.glue.sessions) to your VPC configuration. Existing AWS Glue VPC endpoints don’t cover Spark Connect traffic.

For detailed instructions, see Connecting to a Spark Connect session in the AWS Glue Developer Guide.


About the authors

Zach Mitchell

Zach Mitchell

Zach is a Senior Big Data Architect at AWS Worldwide Specialist Organization for Analytics. He works with customers to design and build data applications on AWS, with a focus on SageMaker Unified Studio, AWS Glue, and AWS Lake Formation. Outside of work, he enjoys building things with code and occasionally writing about it.

Shrey Malpani

Shrey Malpani

Shrey is a Senior Technical Product Manager at AWS Analytics. He is focused on building and scaling data processing, data integration, and data management capabilities across services like AWS Glue, Amazon EMR, and Amazon Redshift that help customers build AI-ready data platforms for their analytics or machine learning workflows.

Vaibhav Naik

Vaibhav Naik

Vaibhav is a Software Engineer at AWS Glue, where he leads the development of enterprise Generative AI managed services and Agentic data systems. He has over a decade of experience designing massive-scale cloud infrastructure and distributed computing platforms.

Tom Olson

Tom Olson

Tom is a Software Development Engineer on the AWS Glue team, focused on Interactive Sessions and operational excellence. He brings over 20 years of software development experience, including government contracting and EC2 Networking at AWS. Outside of work, he enjoys running and playing board games.

Gaurav Krishnan

Gaurav Krishnan

Gaurav is a Software Development Engineer at AWS Glue. He has a deep interest in distributed systems and creating low-friction developer experiences for interactive data workloads on Apache Spark. In his spare time, he enjoys running and trying new restaurants.

Amazon Redshift RG: Faster and lower cost, Graviton-powered

Post Syndicated from Stefan Gromoll original https://aws.amazon.com/blogs/big-data/amazon-redshift-rg-faster-and-lower-cost-graviton-powered/

Amazon Redshift recently announced the general availability of a new Graviton-powered instance called RG. Built on Amazon’s own Graviton processors, RG delivers:

  • Up to 2.2x faster performance for data warehouse workloads compared to RA3.
  • Up to 2.4x faster for Iceberg queries and 1.5x faster for Parquet queries through an integrated vectorized data lake engine.
  • No per-TB scan charges for data lake queries, eliminating the Amazon Redshift Spectrum cost applied on RA3 clusters.
  • 30 percent lower cost per vCPU compared to RA3.

RG is both faster and cheaper. While cloud vendors typically charge more for faster performance or newer generation hardware, Amazon Redshift delivers better performance at lower cost.

In this post, we describe the innovations that make RG instances so much faster. We also share benchmark results showing that RG delivers up to 4.2x better price-performance than other leading data warehouses.

What makes RG so fast

The new RG instances are built from the ground up to take advantage of Graviton processors. The vectorized engine of Amazon Redshift is optimized with Graviton-based single instruction, multiple data (SIMD) kernels to deliver accelerated, parallelized execution for analytics workloads. Operations like predicate evaluations over Parquet encodings use Graviton vector comparison, table lookup, and vector manipulation intrinsics. To support these increased processing speeds, RG instances use custom-built Nitro SSDs. This lets RG use faster local storage as a caching layer for Amazon Redshift Managed Storage (RMS), data lake scans, and intermediate result sets for computations that can’t fit in memory. RG’s JIT (Just-In-Time) Analyze feature also collects statistics from data lake files automatically as queries run, so the optimizer can produce significantly better query plans. Together, these represent innovation across the entire stack: hardware acceleration with Graviton, vectorized execution with SIMD kernels, high-speed storage with Nitro SSDs, and intelligent query planning with JIT Analyze.

These optimizations, coupled with RG’s purpose-built high-performance vectorized data lake engine, combine to make Amazon Redshift’s new RG instances up to 2.2x faster than RA3 for analytics workloads at 30 percent lower cost.

Purpose-built high-performance vectorized data lake engine

With RA3, data lake queries offloaded scans to a separate compute fleet known as Amazon Redshift Spectrum. Because data lake queries ran on this separate compute, additional overhead was introduced to transfer query metadata and results between RA3 clusters and the Spectrum fleet. Amazon Redshift RG instances include a completely new built-in scan layer designed from the ground up for data lakes. This new scan layer includes a purpose-built I/O subsystem that incorporates smart prefetch capabilities to reduce data latency. The new scan layer is also optimized to process Apache Parquet files, the most commonly used file format for Iceberg, through fast vectorized scans that use SIMD kernels optimized for Graviton. The scan layer includes sophisticated data pruning mechanisms that operate at both partition and file levels, which significantly reduces the volume of data that needs to be scanned. This pruning capability works with the smart prefetch system to create a coordinated approach that maximizes efficiency throughout the entire data retrieval process.

The new purpose-built vectorized data lake engine is up to 2.4x faster than RA3 for Iceberg queries and 1.5x faster than RA3 for Parquet queries.

Because this new vectorized data lake engine integrates directly with the core execution engine of Amazon Redshift, new performance optimizations are possible compared to RA3. With this architecture, data lake queries on RG now benefit from fast local data caching, improved bloom filters, vectorized Parquet scans, and advanced filtering and pruning.

RG also solves a common problem customers face when querying data in the lake: open-format files like Iceberg in Amazon Simple Storage Service (Amazon S3) often lack useful metadata and statistics, which makes it difficult to run a SQL query optimally.

Statistics are metadata about your data, such as distinct value counts, min/max values, distribution patterns, and row counts. The query optimizer uses this information to choose the most efficient way to run a query. For example, when joining two tables, the optimizer needs to know how many unique values each side produces to pick the right join strategy. Without statistics, it has to guess, which often leads to slower joins and unnecessary data movement across nodes. This is where Amazon Redshift’s new feature called JIT (Just-In-Time) Analyze comes in. RG instances automatically fetch and store statistics of your Iceberg files as queries run, so Amazon Redshift can choose query execution strategies that are far more optimized than it could without these statistics.

These improvements make scans of Iceberg and Parquet data much faster than RA3. Removing Amazon Redshift Spectrum compute also means RG instances remove the $5/TB cost for data lake queries, which makes data lake queries cheaper and costs predictable. This is a triple win for data lake price-performance: faster performance, lower compute cost, and no per-TB scan cost.

Faster insights from faster data loads

Amazon Redshift RG’s fast I/O and Graviton-optimized engine result in faster data loads compared to RA3. To measure this improved performance, we ran the data ingestion step of 10TB TPC-DS and TPC-H on equivalently sized RA3 and RG clusters. RG ingested the TPC-DS dataset 2x faster and the TPC-H dataset 1.4x faster, as shown in the following figure.

Bar chart comparing data ingestion time on RA3 and RG, showing RG loads TPC-DS 2x faster and TPC-H 1.4x faster

The new Graviton-based RG instances are up to 2.0x faster for data loads compared to RA3 instances. This means workloads can see the latest data sooner, and users and agents can get up-to-date insights faster. This faster ingestion on RG comes at 30 percent lower cost compared to RA3, resulting in up to 2.9x better price-performance for data loads compared to RA3 instances.

What customers are saying

Amazon Redshift customers are already seeing performance and cost benefits of switching to RG. Southwest Airlines and tombola tested their business-critical workloads, and found they could get better performance and save on cost:

Southwest Airlines

“Amazon Redshift RG instances have the potential to deliver meaningful business impact for Southwest Airlines. Based on initial testing in our development environment, our data warehouse workloads run 50–60% faster, and data lake analytics are 45% faster—enabling teams to get insights sooner, respond to operational conditions faster, and make data‑driven decisions with less latency. These early results are encouraging, and we are excited to validate and scale these improvements in production. All of this comes without per‑terabyte Spectrum scanning charges, delivering 30% lower cost than RA3 at a time when fuel prices continue to pressure industry margins!!”

— Sean Lynch, Vice President, Data and Architecture, Southwest Airlines

tombola

“The new Graviton-based Amazon Redshift RG instances delivered 1.8x–2x faster write throughput and up to 2.2x faster read speeds compared to RA3 across a diverse set of batch and analytical jobs — enabling us to process 40% more within the same window. Compressed ETL cycles, accelerated time-to-insight, and decision-making no longer bottlenecked by the pipeline — together, these translated directly into fresher data reaching our analysts and business teams sooner. What made this even more compelling was a concurrent 30% reduction in compute spend alongside the gains — delivering more for less is a rare outcome, and one worth highlighting. In a volume-heavy gaming industry at tombola, where query latency and cost compound at scale, this has been one of the more impactful platform decisions we’ve made this year.”

— Akshay Srinivasan, Data Engineer, tombola

Qoala

“After migrating our Amazon Redshift cluster from RA3 to the new Graviton-based RG instances, we saw 60–70% faster query processing times across our BI and analytics workloads. As a growing insurtech platform handling millions of policy transactions, faster time-to-insight means our data team can deliver dashboards and reports to the business sooner. We moved to a larger node configuration to accommodate future growth, and the performance gains far exceeded the incremental investment – making this one of the most impactful infrastructure decisions we’ve made this year.”

— Umar Abdul Aziz, VP of Data, Qoala

Performance results

To see how RG stacks up, we ran benchmarks derived from the industry-standard TPC-DS and TPC-H benchmarks at 10TB scale on the new Amazon Redshift RG instances and on leading alternative data warehouses. These benchmarks are designed to run queries of various operational requirements and complexities, such as ad hoc, reporting, iterative online analytical processing (OLAP), and data mining. We sized each data warehouse at approximately the same on-demand cost ($32/hr) and ran three power runs of each benchmark out of the box, with no special tuning or manual customization. The results are shown in the following charts.

Bar chart of TPC-DS 10TB price-performance showing Amazon Redshift RG leading alternative data warehouses

Bar chart of TPC-H 10TB price-performance showing Amazon Redshift RG leading alternative data warehouses

The new RG instance leads, and by a large margin. Better price-performance means better performance and lower cost.

Conclusion

Amazon Redshift RG instances are the next generation of analytics engine, delivering high performance for data warehouse and data lake workloads. Because RG supports all the same workloads and features as RA3, getting started is straightforward. See our migration guide for how to upgrade and start getting better performance at lower cost.

Find the best price-performance for your workloads

The benchmarks used in this post are derived from the industry-standard TPC-DS and TPC-H benchmarks, and have the following characteristics:

  • We use the schema and data unmodified from TPC-DS and TPC-H.
  • The queries are generated using the official TPC-DS and TPC-H kits with query parameters generated using the default random seed of the kits. TPC-approved query variants are used for a warehouse if the warehouse doesn’t support the SQL dialect of the default queries.
  • The test includes the 99 TPC-DS SELECT queries and 22 TPC-H SELECT queries. It doesn’t include maintenance and throughput steps.
  • Three power runs were run, and the best run is taken for each data warehouse.
  • Price-performance is calculated as the cost per hour (USD) divided by 3,600 seconds/hour times the benchmark geomean in seconds, which is equivalent to the geomean cost per query. The latest published on-demand pricing is used for all data warehouses.

We call this the Cloud Data Warehouse benchmark, and you can reproduce the preceding benchmark results using the scripts, queries, and data available in our GitHub repository. It’s derived from the TPC-DS benchmarks as described in this post, and as such isn’t comparable to published TPC-DS results, because the results of our tests don’t comply with the official specification.


About the authors

Stefan Gromoll

Stefan Gromoll

Stefan is a Principal Engineer with the Amazon Redshift team where he is responsible for Redshift performance. In his spare time, he enjoys cooking, playing with his four boys, and chopping firewood.

Ankit Sahu

Ankit Sahu

Ankit brings over 18 years of expertise in building innovative data products and services. His diverse experience spans product strategy, go-to-market execution, and digital transformation initiatives. Currently, as Sr. Product Manager at Amazon Web Services (AWS), Ankit is driving the vision and strategy for Amazon Redshift.

Mohammed Alkateb

Mohammed Alkateb

Mohammed is an Engineering Manager at Amazon Redshift, leading Software Engineers, Applied Scientists, and Amazon Scholars across query optimization, data lake access, performance engineering, and new instance qualification. Prior to Amazon, he spent over 12 years with the Teradata Optimizer team. Mohammed holds a PhD from The University of Vermont and has many US patents and publications in premier database conferences.

Yousuf Hussain

Yousuf Hussain

Yousuf is a Senior Software Engineer at Amazon Redshift with 11 years of experience in building and operating large-scale cloud data warehouse systems. He is passionate about analytics and focuses on instance strategy, availability, and reliability to deliver a performant experience for Amazon Redshift customers.

Nita Shah

Nita Shah

Nita is a Sr. Analytics Specialist Solutions Architect at AWS based out of New York. She has been building enterprise data platforms, data warehousing, and analytics solutions for over 20 years and specializes in Amazon Redshift. She is focused on helping customers design and build enterprise-scale well-architected analytics and decision support platforms.

Sanket Hase

Sanket Hase

Sanket is an Engineering Manager with the Amazon Redshift team, where he leads query execution teams focusing on data lake analytics, hardware-software co-design, and vectorized query execution. Sanket holds a Master’s in CS from Carnegie Mellon University and has several U.S. patents in the field of database systems

Jingbo Zhang

Jingbo Zhang

Jingbo is a Data Engineer at Amazon Redshift focused on new instance qualification and performance validation. She has contributed to the qualification and launch of multiple Graviton-based Redshift instance families, including RG, r8gd, and r7gd, with a focus on benchmarking, performance analysis, and automation. Jingbo holds a master’s degree in data Analytics from Carnegie Mellon University.

Run log analytics for a fraction of the cost with the new engine for Amazon OpenSearch Service

Post Syndicated from Jagadish Kumar original https://aws.amazon.com/blogs/big-data/run-log-analytics-for-a-fraction-of-the-cost-with-the-new-engine-for-amazon-opensearch-service/

Amazon OpenSearch Service is a real-time retrieval engine for AI, search, and analytics at any scale. As log volumes grow 30–40 percent year over year, organizations face rising infrastructure costs and slower analytical queries across their observability data. Teams are forced to choose between retaining the data they need and staying within budget.

We’re introducing a purpose-built log analytics engine for Amazon OpenSearch Service. This new engine delivers up to 4x price performance, 2x faster data ingestion, up to 2x faster analytical queries, and up to 70 percent lower storage costs. You get all of this without sacrificing search capabilities on the same data.

In this post, you learn how to take advantage of these benefits, see how to get started, and review benchmark results at billion-document scale.

How the optimized engine works

The optimized engine is a new engine mode within the same Amazon OpenSearch Service domain. You use the same console, APIs, security model, and networking configuration that you already use with the general-purpose engine.

OpenSearch Service stores all data in Apache Parquet format. For fields configured as searchable, OpenSearch Service also writes the data to the inverted index. Apache Calcite parses and optimizes each query, then routes operations to the engine best suited to execute them: Apache DataFusion for analytical operations on columnar data, or Lucene for search predicates. The two hand off mid-query, so a single query can search log content and aggregate the results without additional roundtrips.

You ingest data through the same REST APIs and client libraries you use today and you don’t need to change your agents or pipelines. The optimized engine supports two query languages: Piped Processing Language (PPL) and SQL. Both execute natively through the vectorized engine. The Domain Specific Language (DSL) query API is not supported on the optimized engine at launch.

Getting started

At launch, the optimized engine is a domain-level setting selected at creation time. You can’t add the optimized engine to an existing domain or enable it on individual indices or fields within a general-purpose domain. To adopt the optimized engine, create a new domain and migrate your ingestion pipelines to it.

Create a new domain in the Amazon OpenSearch Service console and select Observability as your use case. The optimized engine is enabled by default. The console provides a side-by-side comparison of capabilities to help you choose.

Amazon OpenSearch Service console showing the Observability use case selected with a side-by-side comparison of engine capabilities

After your domain is ready, ingest JSON documents through the same Bulk API and client libraries you use today. No changes to your ingestion pipelines or application code are required.

Benefits of the optimized engine for log analytics

The optimized engine for log analytics introduces the following performance and cost improvements:

  • Up to 4x better price-performance compared to the existing general-purpose engine on internal benchmarks, while retaining full-text search for incident investigation.
  • Up to 2x faster analytical queries. The engine uses a vectorized query execution path that processes data in columnar batches for fast results across large datasets.
  • Up to 2x higher ingestion throughput. The append-only columnar write path increases sustained ingestion rates.
  • Up to 70 percent lower storage with columnar storage for aggregation workloads. You can retain up to 3x more data at the same cost.

To demonstrate these improvements, we benchmarked observability workloads at billion-document scale. In the following sections, we explore the benchmark methodology, test environment, and results. We recommend testing the optimized engine with your own workload to validate the gains for your use case.

Benchmark methodology

We used the Telemetry Generator for OpenTelemetry to generate synthetic traces and logs at scale, producing three observability datasets: OTEL traces, OTEL logs, and web server access logs. We stored the generated data as bulk-format NDJSON in Amazon Simple Storage Service (Amazon S3). We then ingested it through a pipeline on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. The pipeline reads chunks from Amazon S3, transforms timestamps, and writes to the OpenSearch Bulk API, simulating a production observability flow.

We benchmarked on two OpenSearch Service domains running OpenSearch 3.5, each with 9 data nodes in a 3-Availability Zone configuration:

Configuration Optimized Engine Standard Lucene
Instance type 9x or2.4xlarge.search 9x r8g.4xlarge.search
Leader nodes 3x m7g.large.search 3x m7g.large.search
EBS 2,500 GB gp3, 7,500 IOPS, 500 MB/s per node 2,500 GB gp3, 7,500 IOPS, 500 MB/s per node
Engine mode OPTIMIZED General Purpose (best_compression)

We ingested three data sets totaling 24.4 billion documents and 9.5 TB of raw JSON. All indices used 9 primary shards, 1 replica, and Index State Management (ISM)-managed rollover at 50 GB per primary shard. The Lucene baseline used best_compression (zstd) codec with _source enabled, representing the default customer configuration.

The ingestion pipeline ran on 90 Fargate tasks (16 vCPU, 120 GB RAM each, 48 writer threads per task, bulk size of 3,000 documents) in the same virtual private cloud (VPC) as the OpenSearch Service domains.

Results

Ingestion throughput

The optimized engine’s append-only columnar storage writes segments in bulk-optimized batches without per-document stored field overhead.

Metric Optimized Engine Lucene Baseline
Peak throughput 1.78M docs/sec ~647K docs/sec
Cluster CPU at peak 62% 72%
Write rejections 0 0
Total documents ingested 24.4 billion 15.7 billion

The optimized engine sustained 1.78 million documents per second at matched concurrency, approximately 2x the throughput of the Lucene baseline, while consuming less CPU. Both domains ran with zero write rejections. For teams ingesting terabytes per day, the throughput advantage translates to fewer nodes for the same volume, or longer retention on the same infrastructure.

Storage compression

The columnar Parquet format compresses observability data through dictionary encoding of repeated fields, tight packing of numeric columns, and elimination of per-document JSON overhead.

Measured across 24.4 billion documents:

Dataset Documents Source Optimized Engine Lucene (default)

Compression

vs.

source

Savings vs. Lucene
Web logs 8.76B 2,360 GB 254 GB 614 GB 89% 59%
OTEL logs 8.20B 3,720 GB 815 GB 1,549 GB 78% 47%
OTEL traces 7.43B 4,131 GB 841 GB 1,790 GB 80% 53%
Total 24.4B 9,539 GB 1,910 GB 3,953 GB 80% 52%

The optimized engine stores the same data at 5x compression versus raw JSON (80 percent savings). Against the default Lucene configuration (_source enabled, what most domains run), the optimized engine uses roughly half the storage. The optimized engine derives _source from Parquet columns on read, eliminating the need to store the raw JSON blob while still allowing document retrieval.

Analytical query performance

We measured query latency on a typical observability dashboard pattern: analytical aggregations scoped to a 15-minute time window over billions of log events. The optimized engine uses row-group pruning on the @timestamp column to skip data outside the query window, reading only the relevant subset.

Query pattern Dataset Optimized Engine Lucene baseline Speedup
Error count by service OTEL logs 717 ms 2.8 s 3.9x
Log volume by host OTEL logs 252 ms 17.6 s 70x
5xx errors by service and method OTEL logs 171 ms 885 ms 5.2x
Top services by error OTEL traces 635 ms 569 ms ~1x
Point lookup (single traceId) OTEL traces 394 ms 783 ms 2x

All queries scoped to a 15-minute window. Index sizes: 8.2 billion OTEL log events, 7.4 billion OTEL trace spans.

The optimized engine completes time-filtered analytical queries in 171 ms to 717 ms over billions of documents. The advantage is most pronounced on unfiltered aggregations (log volume by host: 70x) where the columnar engine reads only the columns needed. On queries where the Lucene inverted index provides strong predicate selectivity (top services by error on traces), performance is comparable between the two engines.

Search and point lookups

The optimized engine retains the Lucene inverted index alongside columnar storage. When the query planner recognizes a selective lookup (such as retrieving a single trace by ID), the planner routes the query to the inverted index rather than scanning columnar data. In our benchmark, a single traceId lookup across 7.4 billion spans returned in 165 ms.

This means a real investigation can use both engines in sequence: broad aggregations to localize the problem, then a point lookup to pull the offending trace, all from the same domain.

Now available

The optimized engine for Amazon OpenSearch Service is generally available today in all commercial AWS Regions (Regions other than the AWS GovCloud (US) Regions and the China Regions) where OpenSearch Optimized Instances are available.

Pricing follows standard Amazon OpenSearch Service rates for instances and storage, with no additional premium for the optimized engine. For more information, see Amazon OpenSearch Service Pricing.

To learn more about configuring and using the optimized engine, see Optimized for Log Analytics in the Amazon OpenSearch Service documentation. For an overview of the service, visit Amazon OpenSearch Service Log Analytics.

Give it a try and send feedback to AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.


About the authors

Jagadish Kumar

Jagadish Kumar

Jagadish is a Senior Solutions Architect at Amazon Web Services, focused on OpenSearch and analytics workloads.

Rohin Bhargava

Rohin Bhargava

Rohin is a Senior Product Manager for Amazon OpenSearch Service.

Michael Supangkat

Michael Supangkat

Michael is a Solutions Architect at Amazon Web Services specializing in search and observability.

Scale analytics with Amazon Redshift multi-warehouse enhancements

Post Syndicated from Raza Hafeez original https://aws.amazon.com/blogs/big-data/scale-analytics-with-amazon-redshift-multi-warehouse-enhancements/

Onboard analytics workloads at scale with Amazon Redshift’s improved remote table data definition language (DDL), materialized view improvements, and concurrency scaling enhancements for zero-ETL and auto-copy.

As organizations scale their analytics capabilities, they need the ability to add workloads without disrupting production operation or being constrained by the resources of a single data warehouse. In this post, we introduce new capabilities of Amazon Redshift that enhance our multi-warehouse and scaling capabilities: remote materialized view (MV) operations, remote table DDL support, and concurrency scaling enhancements for zero-ETL and S3 event integration. These features help you build more scalable, performant decentralized analytics architectures on Amazon Redshift.

Let us review how these new features enable you to run analytics at scale.

New remote materialized view operations

New remote table DDL operations

  • ALTER TABLE ALTER DISTSTYLE operations now work on remote warehouses through concurrency scaling and data sharing. You can dynamically optimize data distribution across distributed environments, improving query performance and resource utilization without requiring data migration. This is especially valuable for data engineers fine-tuning performance across multiple warehouses and administrators adapting to changing query patterns.
  • ALTER TABLE APPEND operations now extend to remote warehouses through concurrency scaling and data sharing. This consolidates data across distributed environments, so you can efficiently combine tables without complex data movement or extract, transform, and load (ETL) processes. Organizations managing dynamic table operations across multiple environments can maintain data consistency while reducing operational overhead.

Concurrency scaling improvements

With these new concurrency scaling capabilities, you can maintain consistent data freshness without compromising existing warehouse performance. This eliminates the traditional trade-off between analytics and data loading. Apart from turning on concurrency scaling, no additional changes are required to take advantage of these features.

Customer use cases

This section covers two industry use cases: the first for a financial services customer and the second for a gaming industry customer.

Financial services use case

The following is a sample architecture for a large financial services customer with global operations. This customer uses a multi-warehouse architecture built on Amazon Redshift.

Financial services multi-warehouse architecture using STG, DWH, ETL, and USR Amazon Redshift warehouses

The staging (STG) warehouse serves as a raw zone for data from various sources, like the bronze layer of a medallion architecture. This warehouse also cleanses and standardizes the raw data to the silver layer and makes it available for further processing. The STG warehouse uses MVs to process millions of nested JSON messages and extract attributes into scalar columnar Amazon Redshift tables.

CREATE MATERIALIZED VIEW rawdb.fsi.customer_orders_raw
distkey(c_custkey) sortkey(c_custkey) AS (
    SELECT c_custkey,
        o.o_orderstatus,
        o.o_totalprice,
        o_idx
    FROM customer_orders_lineitem c,
        c.c_orders o AT o_idx
);
REFRESH MATERIALIZED VIEW rawdb.fsi.customer_orders_raw;

The DWH warehouse serves as the primary Amazon Redshift instance and gold layer, providing data to consuming applications like Business Objects and Tableau. The zero-ETL concurrency scaling improvements provide consistent data freshness even when zero-ETL ingestion spikes occur alongside heavy DWH workloads. The DWH MVs provide fast access to aggregated data for Tableau extracts and Business Objects live reports. The DWH warehouse takes advantage of concurrency scaling when multiple MVs need to be refreshed on the DWH instance.

CREATE MATERIALIZED VIEW bodb.final.customer_churn_tbl
AS (
    SELECT state,
        account_length,
        area_code,
        total_charge/account_length AS average_daily_spend,
        cust_serv_calls/account_length AS average_daily_cases,
        churn
    FROM custdb.final.customer_activity_all
);
REFRESH MATERIALIZED VIEW bodb.final.customer_churn_tbl;

The ETL01/02 warehouses serve as dedicated compute environments for running project-specific ETL jobs, while the USR01/02 warehouses handle user workloads such as ad-hoc analysis or model building from dbt. When new objects are required by user workloads, they are created and maintained on the remote producer warehouse (DWH).

ALTER TABLE salesdb.final.sales_report_all
ALTER DISTKEY sales_id;
ALTER TABLE APPEND salesdb.final.sales_report_all
FROM stagingdb.sales.sales_2026_02;

Gaming industry use case

A leading gaming company has built their entire analytics infrastructure on AWS, with their analytics team managing data streaming from games, data warehousing, and business intelligence tools. They standardized Amazon Redshift across the organization, migrating off Vertica running on Amazon Elastic Compute Cloud (Amazon EC2). After overcoming early challenges with cluster resize operations, the team became strong advocates for Amazon Redshift and now runs their primary production cluster on 32 ra3.16xlarge nodes.

As their data ingestion pipeline grew, query workloads began competing with data ingestion processes, creating performance bottlenecks. Rather than scaling up their primary cluster, they implemented a workload isolation strategy using Amazon Redshift data sharing. The customer launched a second 16-node ra3.4xlarge cluster as a data share consumer, with the primary cluster serving as the producer. This architecture allowed them to migrate consumption workloads to the consumer cluster while the producer focused on data ingestion, effectively supporting growth without increasing the primary cluster size.

Gaming company architecture with a producer Amazon Redshift cluster sharing data to a consumer cluster

Recognizing the advantages of this distributed architecture, the gaming company expanded their approach by migrating workloads to Amazon Redshift Serverless, further using the data sharing model for workload isolation. Amazon Redshift’s remote materialized view capability allowed the gaming company to create materialized views directly on the data shared by the producer cluster. Each consumer cluster could now build materialized views optimized for its specific workload patterns. This created pre-aggregated datasets, custom join strategies, and workload-specific data distributions, without impacting the producer cluster’s performance or requiring data duplication. The producer warehouse maintains data distribution and sorting strategies designed for generic enterprise needs, providing consistent data quality across all consumers. Meanwhile, consumer warehouses used remote materialized views to fine-tune query performance for their distinct analytical requirements, whether supporting real-time player analytics, business intelligence dashboards, or ad-hoc data science workloads. This distributed approach to data consumption optimization proved essential for the gaming company. It delivered fast query performance across diverse analytical workloads while maintaining a single source of truth in the producer cluster and avoiding the operational overhead of managing redundant data copies.

Best practices

To get the most out of these new capabilities, consider the following best practices:

  • Enable concurrency scaling on your Amazon Redshift clusters and Serverless workgroups to allow ETLs and user queries to run even faster, providing consistent report and dashboard performance.
  • Set up usage limits for concurrency scaling on both Amazon Redshift provisioned clusters and Serverless workgroups by configuring an appropriate MaxRPU setting. This helps you avoid unexpected additional costs. For more information, see the Amazon Redshift usage limits documentation.
  • Use remote MVs to offload resource-intensive MV creation and refresh operations from your primary warehouse to remote data share clusters.

Conclusion

In this post, we walked through the new MV refresh features, remote table DDL capabilities, and expanded concurrency scaling support for zero-ETL and S3 auto-copy. These features help you move beyond the constraints of a single warehouse. They are particularly valuable for organizations managing distributed data architectures that require dynamic table management across multiple environments while maintaining data consistency and adapting quickly to changing workloads. To get started, make sure you are running the latest Amazon Redshift version. Then visit the Amazon Redshift documentation to learn more about concurrency scaling, data sharing, and materialized views.


About the authors

Raza Hafeez

Raza Hafeez

Raza is a Senior Product Manager, Technical at Amazon Redshift. He has 15+ years of experience building and optimizing enterprise data warehouses and is passionate about making cloud analytics accessible and cost-effective for customers of all sizes.

Ravi Animi

Ravi Animi

Ravi is a senior product leader in the Amazon Redshift team and manages several functional areas of the Amazon Redshift cloud data warehouse service, including spatial analytics, streaming analytics, query performance, Spark integration, and analytics business strategy. He has experience with relational databases, multidimensional databases, IoT technologies, storage and compute infrastructure services, and more recently, as a startup founder in the areas of artificial intelligence (AI) and deep learning, computer vision, and robotics.

Satesh Sonti

Satesh Sonti

Satesh is a Principal Analytics Specialist Solutions Architect based in Atlanta, specializing in building enterprise data platforms, data warehousing, and analytics solutions. He has over 20 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Milind Oke

Milind Oke

Milind is a senior Redshift specialist solutions architect who has worked at Amazon Web Services for three years. He is an AWS-certified SA Associate, Security Specialty and Analytics Specialty certification holder, based out of Queens, New York.

Detecting fraud patterns across Snowflake and AWS using SageMaker Data Agent

Post Syndicated from Akash Gupta original https://aws.amazon.com/blogs/big-data/detecting-fraud-patterns-across-snowflake-and-aws-using-sagemaker-data-agent/

Financial services organizations increasingly run analytical workloads across multiple systems. For example, customers typically store transaction records in Snowflake for its concurrency handling during peak volumes, while they store risk scores, customer profiles, and behavioral signals on AWS. To bridge that divide, practitioners have had to stitch together manual exports, custom extract, transform, and load (ETL) code, and external business intelligence (BI) tools to query both sources, cache expensive aggregations, and visualize results.

Amazon SageMaker Data Agent now closes these gaps with three new capabilities in Amazon SageMaker Unified Studio notebooks: SQL analytics on Snowflake data sources, materialized view management, and interactive charting. Practitioners can use them together to query Snowflake alongside AWS data, pre-compute and schedule repeated aggregations, and create interactive visualizations from natural language prompts in a single notebook, without writing boilerplate code or switching tools.

In this post, we describe the challenges these capabilities address, introduce each one, and walk through a fraud analytics scenario that demonstrates them working together in an end-to-end investigation workflow.

Challenges with fraud detection

Fraud analytics teams working in SageMaker Unified Studio notebooks encounter several recurring friction points that slow their path from alert to insight:

  • Querying across AWS and third-party warehouses. Customers store transaction data in Snowflake and maintain risk scores and customer profiles on AWS. SageMaker Data Agent supported SQL generation for AWS-native engines: Amazon Athena, Amazon Redshift, Apache Spark, and DuckDB. However, it didn’t yet generate Snowflake-dialect SQL. This created a gap for customers working with data distributed across both AWS services and Snowflake. Analysts had to write Snowflake SQL manually and export results as CSV files to join with AWS data. The process consumed 1–2 hours before any actual investigation could begin.
  • Rich visualization requires coding expertise. When analysts want to plot query results, they must write Python code using packages like matplotlib, seaborn, or plotly. They must choose the right chart type, format axes, handle data transformations, and debug rendering issues. For fraud teams whose expertise is in investigation rather than data visualization code, each chart becomes a detour: either learn the package interface, ask an engineer for help, or export to an external BI tool. This slows the exploratory cycle that fraud investigations depend on, where every new angle (time-of-day patterns, category breakdowns, geographic clusters) ideally takes seconds, not minutes of code iteration.
  • Expensive repeated queries with no caching. Fraud signal queries flag transactions that exceed a customer’s historical average and compute risk-score distributions by merchant category. These queries re-scan entire tables on each execution. A team running the same aggregation every morning over millions of rows pays the full compute cost each time, with no mechanism to pre-compute results or schedule automatic refreshes. For fraud teams, this means investigations start with a 30-minute wait for queries that ran identically yesterday.

These three friction points (accessing data across platforms, visualizing it interactively, and operationalizing repeated analyses) are what the new Data Agent capabilities address together.

What’s new in Data Agent

Snowflake connectivity

SageMaker Data Agent can now connect to Snowflake data warehouses through connections registered in Amazon SageMaker Unified Studio. The agent discovers available Snowflake databases, browses schemas progressively (databases → schemas → tables → columns), and generates Snowflake-dialect SQL, including Snowflake-specific syntax like FLATTEN, VARIANT column access, and semi-structured data handling. Analysts query Snowflake tables alongside AWS data sources from a single notebook conversation, and the agent handles dialect differences automatically: Snowflake SQL for extraction, Spark SQL for Amazon Simple Storage Service (Amazon S3) Tables operations, with no manual translation required.

Materialized view management

Data Agent now creates and manages materialized views through natural language prompts. Analysts describe the aggregation they want, for example, “create a materialized view that flags transactions where risk_score is above 0.7, refreshed every 6 hours,” and the agent generates the Spark SQL DDL, including SCHEDULE REFRESH syntax. Materialized views store pre-computed results in Apache Iceberg format for fast repeated access, turning expensive full-table scans into sub-second queries. Supported operations include create, refresh, drop, describe, and scheduled refresh. When asked, Data Agent can also analyze notebook query patterns and recommend which queries would benefit from materialization.

Interactive charting

Instead of generating matplotlib code that produces static images, Data Agent now creates native interactive chart cells powered by Vega-Lite. Supported chart types include bar, line, scatter, pie, area, heatmap, and more. Charts render inline in the notebook with hover tooltips, zoom, and filtering. Analysts can reconfigure them through the sidebar or by typing inline instructions like “change this to a heatmap showing volume by hour and category.” This removes the cycle of modifying Python plotting code or exporting to an external BI tool every time the analysis needs a different view.

Detecting fraud patterns across Snowflake and AWS: a walkthrough

Solution overview

In this section, we walk through how these three capabilities work together in a realistic fraud investigation. A fraud analytics lead at a mid-size fintech processes a high volume of card transactions daily. Customers store transaction data in Snowflake and maintain customer risk profiles on AWS.

This morning, the real-time alerting system flagged an unusual spike in declined transactions from a cluster of new accounts, all purchasing high-value electronics. The analyst suspects a fraud ring using synthetic identities, fabricated customer profiles that pass initial verification but share telltale patterns like similar device fingerprints or overlapping IP ranges. The analyst has three goals:

  • Confirm the fraud ring hypothesis. Determine whether the flagged accounts share device fingerprints, IP ranges, or behavioral patterns indicating coordinated fraud.
  • Quantify the exposure. Calculate total fraudulent transaction volume and identify all affected accounts, not only the ones that triggered today’s alert.
  • Set up ongoing monitoring. Create a reusable, auto-refreshing query so the team catches the next ring faster.

The analyst wants to do all of this without leaving the SageMaker notebook, without writing boilerplate data-engineering code, and within a single morning standup cycle so the investigations team can be briefed by noon.

How Data Agent approaches this analysis

Data Agent is context-aware. It discovers your actual table names, column schemas, and data source connections through Amazon SageMaker Unified Studio rather than requiring you to specify them manually. It generates SQL in the correct dialect for each source (Snowflake SQL for Snowflake, Spark SQL for S3 Tables) and operates within your existing AWS Identity and Access Management (IAM) permissions boundaries.

You interact with Data Agent in two modes: the Agent Panel for multi-step investigations like the example walkthrough that follows, where each prompt builds on previous context, and inline interactions for quick adjustments like “change this to a heatmap” directly on a chart cell.

Prerequisites

Before starting this walkthrough, verify that you have:

  • An Amazon SageMaker Unified Studio domain with a project configured.
  • A Snowflake account with a warehouse and USAGE grants on the database and schemas you want to query.
  • A Snowflake connection registered in your SageMaker Unified Studio project.
  • An S3 Tables catalog in your project containing customer data (or equivalent AWS-hosted tables for joining with Snowflake data).
  • A notebook open in SageMaker Unified Studio with Data Agent available in the chat panel.

Step 1: Explore Snowflake transaction data

What the analyst wants: Before investigating the fraud ring, the analyst must understand what data is available in Snowflake and verify recent transactions are accessible. The schema isn’t memorized (the payments team manages these tables), so Data Agent needs to discover the structure.

In the SageMaker notebook Agent Panel, the analyst types:

“Show me a preview of transactions over $500 for the last 24 hours. I’m looking for repeated high-value purchases that might indicate synthetic identity fraud.”

What Data Agent does for you: Data Agent discovers the Snowflake connection through SageMaker Unified Studio, browses the available databases, and locates PAYMENTS_DB → CARD_TRANSACTIONS schema → transactions table. It surfaces the column structure (transaction_id, customer_id, amount, merchant_category, transaction_timestamp, device_fingerprint, ip_address) so the analyst can confirm the right data is available without writing a single DESCRIBE TABLE statement.

Data Agent then generates a Snowflake-dialect SQL query to preview the last 24 hours of high-value transactions (amount > $500), returning hundreds of results. The preview immediately reveals what was suspected: alongside legitimate high-value purchases (mortgage payments, business supplies), there are clusters of electronics purchases at similar price points from different customer_id values but the same device_fingerprint, a classic synthetic identity pattern.

Data Agent querying Snowflake transaction data and generating equivalent code in the cell

Figure 1: Data Agent querying Snowflake transaction data and generating equivalent code in the cell.

Notebook cell results showing high-value Snowflake transactions

Figure 2: Displaying results when the notebook cell runs.

Step 2: Land Snowflake data into S3 Tables and join with risk profiles

What the analyst wants: Pulling historical high-value transactions into S3 Tables makes this data available for downstream analysis, including the materialized view that will cross-reference risk profiles automatically.

“Load the last 90 days of transactions where amount is greater than 500 into S3 Tables.”

What Data Agent does for you: Data Agent queries Snowflake to extract a large volume of high-value transactions from the last 90 days, converts the result to a PySpark DataFrame, creates an Apache Iceberg table at payments.fraud_analytics.high_value_transactions, and writes all the rows. Data Agent stores the transaction data (transaction_id, customer_id, amount, merchant_category, transaction_timestamp, device_fingerprint, ip_address) as Iceberg in S3 Tables, allowing you to query it entirely on AWS.

Data Agent handles the cross-source complexity: Snowflake-dialect SQL for extraction, automatic schema inference for the Iceberg table, and PySpark for the write. The analyst didn’t write a single line of ETL code.

Prompt sent to Data Agent to land Snowflake transactions into an S3 Tables

Figure 3: Sending a prompt to land Snowflake transactions into an S3 Tables catalog.

Generated PySpark code that reads transaction data from Snowflake

Figure 4: Reading data from Snowflake using code Data Agent generated.

Generated cell creating an S3 Tables Iceberg table populated with Snowflake data

Figure 5: Data Agent creating a new cell to create an S3 Tables Iceberg table and populate it with the Snowflake data.

Step 3: Create a materialized view for ongoing fraud monitoring

What the analyst wants: The pattern is confirmed, but re-running this expensive join across two tables every morning isn’t sustainable. A pre-computed view that automatically refreshes and surfaces transactions from high-risk customers means tomorrow’s investigation starts with answers instead of queries (goal #3, ongoing monitoring).

“Create a materialized view called mv_fraud_signals that joins high_value_transactions with customer_risk_profiles, flagging transactions where risk_score is above 0.7. Refresh it every 6 hours.”

What Data Agent does for you: Data Agent browses the S3 Tables catalog to discover both tables and their schemas, generates the Spark SQL DDL with SCHEDULE REFRESH EVERY 6 HOURS, and creates an INNER JOIN on customer_id with a risk_score > 0.7 filter. The resulting materialized view contains only the high-risk subset of transactions, and subsequent queries against it return significantly faster compared to a full table scan.

Data Agent can also recommend materialized views when asked. If the analyst prompts “analyze my notebook and suggest which queries would benefit from materialized views,” Data Agent examines query patterns and suggests candidates. This is useful when a team runs the same expensive aggregations repeatedly without realizing a materialized view would help.

New cell created by Data Agent to create the mv_fraud_signals materialized view

Figure 6: Data Agent creates a new cell to create the materialized view.

Generated query against the newly created materialized view

Figure 7: Data Agent adds code to query the newly created materialized view.

Step 4: Visualize fraud patterns with interactive charting

What the analyst wants: The data is ready, but the investigations team needs a clear visual story by noon to see which merchant categories are targeted and what time of day the fraud occurs, so they can build detection rules. The team needs interactive charts that can be explored on the fly, not static matplotlib images that need regenerating every time someone asks “what about category X?”

“Show me a scatter plot of flagged transactions: amount vs risk_score, colored by merchant_category.”

What Data Agent does for you: Data Agent queries the materialized view, generates a Vega-Lite specification, and renders an interactive scatter plot directly in the notebook cell, with no matplotlib code and no BI tool export. Hovering over any point reveals the transaction details. A dense cluster immediately stands out: Electronics & Computers transactions with risk scores between 0.75–0.95, all in the $950–$1,000 range.

Generated scatter plot of flagged transactions colored by merchant category

Detail view of the scatter plot highlighting the Electronics cluster

Figures 8, 9, and 10: Data Agent creates a scatter plot showing a dense cluster of Electronics transactions in the $950–$1,000 range with risk scores between 0.75–.95.

The analyst follows up with a second prompt to explore temporal patterns:

“Change this to a heatmap showing transaction volume by hour of day and merchant category.”

What Data Agent does for you: Data Agent generates a new heatmap visualization from the same materialized view. The heatmap reveals that Business Supplies and Mortgage Payments maintain steady transaction volumes throughout the day. However, Electronics shows a distinctly uneven temporal distribution, with noticeable volume dips during early morning hours (midnight to 5 AM) and late evening. This variability, absent in legitimate purchase categories, is a signal the detection rules team can act on immediately.

Heatmap of transaction volume by hour and merchant category

Detail view of the heatmap showing off-hours dips in the Electronics row

Figures 11 and 12: Data Agent creates a heat map to show transaction volume by hour of day and merchant category, revealing uneven temporal distribution in high-risk categories.

From insight to action

This investigation, from Snowflake connection to visual evidence, streamlined a workflow that previously required significant time across multiple tools. The analyst shares the notebook link with the investigations team, who confirm a fraud ring of dozens of synthetic identities responsible for significant fraudulent purchases. The temporal pattern, uneven Electronics transaction distribution with off-hours variability, is added to the company’s real-time detection rules that same afternoon.

The materialized view continues refreshing every 6 hours. The next morning, it flags three new accounts matching the same pattern, caught within hours of their first transaction instead of days.

Why SageMaker Data Agent for fraud analytics

This walkthrough demonstrates three new capabilities working together:

  • SQL analytics on Snowflake data sources removed the CSV export and manual ETL that consumed half of the investigation time.
  • Materialized view management turned a one-time query into persistent, auto-refreshing monitoring, transforming reactive investigations into proactive detection.
  • Interactive charting kept the entire analysis in the notebook, removing the BI tool context switch and making the inline exploration that revealed the Electronics temporal anomaly possible.

For the team, the combined effect is a reduction in time-to-insight, allowing faster fraud pattern analysis. This means daily fraud pattern reviews instead of weekly, and an investigation workflow that’s reproducible. The notebook itself serves as documentation for compliance and audit purposes.

Cleanup

The walkthrough creates notebook cells, SQL queries, and materialized views in your SageMaker Unified Studio session. To remove the generated cells, delete them from your notebook or delete the notebook itself.

If you created resources specifically for this walkthrough, remove the following to avoid ongoing charges:

  • Materialized view. In the notebook Agent Panel, prompt: “Drop the materialized view mv_fraud_signals.” This removes the Iceberg table from S3 Tables and cancels the scheduled refresh. Alternatively, run the Spark SQL statement DROP MATERIALIZED VIEW payments.fraud_analytics.mv_fraud_signals directly.
  • Landed Iceberg tables. Drop any tables created during the data landing step (for example, payments.fraud_analytics.high_value_transactions) by prompting Data Agent or running DROP TABLE in a Spark SQL cell. This removes the data from S3 Tables and the underlying Amazon Simple Storage Service (Amazon S3) storage.
  • SageMaker Unified Studio domain. If you created a domain solely for this walkthrough, delete it to stop incurring charges. Refer to the SageMaker Unified Studio administration guide for deletion steps.
  • Amazon S3 storage. Verify that dropping the materialized view and Iceberg tables removed the associated S3 objects. If residual Iceberg metadata files remain in your S3 Tables bucket, delete them manually.
  • Snowflake compute. No persistent Snowflake resources are created. Queries use your existing warehouse. Review your Snowflake query history to estimate the compute credits consumed during the walkthrough.

Conclusion

In this post, we walked through three new capabilities in Amazon SageMaker Data Agent for notebooks: Snowflake connectivity, materialized views, and native interactive charting. Using a fraud analytics scenario, we demonstrated how these features work together. We connected to a Snowflake warehouse to explore transaction data, landed results into S3 Tables and joined them with AWS-hosted risk profiles, created a materialized view for ongoing fraud monitoring, and visualized patterns with interactive charts that revealed temporal anomalies in Electronics transactions linked to dozens of synthetic identities.

These capabilities are available now in Amazon SageMaker Unified Studio. To get started, open a notebook in your SageMaker Unified Studio domain and begin a conversation with Data Agent in the chat panel.

To learn more, see the following resources:


About the authors

Akash Gupta

Akash Gupta

Akash is a Software Development Engineer on the Amazon SageMaker Unified Studio team, where he builds integrated tools and agentic experiences. An alumnus of Santa Clara University, he is passionate about building scalable solutions that simplify how customers interact with their data. In his spare time, he enjoys singing and cooking.

Mukesh Sahay

Mukesh Sahay

Mukesh Sahay is a Software Development Engineer at Amazon SageMaker, focused on building the SageMaker Data Agent. The agent provides intelligent assistance for code generation, error diagnosis, and data analysis recommendations for data engineers, analysts, and scientists. His work spans agentic AI architectures that transform natural language prompts into executable code and analysis plans across diverse data sources. An alumnus of San Jose State University, Mukesh brings over a decade and a half of experience in building scalable, intelligent data systems.

Eason Ma

Eason Ma

Eason is a Software Development Engineer within SageMaker’s Agentic AI Experiences. His focus is on building agentic infrastructure and intelligent data experiences that help users seamlessly interact with their data across multiple sources. He holds a Master’s in Computer Science from the University of Illinois at Urbana-Champaign and a Bachelor’s in Computer Science from the University of Tennessee, Knoxville. A proud Vol, he brings that same volunteer energy to everything he builds.

Anagha Barve

Anagha Barve

Anagha is a Software Development Manager on the Amazon SageMaker Unified Studio team. Her team is focused on building tools and integrated experiences for the developers using Amazon SageMaker Unified Studio. In her spare time, she enjoys cooking, gardening and traveling.

Siddharth Gupta

Siddharth Gupta

Siddharth is heading Generative AI within SageMaker’s Unified Experiences. His focus is on driving agentic experiences, where AI systems act autonomously on behalf of users to accomplish complex tasks. An alumnus of the University of Illinois at Urbana-Champaign, he brings extensive experience from his roles at Yahoo, Glassdoor, and Twitch.

Introducing Private Networking for Amazon MQ for RabbitMQ

Post Syndicated from Jean-Sébastien Dominique original https://aws.amazon.com/blogs/big-data/introducing-private-networking-for-amazon-mq-for-rabbitmq/

With Private Networking for Amazon MQ for RabbitMQ, your brokers can establish outbound connections to private resources in your VPC without exposing those resources publicly. This post explains how the feature works and walks you through setting it up.

Amazon MQ for RabbitMQ brokers could previously only reach external destinations over the public internet. If you used a private Lightweight Directory Access Protocol (LDAP) server for broker authentication, you had to expose that server publicly. If you wanted to federate messages between private brokers, you needed workarounds like Network Load Balancers with IP allowlisting, as described in Implementing Federation on Amazon MQ for RabbitMQ Private Brokers. Private Networking removes those constraints.

You can connect your broker to private identity providers, other Amazon MQ for RabbitMQ brokers, or self-hosted RabbitMQ brokers running in private subnets. Combined with cross-Region networking services like AWS Transit Gateway, you can extend these connections across AWS Regions and accounts, with traffic staying on the AWS private network.

How it works

Private Networking connects your broker to private destinations using three AWS services: Amazon VPC Lattice, AWS Resource Access Manager (AWS RAM), and AWS PrivateLink.

You create a VPC Lattice resource gateway in a VPC that can reach your private destination. You then create a VPC Lattice resource configuration that defines the destination, such as an IP address or Domain Name System (DNS) name. You add the resource configuration to a RAM resource share and associate the resource share with your broker through the UpdateBroker API operation. After rebooting the broker, the network path is active and your broker can reach the private destination.

The broker does not need to be private. A publicly accessible broker works the same way.

What you can connect to

Private Networking supports three use cases.

Private identity providers

If you use an LDAP server or other identity provider for RabbitMQ authentication, you no longer need to expose it publicly. Create a resource configuration pointing to your identity provider, associate it with your broker, and use the DNS name returned by the DescribeSharedResources API operation in place of the public endpoint. Follow the existing guidance for setting up an identity provider, substituting the private DNS name.

Self-hosted RabbitMQ brokers

You can use Shovel or Federation to connect your Amazon MQ for RabbitMQ broker to a self-hosted RabbitMQ broker running in a private subnet. Create a resource configuration pointing to the self-hosted broker and use the DNS name from the DescribeSharedResources API operation in your Shovel or Federation configuration.

This pattern is useful for hybrid cloud architectures where you run RabbitMQ on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), or on-premises infrastructure and want to exchange messages with Amazon MQ without exposing either side publicly.

Other Amazon MQ for RabbitMQ brokers

You can federate or shovel messages between two Amazon MQ for RabbitMQ brokers using Private Networking. Create a resource configuration pointing to the destination broker’s endpoint and specify that same endpoint as the custom domain name on the resource configuration. This helps to verify that the DNS name resolves correctly and Transport Layer Security (TLS) peer verification succeeds.

This extends to brokers in different AWS Regions and different AWS accounts. By combining Private Networking with cross-Region networking services like AWS Transit Gateway or VPC peering, you can build a fully private federation or shovel path between brokers, with no public endpoints required.

DNS names and custom domains

Each resource configuration can include a custom domain name. If you add a verified domain, that domain resolves to the private destination. If you do not add a verified domain, Amazon MQ provides a DNS name for the broker’s private connection. Retrieve this DNS name with the DescribeSharedResources API operation.

If you specify an unverified domain on a resource configuration, it is ignored. The broker’s private connection receives a private DNS name instead, which you can retrieve with the DescribeSharedResources API operation.

For more details on custom domain names and domain verification with VPC Lattice, see Custom domain names for VPC Lattice resources.

TLS peer verification in RabbitMQ 4

Note: If you are running RabbitMQ 4, review this section before configuring Shovel or Federation connections.

RabbitMQ 4 enforces TLS certificate peer verification by default for Shovel and Federation connections. RabbitMQ 3 does not enforce this by default. When using Private Networking, the DNS name that Amazon MQ assigns to the private connection will not match the TLS certificate of the destination, which causes peer verification to fail.

The recommended approach is to specify the destination broker’s endpoint (for example, b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111.mq.us-east-1.on.aws) as the custom domain name on the resource configuration. This exception only applies to Amazon MQ for RabbitMQ broker endpoints. You cannot use an unverified domain for self-hosted brokers. Specifying the Amazon MQ endpoint causes the DNS name to match the destination’s TLS certificate, and peer verification succeeds. This approach works regardless of your RabbitMQ version and avoids the issue entirely.

Getting started

To get started with Private Networking for Amazon MQ for RabbitMQ, follow these steps.

Prerequisites

Before you begin, verify you have the following:

  • An AWS account.
  • The AWS Command Line Interface (AWS CLI) installed and configured.
  • AWS Identity and Access Management (IAM) permissions to manage Amazon MQ, VPC Lattice, and AWS RAM resources.
  • An existing VPC with connectivity to your private destination.

Walkthrough

After you have the prerequisites, follow these steps:

  1. Create an Amazon MQ for RabbitMQ broker if you do not already have one.
  2. Create a VPC Lattice resource gateway in a VPC that can reach your private destination. Make sure the resource gateway’s security group allows outbound traffic to your destination on the required port (for example, port 5671 for AMQPS (AMQP over TLS) or port 636 for LDAPS (LDAP over TLS)). The resource gateway must share at least one Availability Zone with the broker. Cluster brokers cover multiple Availability Zones, so this is satisfied. For single-instance brokers, verify the Availability Zone overlap.
  3. Create a VPC Lattice resource configuration pointing to your private destination (IP address or DNS name). If you’re connecting to another Amazon MQ broker, specify the destination broker’s endpoint as the custom domain name on the resource configuration, as shown in the following figure.VPC Lattice resource configuration showing the custom domain name field and resource definition populated with the Amazon MQ broker endpointFigure 1: VPC Lattice resource configuration showing the custom domain name field and resource definition populated with the Amazon MQ broker endpoint.
  4. Add the resource configuration to a RAM resource share. The resource share must allow external principals, as shown in the following figure.RAM resource share configuration with the Allow external principals option selectedFigure 2: RAM resource share configuration with the Allow external principals option selected.
  5. Associate the resource share with your broker by editing the broker and adding the resource share. You can also do this using the update-broker command with the AWS CLI. You must pass the entire list of resource share ARNs you want on the broker. This is a put operation, not an add or remove operation.
    aws mq update-broker \
      --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \
      --resource-share-arns arn:aws:ram:us-east-1:111122223333:resource-share/a1b2c3d4-5678-90ab-cdef-EXAMPLE22222

    The associated RAM resource share appears as shown in the following figure.

    Network settings view with associated RAM resource shares

    Figure 3: Network settings view with associated RAM resource shares.

    Select the resource share in the Associated RAM resource shares section. The network status of each shared resource is displayed in the Shared resources section, as shown in the following figure.

    RAM resource share selection showing the network status of each shared resource

    Figure 4: RAM resource share selection showing the network status of each shared resource.

  6. Reboot the broker from the AWS Management Console or the AWS CLI to create the network path:
    aws mq reboot-broker --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

  7. Retrieve the DNS names for your RabbitMQ configuration. This operation also surfaces issues encountered during setup:
    aws mq describe-shared-resources --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

  8. Use the DNS name returned in the output in your Shovel, Federation, or identity provider configuration. Adding new resource configurations to an existing RAM resource share does not automatically update the broker. You must call update-broker and reboot the broker for the new resource configurations to take effect.

Cleaning up

Private Networking uses VPC Lattice and PrivateLink resources that incur ongoing charges. If you no longer need the private connection:

  1. Call update-broker with the resource share removed from the list (or an empty list to remove all), then reboot the broker.
  2. After the broker reboot completes and the resources are no longer in use, delete the VPC Lattice resource configuration and resource gateway.
  3. Optionally, remove the Amazon MQ account principal from the RAM resource share. This principal may still be in use if other brokers are associated with the same resource share, so only remove it if no other brokers depend on it.
  4. If you created a new Amazon MQ for RabbitMQ broker for this walkthrough and no longer need it, delete the broker from the Amazon MQ console or with the delete-broker command.

Operational behavior: Resource access and reboots

Removing a VPC Lattice resource configuration from a RAM resource share while the broker is actively using it revokes access immediately, with no reboot required. Removing a principal from a RAM resource share has the same effect: brokers associated through that principal lose access to the resources in the share immediately. These are intentional security behaviors managed by RAM and VPC Lattice.

Adding new resource configurations to an existing resource share does not take effect automatically. You must call update-broker and reboot the broker for the new resource configurations to take effect. This is by design. It helps verify that changes to a resource share only reach the broker when someone with broker management permissions explicitly triggers the update, providing clear security separation between share management and broker management.

Private Networking is available for Amazon MQ for RabbitMQ brokers in all the AWS Regions where Amazon VPC Lattice is available. Amazon MQ for ActiveMQ brokers do not support this feature.

Pricing

Private Networking uses Amazon VPC Lattice and AWS PrivateLink. Data processing and data transfer charges apply to traffic sent through the private connection. There is an Amazon MQ pricing of $0.01 per GB of data processed through the resource endpoint. For details, see the Amazon MQ pricing page, VPC Lattice pricing page and AWS PrivateLink pricing page.

Conclusion

In this post, we explained how Private Networking for Amazon MQ for RabbitMQ works and walked through the setup process. Whether you’re securing a private identity provider, federating messages between brokers, or connecting to self-hosted RabbitMQ, your broker can now reach private destinations without exposing them publicly.

To learn more, see the Amazon MQ Private Networking documentation.

If you have questions or feedback, leave a comment on this post.


About the authors

Jean-Sébastien Dominique

Jean-Sébastien Dominique

Jean-Sébastien is a Software Development Engineer at Amazon Web Services with 20 years of experience across a wide range of software development domains. He’s interested in the intersection of systems design, human factors, and AI – how people and complex systems interact in practice.

Ishita Chakraborty

Ishita Chakraborty

Ishita is a Senior Technical Account Manager at Amazon Web Services with expertise in serverless and messaging architectures. She works with enterprise customers to deliver technical solutions and strategic guidance – from infrastructure optimization to AI/ML adoption.

Spring 2026 SOC 1 and 2 reports are now available in OSCAL format

Post Syndicated from Thomas Fischer original https://aws.amazon.com/blogs/security/spring-2026-soc-1-and-2-reports-are-now-available-in-oscal-format/

Amazon Web Services (AWS) is excited to release the Spring 2026 System and Organization Controls (SOC) 1 and 2 reports in machine-readable OSCAL format alongside the PDF version of the reports. The reports cover 188 services over the 12-month period from April 1, 2025 to March 31, 2026, giving customers a full year of assurance. These reports demonstrate our continuous commitment to adhering to the heightened expectations of cloud service providers.

AWS is the first major cloud provider to offer key compliance reports to customers in the National Institute of Standards and Technology’s (NIST) Open Security Controls Assessment Language (OSCAL), as of June 2026. OSCAL is an open source, machine-readable (JSON) format for security information. The SOC 1 and SOC 2 report package in OSCAL format is now available as a distinct package in AWS Artifact, marking a milestone toward open, standards-based compliance automation. This machine-readable version of the SOC report package enables workflow automation to reduce manual processing time and modernize security and compliance processes. Your use cases for this content are innovative, and we want to hear about them through the contact information found in the OSCAL report package.

You can download the Spring 2026 SOC 1 and 2 reports in OSCAL format through AWS Artifact, a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact. The SOC 3 report can be found on the AWS SOC Compliance page and in AWS Artifact.

AWS strives to continuously bring services into the scope of its compliance programs to help customers meet their architectural and regulatory needs. You can view the current list of services in scope on our Services in Scope page. As an AWS customer, you can reach out to your AWS account team if you have any questions or feedback about SOC compliance.

To learn more about AWS compliance and security programs, see AWS Compliance Programs.

If you have feedback about this post, submit comments in the Comments section below.

Thomas Fischer

Thomas Fischer

Thomas is a Principal at AWS, focused on scaling product delivery and applications to transform security, risk, and compliance. He has over 20 years of experience in enterprise IT transformation and worked for different consulting companies managing large teams and programs across multiple regulated industries and sectors. Thomas holds CISSP, CCSP, CGEIT, and PMP certifications.

Tushar-Jain

Tushar Jain

Tushar is a Compliance Program Manager at AWS where he leads multiple security and privacy initiatives. Tushar holds a Master of Business Administration from Indian Institute of Management Shillong, India and a Bachelor of Technology in electronics and telecommunication engineering from Marathwada University, India. He has over 14 years of experience in information security and holds CISM, CCSK and CSXF certifications.

Fritz Kunstler

Fritz is a Principal Security Engineer at AWS, currently focused on AI applications to transform security governance, risk, and compliance. Fritz has been an AWS customer since 2008 and an Amazonian since 2016.

Baj Bajwa

Baj Bajwa

Baj is a Security Assurance Manager at AWS, where he leads the Global Third-Party Assurance product portfolio within the Compliance and Security Assurance (CSA) organization. He has over 15 years of experience in information security, compliance, and risk management, and holds a master’s degree in cybersecurity. Baj maintains CISSP, CISA, PMP, CCSK, GISF, and ICAgile certifications.

Top announcements of the AWS Summit in New York, 2026

Post Syndicated from AWS News Blog Team original https://aws.amazon.com/blogs/aws/top-announcements-of-the-aws-summit-in-new-york-2026/

Today at the AWS Summit in New York City, Swami Sivasubramanian, AWS VP of Agentic AI, provided the day’s keynote. Here’s our roundup of the biggest announcements from the event:

New in Amazon Bedrock AgentCore
We’re introducing new capabilities on Amazon Bedrock AgentCore: connecting AI agents to organizational, web, and paid knowledge, helping teams find and fix what’s going wrong in production, and enforcing controls that scale as agents grow more capable.

Together, these capabilities help you build more capable agents faster, govern those agents with controls that scale, and improve them continuously. To learn more, read our blog post covering all the new features.

New in AI-based security tools

New in building AI-based applications 

  • Introducing Kiro for iOS — Kiro introduces a native iOS app, available in a gated preview, built for real engineering work that gives developers a new surface to kick off, monitor, steer, and interact with their Kiro sessions directly from their phone. That means you can now start sessions, check back when they’re done, review diffs, and approve changes all while staying connected to your work with no laptop running.
  • AWS DevOps Agent adds release management capabilities to assess code changes before production — You can use a new release readiness review of code changes and autonomous release testing. These new features verify every change against the natural language standards you give to the DevOps Agent and run change-specific tests in production-like environments.
  • Proactively reduce tech debt autonomously with AWS Transform – continuous modernization — You can use continuous analysis (preview) to automatically scan your code repositories against configurable baselines and generates findings in hours, not weeks. Once you’ve identified and prioritized findings, you can configure autonomous remediations that generate pull requests for affected repositories automatically.

In addition to the keynote announcements, we have other important launches this week:

Introducing AWS Continuum: Security at machine speed

Post Syndicated from Chet Kapoor original https://aws.amazon.com/blogs/security/introducing-aws-continuum-security-at-machine-speed/

What we believe

We’ve been thinking deeply about enterprise security. The operating model that served us for the past decade (collect telemetry, store it, query it, build dashboards to watch it) is no longer keeping pace. We need to shift to the new world: telemetry, context, reasoning, and actions. An approach that produces outcomes. The latest cybersecurity frontier models further made this shift urgent. Models like Claude Mythos can now find software vulnerabilities and reason through complex attack paths at machine-speed, leading to an exponentially increasing backlog of vulnerabilities.

Introducing AWS Continuum for code vulnerabilities

Today, we’re announcing AWS Continuum for code vulnerabilities, now available in gated preview. Continuum for code vulnerabilities addresses the full lifecycle of a code vulnerability at machine speed: from discovery through actions. It reasons over your environment, confirms what is real, and drives toward resolution. It’s model agnostic, using multiple frontier models where each performs best, and is built to incorporate the latest and most capable models as they emerge.

Continuum is built on lessons learned from running security across AWS and Amazon.com. Securing businesses that operate in different industries required a system that understands business context rather than applying generic rules uniformly.

How it works

Continuum for code vulnerabilities reasons over your full environment. This context includes structured data already living in Amazon Web Service (AWS) (your infrastructure, permissions, network topology, code) and the unstructured data that captures how your organization operates and your risk profile (your documents, communications, business priorities).

Continuum for code vulnerabilities operates in four continuous phases.

  1. Discovery: Security teams tackle a backlog of vulnerabilities, and many are already using frontier models to find more. Continuum starts by ingesting that existing backlog and performing its own vulnerability scan of your environment. This creates a more comprehensive view of vulnerabilities and the associated attack paths.
  2. Prioritization: Continuum uses context to evaluate, enrich, and prioritize every finding. Is the affected component deployed, is it reachable, is it in a production path, and what would the business impact be if exploited? The result is an evidence-backed list of priorities, allowing Continuum and your team to focus on what’s most important.
  3. Validation: Continuum validates findings to surface false positives before they waste your team’s time. It contextualizes vulnerabilities against your environment. It then constructs working exploit examples in a sandboxed environment that provide concrete, reproducible evidence of the issue.
  4. Mitigation and remediation: Continuum assesses existing defenses around a validated issue, including blocking and compensating controls along with detection mechanisms. It then draws on its understanding of the codebase, context, and findings to recommend mitigation or remediation of the vulnerability with a network change, policy change, or code patch. The patch recommendation is validated using the same system that confirmed the vulnerability. It also provides blast radius visibility and rollback paths where feasible.

This is just the beginning. We’re starting with code (1st and 3rd party) and then expanding to other aspects of security.

Trust is graduated

Continuum starts in learn mode with a human in the loop. Every recommendation includes the reasoning behind it. As you gain confidence, you can graduate Continuum to enforce mode, enabling remediation that can be increasingly automated based on categories and risk profiles you define.

Continuum capabilities

In addition to Continuum for code vulnerabilities, Continuum includes capabilities you might already know. The AWS Security Agent penetration testing and code scanning functionality is now part of Continuum as Continuum pen testing and Continuum code scanning (Preview). We’re also launching Continuum threat modeling in preview, which automatically generates comprehensive threat models from design documents or source code and outputs results in STRIDE format. These capabilities serve as detection and analysis sources that feed into the broader Continuum loop of discovery, prioritization, validation, and remediation.

Getting started

We’re working with customers across financial services, automotive, and technology to shape AWS Continuum. Customer feedback confirms the direction: security teams want tools that earn trust and take action.

AWS Continuum for code vulnerabilities is available in gated preview. Sign up to request access at AWS Continuum.

If you have feedback about this post, submit comments in the Comments section below.


Chet Kapoor

Chet Kapoor

Chet is Vice President of Search, Security, and Observability at Amazon Web Services. With more than two decades in enterprise technology, he has led companies through some of the industry’s most consequential platform shifts — from APIs and open source to cloud and AI — building and scaling businesses through periods of rapid growth, transformation, acquisition, and IPO. He brings a builder’s mindset, deep operational experience, and a strong customer orientation to helping organizations adopt emerging technologies securely and at scale.

Introducing Amazon Bedrock Managed Knowledge Base for faster, more accurate enterprise AI applications

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/introducing-amazon-bedrock-managed-knowledge-base-for-faster-more-accurate-enterprise-ai-applications/

Today, we’re announcing Amazon Bedrock Managed Knowledge Base, a new set of capabilities that enables developers to build enterprise-grade generative AI applications with their proprietary data in minutes. Organizations building agentic AI applications need secure, reliable, and up-to-date access to enterprise-wide data to deliver accurate, fast, and trusted outcomes. Managed Knowledge Base abstracts away the complexity of building and managing retrieval-augmented generation (RAG) pipelines, allowing developers to focus on business outcomes rather than infrastructure management.

Developers building knowledge bases for their agents face three key challenges today:

  • Connecting to enterprise data – Enterprise knowledge lives across disparate systems with different content types, access control lists, and document formats. Building and maintaining custom connectors for each source adds complexity that slows down development.
  • Optimizing RAG accuracy – Best practices for retrieval-augmented generation keep evolving. Developers need to experiment with different parsing strategies, chunking approaches, embedding models, and agentic retrieval behaviors to get accurate answers from their data.
  • Managing infrastructure at scale – Organizations need to serve large knowledge bases with millions of documents, or manage thousands of smaller knowledge bases across teams. Both patterns require reliable infrastructure, security enforcement, and cost control.

These challenges require developers to repeatedly perform undifferentiated work instead of focusing on their applications.

Amazon Bedrock Managed Knowledge Base addresses these challenges by abstracting away the multiple infrastructure components developers traditionally have to assemble and maintain themselves (storage, retrieval, embeddings, re-ranking, and foundation model selection) into a single managed primitive. By default, the service automatically selects and manages a default embeddings model, re-ranker model, and foundational model on your behalf, so you can get up to speed quickly without needing to pick or maintain one yourself. On top of this managed foundation, three core innovations further improve ease of use and accuracy:

  • Native data connectors – Six pre-built ingestion connectors that natively pull enterprise data and permissions from SaaS applications, eliminating the overhead developers face in managing application-specific requirements. At launch, we support Amazon S3, SharePoint, Confluence, Web Crawler, Google Drive, and OneDrive.
  • Smart Parsing – Different content types and sources require different approaches to achieve accurate retrieval. Smart Parsing handles this complexity automatically, selecting the right parsing strategy for each data type and connector to provide the highest accuracy for your agents.
  • Agentic Retriever – Optimized for complex queries that require multiturn, multihop retrieval within a single knowledge base or across multiple knowledge bases. Agentic Retriever automatically infers end-user intent and draws relevant context from institutional knowledge spread across data sources and modalities.

With just a few lines of code, Amazon Bedrock Managed Knowledge Base automatically manages and scales the end-to-end RAG pipeline that powers your enterprise knowledge agents. For agent builders, it’s available as a pre-built target type in Amazon Bedrock AgentCore Gateway, reducing integration to a few lines of code, auto-generating role-based permissions, and providing observability and evaluation metrics in the AgentCore Observability dashboard.

Getting started with Amazon Bedrock Managed Knowledge Base
Creating a Managed Knowledge Base is straightforward. Navigate to the Amazon Bedrock AgentCore console or the Amazon Bedrock console, open the Knowledge Bases page, and choose Create Managed KB. The experience is the same in both consoles. You will see that Unstructured Vector Store KB is now available as the recommended option, alongside the other knowledge base types you may already be familiar with:

Picture 1 – Knowledge Bases list page in the Amazon Bedrock AgentCore console showing the Type column with different KB types and the Create Managed KB button

When creating a new Knowledge Bases, you can connect to your enterprise data sources by choosing from the list of supported connectors directly from a dropdown. AWS Identity and Access Management (IAM) roles are automatically created, and you can choose to edit these permissions if needed:

Picture 2 – Create Knowledge Base page showing the Data source dropdown expanded with all supported connectors: Amazon S3, Confluence, Custom, Google Drive, One Drive, SharePoint, and Web Crawler

An optimized set of defaults will be presented, allowing you to create your knowledge base in just a few clicks. Once the data is synced, you can integrate the knowledge base with your agent or provide it as a tool for your foundation model and start querying.

Smart Parsing for accurate data ingestion
One of the key challenges in building knowledge bases is preparing diverse data types for accurate retrieval. Once you point Managed Knowledge Base at your data sources, Smart Parsing automatically determines the optimal parsing strategy for each data type and connector, no extra configuration is required.

Smart Parsing combines multiple techniques:

  • Connector-specific data models – Optimized handling for each data source. For example, the Web Crawler connector preserves HTML structure including embedded images and tables, ensuring rich content is not dropped during ingestion. SharePoint connectors maintain document hierarchy and relationships between files.
  • Multimodal processing – Automatic detection and processing of different content types within documents. The system identifies bounding boxes in documents, then sends them to foundation models for data extraction, captioning, and scene description in video files.
  • Optimized chunking – Smart Parsing leverages foundation models to understand document structure and extract meaningful content, ensuring that complex documents with mixed formats are properly indexed. Intelligent defaults balance retrieval accuracy with performance based on document type and content structure, while advanced users can customize chunking strategies when needed.

This automated approach eliminates weeks of experimentation typically required to achieve production-quality retrieval accuracy, while still preserving the flexibility to customize when needed.

Using Agentic Retriever for complex queries
After your data is ingested, you can start querying your knowledge base. Generative AI applications often struggle with complex user queries that require reasoning, recursive multi-step retrieval, and intermediate evaluations of results. Consider a user asking two related questions: “What is the cloud infrastructure budget for the ML platform team?” and “Does our expense policy allow prepaying annual commitments?” A single retrieval step might surface documents about the ML platform team but fail to connect the budget information with the expense policy needed to fully answer the question.

Picture 3 – Agentic Retriever decomposes complex user queries into a step-by-step plan, performing multi-hop retrieval across multiple knowledge bases and combining results to deliver accurate, grounded responses

Agentic Retriever solves this by creating a step-by-step query plan: 1. Which team owns the ML platform, and what is their cloud infrastructure budget? 2. What does the expense policy say about prepaying annual commitments? 3. Does the policy allow the ML platform team to prepay against this budget?

The system performs multi-hop retrieval and reasoning at each step, and once it has gathered sufficient relevant passages, it stops the search process and returns the top results. By abstracting away the complexity of building a separate multi-hop reasoning pipeline, this approach dramatically improves accuracy for complex queries while letting developers focus on their agentic search applications instead of orchestration logic.

You can try Agentic Retriever directly from the test panel of your knowledge base in the Amazon Bedrock AgentCore console. Select Agentic retrieval only as the retrieval type to let the system automatically plan and execute multi-step queries across your knowledge bases:

Picture 4 – Test Knowledge Base panel showing Agentic retrieval with answer generation selected as the retrieval type, with model selection and maximum agentic iterations options

Enabling MCP with Bedrock AgentCore
Amazon Bedrock Managed Knowledge Base seamlessly integrates with AgentCore Gateway as a native target type. This integration eliminates the need for manual integration and provides built-in observability, policy enforcement, and automatic permission management.

You can navigate to the Amazon Bedrock AgentCore console or SDK and create an AgentCore Gateway or select an existing one. When adding targets to your gateway, you will find Knowledge Base as a new pre-built target type alongside other options such as MCP server, Lambda ARN, REST API, and other integrations. Simply select your knowledge base ID to expose it through the gateway:

Picture 5 – Add targets page in AgentCore Gateway showing Knowledge Base as a new pre-built target type, with the knowledge base ID selector and runtime retrieval mode options

Add targets page in AgentCore Gateway showing Knowledge Base as a new pre-built target type, with the knowledge base ID selector and runtime retrieval mode options

Gateway exposes the standard Model Context Protocol (MCP), so the knowledge base tools are automatically discovered by clients from any MCP-compatible framework, including Strands Agents, LangChain, CrewAI, LlamaIndex, and LangGraph. No custom integration code is required.

Model choice and flexibility
Amazon Bedrock Managed Knowledge Base preserves the flexibility developers expect from Amazon Bedrock. Every foundation model available on Bedrock can power the generation step, and developers can select from different embedding and re-ranking models to optimize retrieval for their specific use case, enabling teams to fine-tune accuracy and cost-performance without changing infrastructure.

Unlike managed solutions that lock you into specific model providers, Amazon Bedrock Managed Knowledge Base separates the infrastructure management (connectors, parsing, storage, retrieval orchestration) from model selection. This means you can:

  • Take advantage of the latest models – Adopt the latest embedding, re-ranking, and foundation models as they become available to improve accuracy, latency, and cost for your application without rebuilding your RAG pipeline.
  • Optimize for price-performance – Choose smaller, faster models for simple queries and more capable models for complex reasoning tasks, all using the same knowledge base infrastructure.
  • Use Bedrock embedding models – While Smart Parsing provides optimized defaults, you can configure Bedrock embedding models when your domain requires specialized semantic understanding.
  • Maintain consistency with existing applications – If you’re already using Bedrock Knowledge Bases APIs (Retrieve, StartIngest, StopIngest, IngestKnowledgeBaseDocuments), Managed Knowledge Base uses the same APIs, so migration requires no code changes, just point to the new knowledge base ID.

This approach ensures you can spend time on your generative AI application without losing the ability to change models based on evolving requirements or new model capabilities.

Get started today
Amazon Bedrock Managed Knowledge Base is available today in the US East (N. Virginia), US West (Oregon), Asia Pacific (Sydney, Tokyo), Europe (Dublin, Frankfurt, London), and AWS GovCloud (US-West) Regions. For Regional availability and future roadmap, visit AWS Capabilities by Region.

With Bedrock Managed Knowledge Base, you pay for what you use with no upfront commitments. Pricing is based on two dimensions: the size of indexed data stored and the number of retrievals performed (on-demand). For detailed pricing information, visit the Amazon Bedrock pricing page. Bedrock is also a part of the AWS Free Tier that new AWS customers can use to get started at no cost and explore key AWS services.

These capabilities work with any open source framework such as CrewAI, LangGraph, LlamaIndex, and Strands Agents, and with any foundation model. Bedrock services can be used together or independently, and you can get started using your favorite AI-assisted development environment with the AgentCore open source MCP server.

To learn more and get started quickly, visit the Bedrock Knowledge Bases Developer Guide.

Daniel Abib