It’s only been a few weeks since Anthropic announced the Claude Mythos Preview model and launched Project Glasswing with AWS and other leading organizations. This has generated a lot of discussion about the future of cybersecurity and what the ever-increasing capabilities of foundation models mean to organizations.
As AWS CISO Amy Herzog pointed out in the Project Glasswing announcement, “At AWS, we build defenses before threats emerge, from our custom silicon up through the technology stack. Security isn’t a phase for us; it’s continuous and embedded in everything we do.”
While the discussion around the future of cybersecurity is important, the only thing we know for certain is that organizations need to be able to react quickly to the rapid changes AI is bringing to technology and business in general. And you can’t react quickly if your security fundamentals aren’t dialed in.
The security hygiene gap
It’s easy to assume you have the foundational security elements covered, or to overlook some completely. Basic security use cases like identity management, threat detection, vulnerability management, data protection, and network security can be inconsistently implemented across cloud environments. While AI is reshaping the security landscape, strong security fundamentals continue to be essential for every organization, regardless of size or industry.
These are the security basics that matter whether or not you’re adopting AI: patching consistently, enforcing least-privilege access, enabling logging and monitoring, encrypting data at rest and in transit, and reviewing security configurations regularly. When these fundamentals are in place, you’re better positioned to take advantage of AI-driven tools and respond to newly discovered vulnerabilities, wherever they come from.
While the concepts that drive security fundamentals are universal, implementing them in your environment is best done with an understanding of the context unique to your organization. That’s why we have a multitude of freely available materials—like the AWS Well-Architected Framework—that you can use to help ask the right questions and implement changes in your environment. We also offer programs like the Security Health Improvement Program (SHIP) to help you improve your security posture through prescriptive guidance and continuous improvement.
What is the Security Health Improvement Program (SHIP)?
SHIP is a no-cost program available to every AWS customer, regardless of support tier. SHIP provides a proven, data-driven methodology to:
Assess your current security posture using data from your AWS environment
Identify specific opportunities to improve across 10 core security use cases
Build a prioritized action plan tailored to your environment
Establish a mechanism for continuous security improvement
The program is led by AWS Solutions Architects and Technical Account Managers who take you through a personalized report, contextualize findings for your environment, and help you build a prioritized action plan.
Why SHIP matters in the AI era
Project Glasswing highlights an important shift: AI-powered tools are accelerating the pace of vulnerability discovery, which means organizations need to be prepared to assess and respond to findings and changing situations faster than before. In addition to external factors, as organizations adopt AI—whether deploying foundation models, building agentic workflows, or using AI-powered services—how they implement their security controls must change as well. A strong security foundation is what makes confident AI adoption possible.
Here’s how SHIP helps:
Address foundational security gaps proactively
SHIP uses a data-driven methodology to identify opportunities to improve and optimize across 10 core security use cases: threat detection, cloud security posture management, application security testing, configuration management, access governance, vulnerability management, application protection, network security, encryption, and secrets management. The program includes a SHIP assessment to identify critical security findings related to your current security posture, so your team can build a prioritized roadmap for improvement tailored to your environment.
Establish the security baseline AI workloads require
Before you deploy your first model on Amazon Bedrock or build agentic workflows with Amazon Bedrock AgentCore, you need confidence that your underlying infrastructure follows security best practices. SHIP uses actual data from your environment to provide prescriptive, specific guidance rather than generic security recommendations. This is especially relevant as AI-driven vulnerability discovery tools become more widely available: organizations with strong baselines will be able to act on new findings quickly and effectively.
Build a mechanism for continuous security improvement
As AI capabilities evolve, organizations benefit from having a repeatable process to assess and strengthen their security posture over time. SHIP establishes the methodology and mechanisms for your team to continuously assess, prioritize, and improve. By building this operational capability, you’re strengthening your organization’s ability to adapt and contributing to broader industry resilience. As the cybersecurity community integrates AI into defense strategies, SHIP helps you maintain foundational best practices so you can adopt these innovations effectively and with confidence.
Getting started is straightforward
SHIP is available today, at no cost, to every AWS customer. Here’s how to get started:
Talk to your AWS account team. Ask about scheduling a SHIP engagement, or request one directly on the SHIP page.
Attend a SHIPActivation Day. AWS regularly hosts hands-on workshops where you can run the SHIP assessment with AWS Solutions Architects and start building your improvement plan.
Explore the prescriptive guidance. Consult the AWS Well-Architected Framework – Security Lens for documentation, reference architectures, and implementation guides you can start using today.
Take the next step together
AWS is committed to being the most secure cloud, from our participation in Project Glasswing to the security embedded in every layer of our infrastructure. Security is a shared responsibility, and programs like SHIP give customers the tools, guidance, and support to strengthen their security foundations so they can build confidently, no matter what comes next.
Ready to improve your security posture? Contact your AWS account team to schedule a SHIP engagement, or visit the SHIP resources page to learn more.
The AWS Customer Incident Response Team (AWS CIRT) regularly encounters patterns that repeat across their engagements when helping customers respond to security incidents. We’re passionate about making sure that information is widely accessible so that everyone can improve their security posture and their organization’s resilience to disruption. The primary method we use to share this information is the Threat Technique Catalog for AWS (TTC). The latest update to the catalog for March 2026 addresses identity, persistence, infrastructure destruction, and privilege escalation. Each new entry reflects something we’ve encountered in practice, and each provides straightforward mitigations. This post breaks down what changed, why it matters, and what you can do about it today.
What we’re seeing
Based on recent observations, we’ve added three new entries to the TTC.
Cognito refresh token abuse: The quiet persistence mechanism
Amazon Cognito refresh tokens are designed for convenience. They let applications obtain new access and ID tokens without requiring users to re-authenticate. The default lifetime is 30 days and is configurable up to 10 years. Cognito provides the flexibility to address a wide range of use cases, however the AWS CIRT has seen this lifetime window used by threat actors in an unauthorized way to maintain persistence by refreshing credentials.
When a threat actor obtains a valid refresh token—through credential theft, compromised client-side storage, or elevated permissions—they can call cognito-idp:GetTokensFromRefreshToken to silently generate fresh tokens. The legitimate user’s session continues normally because their application independently refreshes tokens as needed—the threat actor’s refresh calls don’t invalidate the user’s token. This creates a parallel, persistent foothold that’s invisible to the user. In environments where refresh token rotation isn’t enabled, the same token can be reused indefinitely within its validity window.
This method of gaining persistent access is often overlooked by response teams who were confident that the initial compromise was contained, only to discover ongoing unauthorized access weeks later through a refresh token they didn’t know existed.
Enabling refresh token rotation and reducing the lifetime of tokens can help mitigate this risk. Dive deeper in the TTC (T1098.A006).
AMI image deletion: Targeting recovery capabilities
Amazon Machine Images (AMI) are a core part of many solutions and foundational to disaster recovery. They often contain the operating system, application configurations, and everything needed to rebuild your infrastructure. Threat actors know this, and we’re seeing ec2:DeregisterImage used to make it more difficult to recover from an incident.
By default, when an AMI is deregistered, it’s gone. Recycle Bin retention rules can allow the recovery of the AMI, but if you haven’t explicitly enabled that functionality, there’s no way to undo the deregister action. Working with customers, we’ve seen cases where the impact of this action goes beyond the immediate loss because the threat actors have also removed the golden images the teams planned to restore from.
The TTC has more information about how to detect and mitigate this technique, including how to enable Recycle Bin retention rules for key AMIs (T1485.A002).
Additional cloud roles: The trust policy blind spot
We’ve updated T1098.003: Additional Cloud Roles to now include UpdateAssumeRolePolicy as a tracked API call. We’ve seen an increase in the use of this call to avoid detections set to flag new role creation (iam:CreateRole). By modifying the trust policy of an existing role, a threat actor with sufficient permissions can use UpdateAssumeRolePolicy to subtly add an external account or an identity they control. No new roles appear. No new policies are created. The existing role simply trusts a new principal which the threat actor can assume.
This persistence and privilege escalation technique blends into the volume of normal AWS Identity and Access Management (IAM) operations. It’s especially effective in environments with a large number of roles where trust policy changes aren’t actively monitored.
The current trend
A common thread runs through all three of these updates: threat actors are using subtle, default, or unexpected behaviors to sidestep detection. Refresh tokens working as designed. AMI deregistration completing without guardrails. Trust policies being modified through legitimate API calls. These actions might not trigger alarms in most environments because they look like normal operations.
This is a shift worth paying attention to. Rather than relying on novel exploits or zero-days, the techniques we’re cataloging reflect threat actors who understand how cloud services work and use that knowledge to hide in plain sight. The implication for security teams is clear: prevention and detection strategies need to mature beyond monitoring for obviously malicious actions. Customers need to be watching for legitimate actions happening in illegitimate context—such as the right API call, made by the wrong principal, at the wrong time.
The Threat Technique Catalogue for AWS is designed to help with exactly this. Each technique entry includes detection guidance and mitigations specific to AWS environments. We encourage teams to review the relevant entries and assess whether their current monitoring would catch these patterns:
T1098.A006: Cognito Refresh Token Abuse: Are you monitoring for cognito-idp:GetTokensFromRefreshToken from unexpected sources? Is refresh token rotation enabled?
T1485.A002: AMI Image Deletion: Do you have Recycle Bin retention rules protecting your critical AMIs? Would you know if a production AMI was deregistered outside a maintenance window?
T1098.003: Additional Cloud Roles: Are trust policy modifications tracked and alerted on? Could an external account be added to an existing role without anyone noticing?
Each of these techniques leaves traces in AWS CloudTrail, and the TTC provides specific guidance on what to watch for and how to respond.
Looking ahead
The Threat Technique Catalog for AWS exists because we believe the patterns we observe during security engagements shouldn’t stay behind closed doors. When we see techniques repeating across customers, the most effective thing we can do is document them and make that knowledge available so you can act on it before you’re in the middle of an incident.
This March update adds three new entries, and the catalog will continue to evolve. Our team regularly updates it based on what we’re seeing in the real world when helping customers respond to security events. We encourage security teams to review the catalog regularly, incorporate its techniques into threat modeling exercises, and use it as a shared vocabulary for discussing cloud-specific threats.
Understanding what AWS Identity and Access Management (IAM) policies can control helps you build better security controls and avoid spending time on approaches that won’t work. You’ve likely encountered questions like:
Can I use AWS Organizations service control policies (SCPs) to prevent the creation of security groups that allow traffic from 0.0.0.0/0?
Can I block uploads unless objects are encrypted?
Can I prevent functions with more than 512 MB of memory allocated?
Some of these are possible with IAM policies. Others are not. The difference is determined by a fundamental principle of AWS authorization: Policies make decisions based on information available in the authorization context at the time of the API call.
In this blog post, you learn how to use the AWS Service Authorization Reference to determine what’s achievable with IAM policies, recognize scenarios that need alternative solutions, and build more effective security controls in your AWS environment.
Understanding AWS authorization context
When you make an AWS API request through the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDK, the specific AWS service (such as Amazon S3 or Amazon EC2) receiving the request assembles a request context containing information about that request. This context is used for policy evaluation decisions. Request context is structured using the Principal, Action, Resource, Condition (PARC) model, which has four key components.
Principal: Identifies the requester and their attributes (tags, session context)
Action: Specifies the AWS API operation being requested (for example, s3:PutObject, ec2:RunInstances)
Resource: Defines the target AWS resource using Amazon Resource Names (ARNs)
Condition: Provides additional context available at request time, such as IP address, time, encryption parameters, MFA status, and service-specific attributes
The following example shows the typical request context for an Amazon S3 object upload:
IAM policies can evaluate request metadata like encryption method and storage class being specified. However, it cannot evaluate the actual file contents, object size, or specific data patterns. Policy evaluation occurs at the time of the request, using the information present in the authorization context.
An essential resource: The Service Authorization Reference
The Service Authorization Reference is the authoritative documentation for understanding what policies can control. For every AWS service, it documents:
Actions: Every controllable operation
Resources: Resource types that can be targeted
Condition keys: The exact context information available for policy decisions
Condition keys are broadly divided into two categories. Global condition keys, which can be used across AWS services, and service-specific condition keys, which are defined for use with an individual AWS service. Use the Service Authorization Reference to find the global-condition keys or service-specific condition keys for each AWS service.
How to use the Service Authorization Reference
Follow these steps to determine if your requirement can be controlled with IAM policies:
Find the action you want: Find the API operation you want to control. Be precise, different actions have different available condition keys.
Examine available condition keys: The Condition keys column shows what context information AWS makes available for that action.
Make your feasibility determination: If the information you need isn’t listed as a condition key, you will not be able to control it with IAM policies alone.
Let’s take an example from the Amazon Elastic Compute Cloud (Amazon EC2)ec2:RunInstances action to see what you can and can’t control. In the Service Authorization Reference under the Amazon EC2 section, examine the RunInstances action and check the Resource types column. The RunInstances action affects multiple resource types, each with its own set of condition keys.
For the instance* resource type:
ec2:InstanceType: Can restrict instance types
ec2:EbsOptimized: Can require EBS optimization
aws:RequestTag/: Can enforce tagging requirements
For the network-interface* resource type:
ec2:Subnet: Can control subnet placement
ec2:Vpc: Can limit to specific virtual private clouds (VPCs)
ec2:AssociatePublicIpAddress: Can control public IP assignment
Note: These are a few examples from the many condition keys available for each resource type under the RunInstances action. The Service Authorization Reference lists dozens of condition keys across resource types (instance, network interface, security group, subnet, volume, and so on) that RunInstances affects. Consult the complete reference to see the available options for your specific use case.
Access the Service Authorization Reference programmatically
Beyond the human-readable documentation, AWS provides the Service Authorization Reference in machine-readable JSON format to streamline automation of policy management workflows. Use this programmatic access to incorporate authorization metadata into your development and security workflows. For detailed information about the JSON structure and field definitions, see the Simplified AWS service information for programmatic access. Developers can use tools like the IAM MCP Server for AWS IAM operations. This server provides AI assistants with the ability to manage IAM users, roles, policies, and permissions while following security best practices.
Using IAM policies to control specific scenarios
The following examples show how you can use IAM policies to control specific scenarios.
Example 1: Enforce AES256 server-side encryption on S3 objects
In the Amazon S3 Service Authorization Reference, under s3:PutObject action, the s3:x-amz-server-side-encryption condition key is available in the authorization context, which can be used to control the server-side encryption of S3 objects with AES-256. Here is the required policy.
Policy 1: Deny Amazon S3 object upload if the encryption doesn’t use AES-256
Policy 1 is a resource-based policy that can be applied on an S3 bucket to restrict object uploads. It denies a PutObject request when the server-side encryption isn’t using the AES-256 encryption algorithm.
Example 2: Allow different instance types based on the user’s cost center tag.
When checking the Amazon EC2 Service Authorization Reference for ec2:RunInstances, the ec2:InstanceType condition key, which is resource specific, is available. To restrict instance types based on who is launching them (rather than just what is being launched), you can either combine this with a global condition key or attach different policies to different principals. By using aws:PrincipalTag/tag-key alongside ec2:InstanceType, you can identify the user’s cost center from their IAM identity tags and then apply different instance type restrictions accordingly. This allows a single policy to dynamically enforce different permissions based on the requester’s identity.
Policy 2: Restricting EC2 instance types by cost center
This is an identity-based policy that you can attach to IAM users, groups, or roles to control EC2 instance launches based on cost allocation. In the first statement, aws:PrincipalTag, which is a global condition key (tags attached to the IAM user or role), is used to determine which instance types are allowed. Users tagged with CostCenter=Development can only launch cost-effective T3 instance types (t3.micro, t3.small, t3.medium, and so on)with the service specific key ec2:InstanceType.
In the second statement, users tagged with CostCenter=Production can launch more powerful instance types from the M5 (general purpose), C5 (compute optimized), and R5 (memory optimized) families. This approach lets organizations enforce cost controls and allocate resources based on workload requirements. Each cost center maintains flexibility for its specific needs.
Note: Additional resources are required in the IAM policy to successfully launch EC2 instances. For the complete list, see Launch Instances.
Example 3: Users can only access and update DynamoDB items where the partition key matches their username.
You have identified that GetItem, PutItem,and UpdateItem actions are required. Corresponding to these actions, you can use the condition key to expose partition key values in the authorization context as described in the Amazon DynamoDB Service Authorization Reference
The policy allows users to perform read and write actions (GetItem, PutItem, and UpdateItem) on the UserProfiles table, but only for items where the partition key value equals their own username (using the ${aws:username} policy variable). For example, if user alice attempts to access an item with partition key bob, the request will be denied.
Scenarios that need more than policies alone
Some requirements can’t be met using IAM policies. Here are three common scenarios that aren’t achievable with IAM policies alone.
Scenario 1: Block users from creating security group rules that allow traffic from 0.0.0.0/0 on TCP port 22
Upon checking the Amazon EC2 Service Authorization Reference, you will find that the ec2:AuthorizeSecurityGroupIngress action is required in an IAM policy to add an inbound access rules to a security group.
To verify this in the Service Authorization Reference, navigate to the Amazon EC2 Service Authorization Reference and search for the AuthorizeSecurityGroupIngress action, which is the action that creates security group rules. After you locate this action, review the Condition keys column and look for condition keys related to CIDR blocks, IP ranges, ports, or protocols. Available condition keys for ec2:AuthorizeSecurityGroupIngress include:
Notice there are no condition keys for CIDR blocks (such as 0.0.0.0/0), port numbers (such as 22), or protocols (such as TCP). The authorization context doesn’t include information about the specific CIDR blocks, ports, or protocols being added to the security group rule, so IAM policies can’t control these attributes.
Scenario 2: Prevent creation of Lambda functions with more than 512 MB of memory allocated
Following the same verification methodology described in Scenario 1, navigate to the AWS Lambda Service Authorization Reference and examine the CreateFunction action’s condition keys for the function* resource type.
Available condition keys for lambda:CreateFunction with the function* resource type include:
lambda:CodeSigningConfigArn: Filters access by the ARN of the code signing
configuration-lambda:Layer: Filters access by the ARN of a version of an AWS Lambda layer
lambda:VpcIds: Filters access by the ID of the VPC configured for the Lambda function
There is no condition key for memory allocation (MemorySize parameter), timeout settings, storage configuration (EphemeralStorage), or runtime selection. Because memory allocation isn’t exposed in the authorization context, IAM policies can’t restrict this parameter.
Keep these principles in mind when working with IAM policies:
Policies control what’s in the authorization context, not all elements you see in API documentation
The Service Authorization Reference is authoritative; if something isn’t listed as a condition key, you can’t control it with policies
Different actions have different available contexts even within the same service
Alternative approaches exist. AWS Config, EventBridge, and service-specific controls can be used to achieve your goals when policies alone can’t
Layered security is essential; combine preventive, detective, and responsive controls to help ensure that your data is secure
Conclusion
In this post, you learned how to use the AWS Service Authorization Reference to determine what’s achievable with IAM policies and recognize scenarios that require alternative solutions. By understanding that policies can only make decisions based on information available in the authorization context, you can build more effective security controls and avoid spending time on approaches that won’t work.
The Service Authorization Reference is your authoritative source for understanding policy capabilities. When you need to implement a control, start there to see if the required condition keys exist. If they don’t, you will need to layer in detective or responsive controls using services like AWS Config, Amazon EventBridge, or AWS Lambda.
Remember that effective AWS security isn’t about finding one perfect control, it’s about combining preventive, detective, and responsive measures to create defense in depth. IAM policies are powerful tools for prevention and work as part of a comprehensive security strategy.
As outlined in the AWS post-quantum cryptography (PQC) migration plan, addressing the risk of harvest now, decrypt later (HNDL) attack is an important part of your post-quantum plan. Upgrading the client-side of your workloads to support quantum-resistant confidentiality is an important aspect of your side of the PQC shared responsibility model. Timelines to plan and execute your PQC upgrades vary by region and by industry and will depend on your own business risk profile. To learn more, see the AWS PQC frequently asked questions.
AWS Secrets Manager uses SSL/TLS to communicate with AWS resources, currently supporting TLS 1.2 and 1.3 in all AWS Regions. The service supports using TLS 1.3 with hybrid post-quantum key exchange for clients that support this capability. The hybrid post-quantum approach establishes TLS connections by combining traditional cryptography (such as X25519) with post-quantum algorithms (ML-KEM), and helps to protect your secrets against both current classical attacks and future quantum computer threats. Regardless of how your workload accesses Secrets Manager, this client-side software upgrade is the only action you need to take to address risk to secrets from HNDL. Your secrets at rest are already encrypted using keys managed by AWS Key Management Service (AWS KMS). Properly implemented symmetric encryption is considered quantum-resistant; asymmetric cryptography faces quantum threats. To learn more, watch AWS re:Inforce 2025 – Post-Quantum Cryptography Demystified.
To reduce builder effort for client-side upgrades, we’re pleased to announce the following Secrets Manager clients now enable and prefer post-quantum TLS when initiating connections to Secrets Manager: Secrets Manager Agent (v2.0.0 or later), the AWS Lambda extension (v19 or later) and the Secrets Manager CSI Driver (v2.0.0 or later). For SDK-based clients, hybrid post-quantum key exchange is available in supported AWS SDKs. Enablement requirements vary by language, version, and operating system. See the following table for your SDK client.
This launch is part of the ongoing commitment AWS has made to migrate systems to post-quantum cryptography and making it straightforward for our customers to do the same. See Post-Quantum Cryptography to learn more.
The following table summarizes the behavior for each client. When the client is upgraded to support hybrid post-quantum key exchange, the Secrets Manager service endpoint automatically selects it during the TLS handshake. Upgrading to the versions listed in the table is the only action you need to take for your workload to begin using hybrid post-quantum key exchange when calling Secrets Manager APIs.
The AWS SDK for Python (boto3) uses the OS-provided OpenSSL for TLS. Hybrid PQ key exchange in TLS requires running on a system with OpenSSL 3.5 or later installed.
The Secrets Manager caching libraries are built on the AWS SDKs and inherit their TLS behavior. Note for Java: The JDBC driver flag and Java Caching flag must be set to enable Hybrid PQ key exchange in TLS.
If you’re using the Secrets Manager Agent, the Lambda extension, or the CSI Driver, upgrade to the listed version to use hybrid post-quantum key exchange in TLS as the default. Customers using the AWS SDK for Rust, Go, or Node.js at the versions listed in the table are already upgraded and no additional action is required. The SDK will select the hybrid post-quantum key exchange for API calls. For customers using the AWS SDK for Python, hybrid post-quantum key exchange in TLS requires OpenSSL 3.5 or later to be present on the host system. Guidance on verifying and enabling this is available in the AWS Secrets Manager documentation. For customers using the AWS SDK for Java v2, hybrid post-quantum key exchange in TLS requires using the AWS CRT HTTP client. The postQuantumTlsEnabled(true) must be set on the CRT client to enable hybrid post-quantum key exchange in TLS.
After your client versions meet the requirements listed in the table, you can verify that your connections are actively using hybrid post-quantum key exchange.
How to verify your connection uses hybrid post-quantum key exchange
With hybrid post-quantum key exchange using ML-KEM now enabled by default for Secrets Manager clients (see the preceding table), most customers will not need ongoing monitoring to verify correct behavior or detect regressions. However, security teams and compliance officers might want to confirm that their Secrets Manager API calls are negotiating the hybrid key exchange. On the server side, you can confirm hybrid post-quantum key exchange in TLS by using AWS CloudTrail. On the client side, you can inspect TLS handshake details using a utility like Wireshark or by using developer tools built into major web browsers.
Verification is a two-step process: first, fetch a secret using your Secrets Manager client to generate a GetSecretValue API call, then confirm in AWS CloudTrail that the call negotiated hybrid post-quantum key exchange.
Fetch your secret using your Secrets Manager client
The following examples show how to retrieve your secret using the Secrets Manager Agent, Lambda extension, and CSI Driver—each of which will automatically negotiate hybrid post-quantum key exchange when calling the GetSecretValue API.
To verify hybrid post-quantum TLS with Secrets Manager Agent on EC2 instance: Install the agent on your Amazon Elastic Compute Cloud (Amazon EC2) instance and use it as a client to fetch your secret.
Use the agent to fetch your secret. curl -H “X-Aws-Parameters-Secrets-Token: $(</tmp/awssmatoken)” localhost:2773/secretsmanager/get?secretId=<YOUR-SECRET-ARN>
Wait for about 5 minutes for CloudTrail to deliver the logs.
To verify hybrid post-quantum TLS with Lambda extension: Use the AWS parameters and Secrets Manager Lambda extension to create a Lambda function that will consume your secrets from Secrets Manager using direct API calls.
Confirm hybrid post-quantum key exchange using CloudTrail
CloudTrail logs include a tlsDetails field for Secrets Manager API calls. When hybrid post-quantum key exchange in TLS is active, the keyExchange field in tlsDetails will show X25519MLKEM768. Each CloudTrail record includes a tlsDetails field that contains the cipher suite and, where available, the key exchange group negotiated during the TLS handshake.
If the keyExchange field shows X25519MLKEM768, then hybrid post-quantum key exchange in TLS is active. If it shows a traditional algorithm such as X25519, the client is not advertising ML-KEM support, and you should check the client version and configuration.
Troubleshooting
If your Secrets Manager API calls aren’t negotiating X25519MLKEM768 after updating your clients, check your SDK version, OpenSSL version (Python), and firewall or proxy configuration as shown in the Client Hybrid Post-Quantum Key Exchange Requirements section near the beginning of this post.
What’s next
This launch is one step in a broader migration. AWS is continuing to roll out ML-KEM support across AWS service HTTPS endpoints as part of Workstream 2 of the AWS PQC Migration Plan, with a target of full coverage across public AWS endpoints.
Support for CRYSTALS-Kyber, the pre-standardization predecessor to ML-KEM, is phasing out across AWS endpoints in 2026. Customers on older SDK versions that advertise only CRYSTALS-Kyber support will fall back gracefully to traditional TLS rather than negotiate the deprecated algorithm. To avoid this fallback, upgrade to the SDK versions listed in this post.
The journey of PQC migration extends beyond confidentiality of data in transit. To stay informed about the latest developments in the AWS PQC journey and your side of shared responsibility, follow the AWS Post-Quantum Cryptography page.
Conclusion
AWS Secrets Manager now enables hybrid post-quantum key exchange using ML-KEM by default to help protect your secrets and support your compliance efforts. This update requires no code changes or configuration updates for customers using the latest client versions.
This post covered how AWS Secrets Manager uses hybrid post-quantum cryptography to secure TLS connections, which clients support this capability, and how to verify that your connections are protected against harvest now, decrypt later attacks.
To benefit from this announcement today:
Upgrade your Secrets Manager client (Agent, Lambda extension, or CSI Driver) to the latest available versions to enable hybrid post-quantum key exchange using ML-KEM
If your workload uses the AWS SDK instead of a caching client, upgrade your AWS SDK and underlying dependencies to the minimum versions listed in this post
Verify hybrid post-quantum key exchange in TLS is active by checking the keyExchange field in CloudTrail tlsDetails for your Secrets Manager API calls
Test end-to-end hybrid post-quantum key exchange TLS connectivity in your environment, including network paths that traverse corporate firewalls or proxies
AWS will continue rolling out post-quantum cryptography support. For information about the broader migration effort, see the AWS PQC Migration Plan. Keep an updated cryptographic inventory of your broader environment to identify other uses of traditional public-key cryptography that will require migration. The CISA Quantum-Readiness guidance and the AWS PQC Migration Plan are good starting points.
Regulators worldwide require financial institutions to implement Know Your Customer (KYC) processes that help prevent money laundering, terrorist financing, fraud, and identity theft. KYC has evolved from a compliance checkbox to a core security function for financial institutions. Financial institutions must modernize their KYC architectures because of several factors: rising transaction volumes, increasing regulatory complexity, and customer demands for instant onboarding. Legacy systems create multiple problems. They slow down compliance processes and expose institutions to both operational risks and regulatory penalties. However, traditional KYC orchestration systems, often built on monolithic architectures, struggle to meet these demands because of latency, availability, and scalability challenges. Their reliance on batch processing and manual handoffs leads to higher operational costs and impediments to real-time compliance validation, reinforcing the need for architectural modernization.
This post extends IBM’s approach to real-time KYC validation using generative AI, as previously discussed in the post IBM Digital KYC on AWS uses Generative AI to transform Client Onboarding and KYC Operations. It transforms compliance operations through autonomous decision-making and intelligent automation using agentic AI, event-driven architecture, and AWS serverless services. The solution addresses the fundamental limitations of traditional rule-based systems. It provides autonomous decision-making, dynamic adaptation, and intelligent automation that transforms compliance operations.
Financial institutions can break down KYC workflows into separate business functions. Amazon Managed Streaming for Apache Kafka (Amazon MSK) handles real-time event streaming, which speeds up processing. Amazon Bedrock automates document analysis and risk assessment with AI. AWS Lambda provides serverless computing that scales on demand and supports instant customer onboarding.
The critical role of KYC
KYC protects financial systems by verifying customer identities and detecting fraud in four ways. It supports regulatory compliance with anti-money laundering (AML) and counter-terrorist financing (CTF) regulations. It helps prevent fraud by detecting identity theft and forged documents. It manages risk by assessing customer profiles and monitoring transactions. And it builds customer trust through transparency. As financial institutions broaden their footprint across products, industries, and regions, KYC compliance becomes increasingly complex. Each financial service offering presents unique requirements, from traditional banking to digital wallets, investment systems, and cryptocurrency services. Expansion into retail, SME, and corporate segments brings diverse identity structures and risk profiles. Operating across multiple jurisdictions requires navigation of various regulatory frameworks. These frameworks include the Bank Secrecy Act (BSA) and USA PATRIOT Act in the US, Anti-Money Laundering Directives (AMLD) in the EU, and guidelines from international regulators like the Monetary Authority of Singapore (MAS) and Financial Action Task Force (FATF).
Traditional KYC
Traditional KYC processes verify customer identities, assess risk, and monitor for money laundering. They rely on manual document collection, identity checks across multiple databases, and periodic reviews. While these established processes have served the financial industry for decades, they were designed for a different era with lower transaction volumes, simpler product offerings, and less sophisticated threat landscapes. Today’s digital-first financial environment demands a fundamental reimagining of KYC at scale.
Current challenges
Legacy systems create several bottlenecks. They process requests in batches rather than real-time, making instant onboarding impossible. Manual validation across jurisdictions leads to inconsistent compliance. Without event-driven capabilities, these systems can’t integrate with modern AI and machine learning (ML) services or adapt to new fraud patterns without manual reconfiguration.
Cloud-native KYC solution architecture using agentic AI
This architecture illustrates a comprehensive cloud-native real-time KYC validation system designed to process live customer onboarding requests and validate identity information using AI-powered automation. The architecture uses an event-driven pipeline to process high-volume KYC validations securely in under 5 minutes. The system processes real-time KYC requests containing sensitive financial data including PII while maintaining strict security and regulatory compliance requirements across multiple geographies.
High-level Agentic Architecture for real-time KYC
This architecture diagram illustrates an AI-driven Know Your Customer (KYC) Orchestration Framework built using Amazon Bedrock AgentCore and Amazon Managed Streaming for Apache Kafka (Amazon MSK). The design showcases how multiple specialized AI agents collaborate to automate and optimize KYC workflows, from document ingestion to compliance validation and fraud detection, while maintaining real-time integration with on-premises financial systems.
At the heart of the architecture is the AgentCore Runtime Environment, which provides native orchestration capabilities, session management, and memory persistence. Within this runtime, the KYC Orchestration Supervisor Agent acts as the intelligent coordinator, delegating tasks to five domain-specific sub-agents: Identity Verification, Document Analysis, Fraud Detection, Compliance & Risk, and Customer Experience. Unlike traditional multi-agent systems, AgentCore provides built-in session state management, shared memory across sub-agents, and automatic context preservation throughout asynchronous processing workflows.
The architecture uses asynchronous invocation patterns where MSK consumers trigger AgentCore processing without blocking, enabling sub-5-minute processing times while handling thousands of concurrent KYC requests. Lambda functions serve as the integration layer, consuming events from MSK, invoking AgentCore asynchronously, and publishing results back to Kafka topics for downstream system consumption.
Each sub-agent uses foundation models hosted on Amazon Bedrock for tasks such as optical character recognition (OCR), language processing, behavioral analysis, and regulatory interpretation. These agents operate within the AgentCore Runtime, sharing context through AgentCore Memory (a built-in feature of Bedrock AgentCore that automatically manages session state and context) and accessing external systems through tools defined using OpenAPI schemas and Lambda targets.
The agents use KYC Knowledge Bases, powered by Amazon OpenSearch Serverless and Amazon Simple Storage Service (Amazon S3), to access contextual information from internal policies, compliance rules, vendor documentation, and regulations. This approach provides consistent, explainable, and policy-aligned decision-making. These knowledge bases integrate with AgentCore’s retrieval mechanisms, providing sub-agents with grounded information during processing.
Finally, the solution connects with existing on-premises systems, such as customer management, transaction monitoring, case management, risk/AML systems, and core banking systems. These connections use tools defined with OpenAPI schemas as targets and Lambda-based integrations using AgentCore Gateway. AgentCore Gateway uses these OpenAPI specifications to understand API contracts, handle authentication, validate requests and responses, and manage retries. AgentCore Identity manages authentication and authorization for agents and their tool access, so that only authorized sub-agents can invoke specific tools and access the Knowledge Base. With this approach, financial institutions can achieve an intelligent, scalable, and compliance-aligned KYC process that minimizes manual intervention, improves onboarding speed, and reduces fraud and regulatory risks.
Solution Components
Event-Driven Communication Infrastructure with Amazon MSK
Amazon MSK serves as the communication backbone, enabling asynchronous, real-time message exchange between agentic AI components and enterprise systems. The streaming infrastructure organizes into distinct topic categories supporting bi-directional flows.
Inbound topics capture customer interactions through KYC requests (new applications), document uploads (identity documents), ID verification results (third-party vendor responses), and transaction events (fraud/risk signals). Event listeners pre-process these streams. These listeners filter onboarding requests, prepare documents for OCR, normalize vendor data formats, and correlate transaction signals with customer profiles.
Outbound topics publish KYC decisions with confidence scores and audit trails to core banking systems, route complex cases to human reviewers through case management events, and trigger fraud alerts to security teams. With this decoupled architecture, you can achieve sub-5-minute processing while maintaining full event auditability and allowing independent scaling of individual agents based on workload patterns.
Agentic AI Orchestration Layer
KYC Orchestration Supervisor Agent
The Supervisor Agent implements intelligent routing logic using Amazon Bedrock AgentCore to dynamically determine optimal sub-agent collaboration patterns. Unlike rule-based systems following rigid workflows, the supervisor analyzes case characteristics (document types, customer geography, risk indicators, and historical patterns) to construct context-aware execution plans that invoke sub-agents in parallel or sequentially based on dependencies. The supervisor monitors sub-agent confidence scores to guide decision-making: high confidence (>95%) results in automatic approvals, medium confidence (75-95%) triggers additional verification, and low confidence (<75%) escalates to human review with comprehensive context.
Five Specialized Sub-Agents operate as autonomous decision-makers, each using foundation models for domain-specific tasks:
Identity Verification Sub-Agent validates customer identities against watchlists and sanctions databases. It calls third-party verification APIs and uses natural language processing to handle name variations.
Document Analysis Sub-Agent extracts data from identity documents using OCR. The agent handles poor image quality and multiple languages and detects forgery by analyzing watermarks and security features.
Fraud Detection Sub-Agent identifies suspicious patterns through behavioral analysis. The agent detects multiple applications from the same IP address or inconsistent information across form fields. It correlates current applications with historical fraud cases using semantic similarity search and maintains dynamic risk scores with explainable fraud assessments.
Compliance & Risk Sub-Agent supports regulatory adherence by interpreting jurisdiction-specific KYC requirements across different geographies. It translates regulatory frameworks into concrete validation actions and generates compliance attestations with audit trails for regulatory examinations.
Customer Experience Sub-Agent optimizes the onboarding journey by analyzing application progress in real time, identifying friction points, and recommending strategies to reduce abandonment while identifying upselling opportunities based on customer profiles.
Intelligent Knowledge Management Architecture
The KYC Knowledge Base implements a retrieval augmented generation (RAG) pattern that grounds agent decisions in factual, current information rather than relying solely on foundation model training. Amazon S3 stores source documents, including regulations from financial authorities, institution-specific compliance rules, internal policies, and vendor documentation, enabled to track changes over time. Documents undergo automated preprocessing for text extraction, metadata enrichment, and quality validation before the system indexes them. Amazon OpenSearch Serverless provides semantic search using vector embeddings generated by Amazon Bedrock. When agents query using natural language questions, the system embeds queries into the same vector space and identifies semantically relevant document chunks through cosine similarity search, improving retrieval accuracy over keyword matching.
Context-aware retrieval enriches queries with case-specific information, including customer jurisdiction, document types, and risk levels – facilitating highly relevant regulatory guidance. This continuous knowledge access keeps agent decisions grounded in institutional knowledge rather than hallucinating responses.
Real-Time Decision Store (Amazon DynamoDB) complements the Knowledge Base with sub-millisecond access to frequently accessed structured data, including current KYC decision status, risk scores, customer interaction history, and dynamic configuration parameters controlling agent behavior.
Secure integration with on-premises financial systems
The architecture integrates with on-premises financial systems through Action Groups bridging the cloud-native agentic layer and existing enterprise infrastructure.
Customer Management Systems receive real-time KYC decisions, updating verification status and account activation flags. Transaction Monitoring Systems consume fraud alerts and risk scores, enabling immediate action on suspicious patterns. Case Management Systems receive escalated cases with comprehensive agent analysis context, accelerating human review. Risk and AML Systems integrate bidirectionally to maintain consistent risk assessments. Core Banking Systems receive approved validations, triggering account activation.
Secure connectivity through AWS Direct Connect or AWS Site-to-Site VPN provides encrypted data transmission over dedicated network paths. API calls include comprehensive audit logging through AWS CloudTrail and Amazon CloudWatch, satisfying regulatory requirements.
Security Considerations
The solution should incorporate multi-layered security controls, continuous monitoring, and automated compliance auditing to meet the rigorous expectations of financial regulators and internal risk teams. Financial institutions should conduct a comprehensive threat modelling to identify risks including introduced by agentic AI systems. For further information please refer Security Guidance.
Conclusion
This KYC architecture uses AWS serverless services and Amazon Bedrock to process validations faster and at scale. The parallel agent execution model is designed to reduce KYC validation time from the typical 3-5 days to near-real time for standard cases. This approach enables exponentially faster processing through simultaneous operation of Document Analysis, Identity Verification, and Fraud Detection agents rather than sequential workflows.
With this architecture, financial institutions can handle high-volume validations through elastic scaling, optimize costs through serverless pay-per-use pricing, and improve accuracy through multi-agent collaboration. Automated document processing and intelligent routing are expected to reduce manual review workload, allowing each compliance specialist to handle up to 4x their current caseload while focusing on complex cases requiring human expertise. Explainable AI decisions with comprehensive audit trails support regulatory compliance and enable rapid audit responses.
Event-driven architecture and agentic AI help financial institutions compete in digital landscapes while meeting regulatory requirements.
Note: The architecture presented here is for reference purposes only. IBM and AWS will work closely with you to execute a Proof of Concept and implementation plan in accordance with industry standards and compliance requirements.
IBM Consulting is an AWS Premier Tier Services Partner that helps customers who use AWS to harness the power of innovation and drive their business transformation. They are recognized as a Global Systems Integrator (GSI) for over 30 competencies, including Financial Services Consulting. For additional information, please contact an IBM Representative.
Building on our recent announcement of AWS Security Hub Extended —our full-stack enterprise security offering — we want to show you how we’re simplifying security procurement and operations for your multicloud environments. Whether you’re a security architect evaluating solutions or a CISO looking to streamline vendor management, this post walks through the streamlined experience that transforms how you acquire, deploy, and manage end-to-end enterprise security solutions across endpoint, identity, email, network, data, browser, cloud, AI, and security operations. Security Hub Extended brings together AWS security services with carefully curated security partners. Delivering better outcomes together through unified procurement, billing, and operations that significantly reduce vendor management overhead so you can focus on what matters most: protecting your organization.
The challenge we’re addressing
Security teams today spend too much time on vendor management, evaluating services, negotiating contracts, and managing multiple billing cycles instead of focusing on what matters most: managing risk. But the procurement challenge runs even deeper. Until now, customers really only had one option: sign multi-year agreements based solely on proof-of-concept testing and estimated annual usage. This forces organizations to commit budget before they can validate whether a solution will work for them at scale.
AWS Security Hub Extended transforms this procurement model. Security Hub Extended offers customers the option to get started with pay-as-you-go pricing and no commitments, so they can move fast and validate solutions in their actual environment. After they’ve confirmed a solution works at scale, they can then align their vendor strategy and sign longer-term commitments for even more favorable pricing.
Security Hub Extended provides a curated set of carefully chosen partner solutions with competitive pricing, unified billing through your AWS account, and seamless integration. Our initial launch partners, selected by customers for their proven value, include 7AI, Britive, CrowdStrike, Cyera, Island, Noma, Okta, Oligo, Opti, Proofpoint, SailPoint, Splunk, Upwind, and Zscaler.
Getting started with Security Hub Extended
AWS Security Hub consolidates threat analytics from Amazon GuardDuty, vulnerability management from Amazon Inspector, and sensitive data discovery from Amazon Macie, correlating these signals with Security Hub Exposure findings to determine overall risk, reachability, and assumability. Security Hub Extended builds on this foundation by adding curated partner solutions, extending these unified security operations across your entire organization including multicloud, on-premises, and endpoint environments. If you’re already using Security Hub, you can navigate directly to the Extended plan section.
Getting started with Security Hub is straightforward. From the AWS Management Console, search for Security Hub to start the onboarding walkthrough. If you’re not already a Security Hub customer, you can quickly complete onboarding by designating an AWS organization delegated administrator (DA) account. You can then centrally enable and manage Security Hub across your entire organization’s accounts and AWS Regions from a single location (see Introduction to AWS Security Hub). After you’ve onboarded, navigate to the Extended plan section to add curated partner solutions.
Figure 1: Security Hub centralized configuration
From this single interface, you can enable detection and response capabilities across your entire organization, provide granular configurations at the organizational unit or member account level, select specific Regions, and turn individual features on or off as needed.
Understanding risk through attack paths
The Security Hub risk correlation engine identifies potential exposures by correlating threats, vulnerabilities, and misconfigurations to reveal how they connect and could lead to compromise of critical resources.
The attack path visualization in the preceding figure reveals critical insights including upstream root causes and blast radius, showing the potential impact if a threat actor exploits a vulnerability. You can use this visualization to focus on fixing the root cause rather than addressing symptoms. For example, updating one security group configuration can eliminate the entire attack path, cutting off all downstream exposure.
Accessing Security Hub Extended
You can find Security Hub Extended, shown in the following figure, in the left navigation pane under Management in your Security Hub delegated administrator (DA) account; Security Hub Extended will only be visible from the delegated administrator account. The Extended plan brings curated third-party security solutions directly into the Security Hub experience. Because Extended is built into Security Hub, there’s no separate console to manage. You discover, subscribe to, and operate curated partner solutions from the same place you manage enterprise security, delivering unified operations across your entire security estate.
Transparent, competitive pricing consolidated with Security Hub
Unlike traditional third-party engagements that require lengthy negotiations, private pricing deals, and multi-year commitments, Security Hub Extended offers complete pricing transparency. Every partner solution displays clear, competitive monthly pay-as-you-go rates billed directly with Security Hub requiring no commitments. For example, Cloud Security from Upwind costs $3.75 per resource per month, and Identity Security from Okta costs $20 per user per month.
All Security Hub Extended offerings are also eligible for AWS Enterprise Discount Program (EDP) discounts that will be applied automatically. If you have an existing AWS enterprise discount agreement, those discounts automatically apply to Security Hub Extended offerings, further reducing your effective costs. All partner solutions you deploy through Security Hub Extended appear on your consolidated AWS bill, no separate invoices or payment processes.
Streamlined onboarding
Adopting curated partner solutions through Security Hub Extended is straightforward. Choose View Product to initiate an automated workflow. Depending on the solution, you’ll either be directed to the partner onboarding console or provide information for the partner to guide you through their onboarding process tailored to your environment.
Billing begins only after you’re fully activated on the partner solution and starts automatically, no additional action is required to benefit from the unified billing. If you’re already using one of the curated partner solutions, transitioning to Security Hub Extended for consolidated billing and flexible pricing won’t disrupt your current services. Now, instead of receiving separate invoices for each partner in addition to Amazon Inspector, GuardDuty, and Security Hub CSPM you get one unified bill through Security Hub. This consolidates visibility to support better understanding of spend and to manage cost.
Unified operations
Security Hub Extended unifies security operations by consolidating findings from AWS and curated partner solutions. All findings use the Open Cybersecurity Schema Framework (OCSF) for consistency, without the need for complex data normalization, transformation, and extract, transform, and load (ETL) processes.
When you deploy solutions such as CrowdStrike, Noma, and Upwind alongside Splunk and 7AI through Security Hub Extended, security findings automatically flow into Security Hub and then seamlessly route to Splunk and 7AI. All in OCSF format so your security team can focus on responding to threats, not managing pipelines, so you can quickly identify and respond to security risks that span boundaries—from endpoint compromises to cloud infrastructure—without spending valuable time on manual integration work.
The full-stack security vision
Security Hub Extended represents a shift in how you discover, procure, and build comprehensive security programs. Instead of managing dozens of vendor relationships, negotiating separate contracts, agreeing to multi-year annual commitments, and integrating disparate tools, you now have one procurement process through AWS, one bill with transparent competitive pay-as-you-go pricing, one console for unified security operations, one support channel for AWS Enterprise Support customers, and one schema (OCSF) for all security findings. The result: reduced security risk, improved team productivity, and a more unified approach to security operations across your enterprise.
Get started
Try Security Hub Extended today and experience how simplified procurement and unified operations can transform your security program. Security Hub Extended is generally available globally in all AWS commercial Regions where Security Hub is available. We’ve also published a walk through video to further explain how Security Hub Extended works.
It’s still Day 1, but we’re iterating fast, so share your feedback with us on AWS re:Post for Security Hub or through your AWS Support contacts and watch for future blog posts on our progress.
As your data and machine learning (ML) assets grow, tracking which assets lack documentation or monitoring asset registration trends becomes challenging without custom reporting infrastructure. You need visibility into your catalog’s health, without the overhead of managing ETL jobs. The metadata feature of Amazon SageMaker provides this capability to users. Converting catalog asset metadata into Apache Iceberg tables stored in Amazon S3 Tables removes the need to build and maintain custom ETL pipelines. Your team can then query asset metadata directly using standard SQL tools. You can now answer governance questions like asset registration trends, classification status, and metadata completeness using standard SQL queries through tools like Amazon Athena, Amazon SageMaker Unified Studio notebooks, and BIsystems.
This automated approach reduces ETL development time and gives your team visibility into catalog health, compliance gaps, and asset lifecycle patterns. The exported tables include technical metadata, business metadata, project ownership details, and timestamps, partitioned by snapshot date to enable time travel queries and historical analysis. Teams can use this capability to proactively monitor catalog health, identify gaps in documentation, track asset lifecycle patterns, and make sure that governance policies are consistently applied.
How metadata export works
After you enable the metadata export feature, it runs automatically on a daily schedule:
SageMaker Catalog creates the infrastructure — An Amazon Simple Storage Service (Amazon S3) table bucket named aws-sagemaker-catalog is created with an asset_metadata namespace and an empty asset table.
Daily snapshots are captured — A scheduled job runs once per day around midnight (local time per AWS Region) to export updated asset metadata.
Metadata is structured and partitioned — The export captures technical metadata (resource_id, resource_type), business metadata (asset_name, business_description), project ownership details, and timestamps, partitioned by snapshot_date for query performance.
Data becomes queryable — Within 24 hours, the asset table appears in Amazon SageMaker Unified Studio under the aws-sagemaker-catalog bucket and becomes accessible through Amazon Athena, Studio notebooks, or external BI tools.
Teams query using standard SQL — Data teams can now answer questions like “How many assets were registered last month?” or “Which assets lack business descriptions?” without building custom ETL pipelines.
The export evaluates catalog assets and their metadata properties in the domain, converting them into Apache Iceberg table format. The data flows into downstream analytics operations immediately, with no separate ETL or batch processes to maintain. The exported metadata becomes part of a queryable data lake that supports time-travel queries and historical analysis.
In this post, we demonstrate how to use the metadata export capability in Amazon SageMaker Catalog and perform analytics on these tables. We explore the following specific use-cases.
Audit historical changes to investigate what an asset looked like at a specific point in time.
Monitor asset growth view how the data catalog has grown over the last 30 days.
Track metadata improvements to see which assets gained descriptions or ownership over time.
Solution overview
Figure 1 – SageMaker catalog export to S3 Tables
The architecture consists of three key components:
Amazon SageMaker Catalog exports asset metadata daily to Amazon S3.
S3 Tables stores metadata as Apache Iceberg tables in the aws-sagemaker-catalog bucket with ACID compliance and time travel.
Query engines (Amazon Athena, Amazon Redshift, and Apache Spark) access metadata using standard SQL from the asset_metadata.asset table.
What metadata is exposed?
SageMaker Catalog exports metadata in the asset_metadata.asset table:
Metadata Type
Fields
Description
Technical metadata
resource_id, resource_type_enum, account_id, region
Resource identifiers (ARN), types (GlueTable, RedshiftTable, S3Collection), and location
An Amazon SageMaker project for publishing assets.
For SageMaker Unified Studio domain setup instructions, refer to the SageMaker Unified Studio Getting started guide.
After you complete the prerequisites, complete the following steps.
Add this policy to our IAM user or role to enable metadata export. If using SageMaker Unified Studio to query the catalog, add this policy to the AmazonSageMakerAdminIAMExecutionRole managed role.
Grant describe and select permissions for SageMaker Catalog with AWS Lake Formation. This step can be performed in the AWS Lake Formation console.
Select Permissions -> Data permissions and choose Grant.
Figure 2 – AWS Lake Formation grant permission
Under Principal type, select Principals, IAM users and roles and the AWS managed AmazonSageMakerAdminIAMExecutionRole execution role.
Choose Named Data Catalog resources.
Under Catalogs, search for and select <account-id>:s3tablecatalog/aws-sagemaker-catalog.
Under Databases, select asset_metadata database.
Figure 3 – AWS Lake Formation catalog, database, and table
Figure 4 – AWS Lake Formation grant permission
For Table, select asset.
Under Table permissions, check Select and Describe.
Choose Grant to save the permissions.
Enable data export using the AWS CLI
Configure metadata export using the PutDataExportConfiguration API. The Amazon DataZone service automatically creates an S3 table bucket named aws-sagemaker-catalog with an asset_metadata namespace, and schedules a daily export job. Asset metadata is exported once daily around midnight local time per AWS Region.
The SageMaker Domain identifier is available on domain detail page in the AWS Management Console. Accessing the asset table through the S3 Tables console or the Data tab in SageMaker Unified Studio can require up to 24 hours.
AWS CLI command to enable SageMaker catalog export:
Navigate to Amazon SageMaker Domains in the AWS Management Console.
Select your domain and select Open.
Figure 5 – Open Amazon SageMaker Unified Studio
In SageMaker Unified Studio, choose a project from the Select a project dropdown list.
To query SageMaker catalog data, select Build in the menu bar and then choose Query Editor. To create a new project, follow the instructions in the Amazon SageMaker Unified Studio User Guide.
Figure 6 – Open SageMaker Unified Studio Query Editor
The asset_metadata.asset table is available in Data explorer. Use Data explorer to view the schema and query data to perform analytics from.
Expand Catalogs in Data explorer. Then, select and expand s3tablecatalog, aws-sagemaker-catalog, asset_metadata, and asset.
Test querying the catalog with SELECT * FROM asset_metadata.asset LIMIT 10;.
Figure 7 – Query SageMaker catalog
Queries for observability and analytics
With setup complete, execute queries to gain insights on catalog usage and changes. To monitor asset growth, and view how the data catalog has grown over the last five days:
SELECT
DATE (snapshot_time) as date,
COUNT (*) as total_assets
FROM asset_metadata.asset
WHERE
DATE (snapshot_time) >= CURRENT_DATE - INTERVAL '5' DAY
GROUP BY DATE (snapshot_time)
ORDER BY date DESC;
Figure 8 – Query asset growth
Use the catalog to track metadata changes to determine which assets gained descriptions or ownership over time. Use this query to identify assets that gained business descriptions over the past five days by comparing today’s snapshot with the earlier snapshot.
SELECT
t.asset_id,
t.resource_name,
p.business_description as description_before,
t.business_description as description_now
FROM asset_metadata.asset t
JOIN asset_metadata.asset p ON t.asset_id = p.asset_id
WHERE DATE(t.snapshot_time) = CURRENT_DATE
AND DATE(p.snapshot_time) = CURRENT_DATE - INTERVAL '5' DAY
AND p.business_description IS NULL
AND t.business_description IS NOT NULL;
Investigate asset values at a specific point in time using this query to retrieve metadata from any snapshot date.
SELECT
asset_id,
resource_name,
business_description,
extended_metadata['owningEntityId'] as owner,
snapshot_time
FROM asset_metadata.asset
WHERE asset_id = 'your-asset-id'
AND DATE(snapshot_time) = DATE('2025-11-26');
Clean up resources
To avoid ongoing charges, clean up the resources created in this walkthrough:
Disable metadata export:
Disable the daily metadata export to stop new snapshots:
Optionally, delete the S3 Tables namespace containing the exported metadata to remove historical snapshots and stop storage charges. For instructions on how to delete S3 tables, see Deleting an Amazon S3 table in the Amazon Simple Storage Service User Guide.
Conclusion
In this post, you enabled the metadata export feature of SageMaker Catalog and used SQL queries to gain visibility into your asset inventory. The feature converts asset metadata into Apache Iceberg tables partitioned by snapshot date, so you can perform time-travel queries, monitor catalog growth, track metadata completeness, and audit historical asset states. This provides a repeatable, low-overhead way to maintain catalog health and meet governance requirements over time.
You can process real-time data from your data lake with Amazon Managed Service for Apache Flink without maintaining two separate pipelines. Yet many teams do exactly that, and the cost adds up fast. In this post, you build a unified pipeline using Apache Iceberg and Amazon Managed Service for Apache Flink that replaces the dual-pipeline approach. This walkthrough is for intermediate AWS users who are comfortable with Amazon Simple Storage Service (Amazon S3) and AWS Glue Data Catalog but new to streaming from Apache Iceberg tables.
The dual-pipeline problem
This dual-pipeline approach creates three problems:
Double the infrastructure costs. You run and pay for two separate compute environments, two storage layers, and two sets of monitoring. For example, if you’re spending $10,000/month on separate streaming and batch infrastructure, a meaningful portion of that spend is pure duplication.
Data synchronization issues. Your batch and streaming consumers read from different copies of the data, processed at different times. When a transaction shows up in your real-time dashboard but not in your batch report (or vice versa), debugging the inconsistency takes hours.
Operational complexity. Two pipelines mean two deployment processes, two failure modes to monitor, and two sets of schema evolution to manage. Your team spends time reconciling systems instead of building features.
Where this pattern fits
Before diving into the implementation, consider whether streaming from your data lake is the right approach for your use case.
Streaming from Apache Iceberg tables works well when you need data available within seconds to minutes and you query recent data frequently, multiple times per hour. Common scenarios include:
Operational data stores — Stream customer profile updates to serve downstream applications like recommendation engines. When a customer updates their preferences, those changes reach your operational data store within seconds.
Fraud detection — Stream transactions for immediate analysis. Start with a 3-second monitor interval and adjust based on your detection accuracy needs.
Live dashboards — Power real-time analytics directly from your lake. This is the strongest starting point if you’re evaluating the approach for the first time, because the feedback loop is immediate and straightforward to validate.
Event-driven architectures — Trigger downstream processes based on data changes in your Apache Iceberg tables.
Batch processing remains more cost-effective when you process data once per day or less, or you primarily query historical data. Batch queries on Apache Iceberg tables cost less because they don’t require a continuous Apache Flink runtime.
How Apache Iceberg solves this
Apache Iceberg’s snapshot-based architecture removes the need for a separate streaming pipeline. Think of snapshots like Git commits for your data. Each time you write data to your Iceberg table, Iceberg creates a new snapshot that points to the new data files while preserving references to existing files. Apache Flink reads only the changes between snapshots (the new files that arrived after the last checkpoint), rather than scanning the entire table. Atomicity, Consistency, Isolation, Durability (ACID) transactions prevent your concurrent reads and writes from producing partial or inconsistent results. For example, if your batch extract, transform, and load (ETL) job is writing 10,000 records while your Flink application is reading, ACID transactions mean that your streaming query sees either the complete batch of 10,000 records or none of them, not a partial set that could skew your analytics.
The result is a single pipeline that handles both real-time and batch access from the same data, through the same storage layer, with the same schema.
Solution architecture
Your architecture uses four AWS services and one open source table format working together. The following diagram shows how these components connect, replacing the dual-pipeline pattern shown earlier with a single unified flow.
Your source data lands in Amazon S3 as Apache Iceberg table files. AWS Glue Data Catalog tracks the metadata and schema. When new data arrives, Apache Iceberg creates a new snapshot that your application detects. Your Flink application monitors these snapshots and processes new records incrementally, reading only the files that arrived after the last checkpoint, not the entire table.
You use four main components:
Amazon S3 — Foundational storage layer for your data lake
Data Catalog — Metadata and schema management for Apache Iceberg tables
Apache Iceberg — Table format with snapshot-based streaming capabilities
Amazon Managed Service for Apache Flink — Stream processing and incremental consumption
Important notices
Before implementing this solution, evaluate these risks for your environment:
Data security: Streaming from data lakes exposes data to additional processing systems. Classify your data before implementation—customer profile updates and transaction data typically contain personally identifiable information (PII) and treat them as confidential. Apply encryption at rest and in transit for confidential data. Key risks include unauthorized data access through misconfigured Amazon S3 bucket policies or overly permissive IAM roles. Mitigations: use the resource-scoped IAM policy and TLS-enforcing bucket policy provided in the Security section.
Data integrity: Misconfigured checkpoints or schema changes during streaming can lead to data inconsistency. Mitigations: enable exactly-once processing semantics and test schema evolution in a non-production environment first.
Compliance: Verify that real-time data processing meets your regulatory requirements. For workloads subject to HIPAA, confirm that you use HIPAA Eligible Services and have a Business Associate Agreement (BAA) with AWS. For PCI-DSS or GDPR workloads, review the relevant compliance documentation on the AWS Compliance page. Implement data retention policies that comply with your regulatory framework.
Cost: Nearly continuous streaming incurs ongoing compute costs. Monitor usage to avoid unexpected charges. Cost estimates in this post are based on pricing as of March 2026 and might change. Verify current pricing on the relevant AWS service pricing pages.
Operational: Pipeline failures might impact downstream systems. Implement monitoring and alerting before running in production.
Prerequisites
Before you begin, make sure that you have the following in place. This walkthrough assumes intermediate Python skills (comfortable with functions, error handling, and environment variables), basic Apache Flink concepts (streaming compared to batch processing), and basic AWS Identity and Access Management (AWS IAM) knowledge (creating roles and attaching policies). Plan for approximately 90–120 minutes, including setup, implementation, and testing. First-time setup might take longer as you download dependencies and configure AWS resources. Expected AWS costs: approximately $5–10 if you complete the walkthrough within 2 hours and clean up resources immediately afterward. The primary cost driver is Amazon Managed Service for Apache Flink runtime ($0.11/hour per Kinesis Processing Unit (KPU)). You can minimize costs by stopping your application when not in use.
An AWS account with AWS IAM permissions for: s3:GetObject, s3:PutObject, s3:ListBucket on your data bucket; glue:GetDatabase, glue:GetTable for catalog access; and flink:CreateApplication, flink:StartApplication for Amazon Managed Service for Apache Flink
You need multiple JAR files because your Flink application coordinates between different systems—Amazon S3 for storage, AWS Glue for metadata, Hadoop for file operations, and Apache Iceberg for the table format. Each JAR handles a specific part of this integration. Missing even one causes ClassNotFoundException errors at runtime.
iceberg-flink-runtime-1.19-1.6.1.jar — Core Apache Iceberg integration with Apache Flink
iceberg-aws-bundle-1.6.1.jar — AWS-specific Apache Iceberg functionality for Amazon S3 and AWS Glue
flink-s3-fs-hadoop-1.19.1.jar — Provides Apache Flink read and write access to Amazon S3
flink-sql-connector-hive-3.1.3_2.12-1.19.1.jar — Hive metastore connector for catalog compatibility
hadoop-common-3.4.0.jar — Core Hadoop libraries required by Apache Iceberg
flink-shaded-hadoop-2-uber-2.8.3-10.0.jar — Repackaged Hadoop dependencies that avoid version conflicts with Apache Flink
hadoop-hdfs-client-3.4.0.jar — Hadoop Distributed File System (HDFS) client libraries for file system operations
flink-json-1.19.1.jar — JSON format support for Apache Flink
hadoop-aws-3.4.0.jar — Hadoop integration with AWS services
hadoop-client-3.4.0.jar — Hadoop client libraries
aws-java-sdk-bundle-1.12.261.jar — AWS SDK for authentication and service access
The sample code in this post is available under the MIT-0 license.This section walks you through building the streaming pipeline step by step. You create a single Python file, iceberg_streaming.py, with three functions that run in sequence. Your main() function calls them in order: set up the Apache Flink environment, register the Data Catalog, then start the streaming query.
Set up your Apache Flink environment
To prepare your Apache Flink environment:
Download the required JAR files listed in the prerequisites section.
Place the JAR files in a lib directory in your project folder.
Configure your HADOOP_CLASSPATH environment variable to point to the lib directory.
Create your streaming execution environment by adding the following function to iceberg_streaming.py:
Verify your environment by running flink –version. If the command isn’t found, confirm that Apache Flink 1.19.1 is installed and that your PATH includes the Flink bin directory.
Configure AWS Glue Data Catalog
To connect your Flink application to Data Catalog:
Open your iceberg_streaming.py file.
Add the create_iceberg_source() function shown in the following section.
Replace the placeholder values with your actual AWS resources before running. These values are static configuration strings, not user input — do not construct them from external or untrusted sources at runtime.
Save the file.
def create_iceberg_source(t_env):
"""Register the AWS Glue Data Catalog as an Iceberg catalog."""
try:
catalog_sql = """
CREATE CATALOG glue_catalog WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
'warehouse'='s3://<example-data-lake-bucket>',
'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
'aws.region'='us-east-1',
'hadoop-conf.fs.s3a.aws.credentials.provider'=
'com.amazonaws.auth.DefaultAWSCredentialsProviderChain',
'hadoop-conf.fs.s3a.endpoint'='s3.amazonaws.com',
'property-version'='1'
)
"""
t_env.execute_sql(catalog_sql)
t_env.use_catalog("glue_catalog")
t_env.use_database("streaming_db")
except Exception as e:
print(f"Failed to configure Iceberg catalog: {e}")
raise
Set up streaming logic
This function configures Apache Flink to monitor your Apache Iceberg table continuously and process new records as they arrive. Checkpointing runs every 10 seconds to track progress—if the job restarts, it resumes from the last checkpoint rather than reprocessing the entire table.Notice the monitor-interval parameter, it controls how frequently Apache Flink checks for new Apache Iceberg snapshots. A 3-second interval provides near real-time processing but generates approximately 1,200 Amazon S3 LIST API calls per hour (at $0.005 per 1,000 requests, roughly $0.04/month per table based on pricing as of March 2026). For less time-sensitive workloads, increase this to 30s to reduce API costs by 90%.Replace customer_events with the name of your Apache Iceberg table in Data Catalog:
def process_record(row):
"""Validate and process each record from the stream."""
try:
if row is None:
raise ValueError("Received null row")
required_fields = ["event_type", "timestamp"]
for field in required_fields:
if field not in row:
raise ValueError(f"Missing required field: {field}")
# Validate field types and content
if not isinstance(row.get("event_type"), str) or len(row["event_type"]) > 256:
raise ValueError("event_type must be a string under 256 characters")
if not isinstance(row.get("timestamp"), (str, int)):
raise ValueError("timestamp must be a string or integer")
# Replace with your business logic
print(f"Processing record: {row}")
except ValueError as e:
print(f"Validation error for record {row}: {e}")
except Exception as e:
print(f"Error processing record {row}: {e}")
def stream_data(t_env):
"""Start the streaming query and process results."""
try:
configuration = t_env.get_config().get_configuration()
configuration.set_string("table.dynamic-table-options.enabled", "true")
configuration.set_string("execution.checkpointing.interval", "10000")
query = """
SELECT * FROM customer_events /*+ OPTIONS(
'streaming'='true',
'monitor-interval'='3s',
'table.exec.iceberg.cell-based-snapshot'='true'
) */
"""
table_result = t_env.execute_sql(query)
with table_result.collect() as results:
for row in results:
process_record(row)
except Exception as e:
print(f"Streaming query failed: {e}")
raise
Putting it together
Your main() function calls the three steps in order:
Run the pipeline locally:python iceberg_streaming.pyPackage the application and submit it to Amazon Managed Service for Apache Flink using the console or the AWS Command Line Interface (AWS CLI).
Running in production
Moving from a local test to a production deployment requires tuning four areas: performance, monitoring, cost, and security. This section covers the key decisions for each.
Performance tuning
Determine your latency requirements before tuning. For fraud detection, you need subsecond processing. For daily reporting dashboards, you can tolerate minutes of delay.
Partition pruning reduces the amount of data scanned per query. Proper partitioning can significantly reduce query times for time series data partitioned by date. To implement, create your Apache Iceberg table with partition columns (PARTITIONED BY (date_column) in your CREATE TABLE statement), then include partition filters in your WHERE clause: WHERE date_column >= CURRENT_DATE - INTERVAL '7' DAY.
Parallel processing matches your data volume and throughput requirements. For most workloads under 10,000 records per second, a parallelism of 1–4 is sufficient. Scale up incrementally and monitor backpressure metrics (indicators that data arrives faster than your pipeline processes it, causing queuing) to find the right setting.
Checkpoint tuning balances reliability and latency. Consider how much data you can afford to reprocess after a failure. If you process 1,000 records per second with 10-second checkpoints, a failure means reprocessing up to 10,000 records. When that’s acceptable, 10 seconds works well. For faster recovery or higher volumes, reduce to 5 seconds.
Resource allocation — Right-size your Apache Flink cluster to avoid over-provisioning. Monitor CPU and memory utilization during your initial runs and adjust task manager resources accordingly.
Monitoring
Configure your production deployment with the following checkpoint settings. These work well for moderate data volumes (up to 10,000 records per second), providing exactly-once processing semantics. This means that the pipeline processes each record exactly once, even if your application restarts. Adjust the checkpoint interval based on your latency requirements. Add this to your setup_environment() function after creating the table environment.
Use Amazon CloudWatch to track checkpoint duration, records processed per second, and backpressure metrics. A 10-second checkpoint interval means writing state to Amazon S3 360 times per hour. For a 1 MB state size, that’s approximately 8.6 GB per day in checkpoint storage—at Amazon S3 Standard pricing of $0.023/GB, roughly $0.20/day or $6/month per application based on current pricing. If the checkpoint duration exceeds 50% of your interval, increase the interval or add parallelism.
Cost management
Use Amazon S3 Intelligent-Tiering for your Apache Iceberg data files, which typically have predictable access patterns after initial processing. Configure Apache Iceberg’s table expiration to automatically clean up early snapshots. This can reduce storage costs by an estimated 20–30%, though your results vary depending on write frequency and retention policies.
Right-size your Apache Flink resources based on actual throughput needs. Start with a minimal configuration and scale up based on observed backpressure and checkpoint duration metrics. Use Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances where workload interruptions are acceptable, for example, in development and testing environments.
Set data retention policies on both your Apache Iceberg tables and checkpoint storage to avoid storing data longer than necessary.
Security
Security is a shared responsibility between you and AWS. AWS is responsible for the security of the cloud, including the hardware, software, networking, and facilities that run AWS services. You are responsible for security in the cloud, configuring access controls, encrypting data, and managing your application security. Apply these controls in priority order.
AWS IAM roles — Use AWS IAM roles with least-privilege access, scoped to specific resources. The following example policy restricts permissions to your data lake bucket and AWS Glue catalog:
Scoping permissions to specific Amazon S3 buckets, AWS Glue databases, and AWS Key Management Service (AWS KMS) keys restrict access to only the resources your pipeline requires. Review IAM policies quarterly using the IAM Access Analyzer to identify and remove unused permissions.
Encryption — Configure server-side encryption with AWS Key Management Service (AWS KMS) customer managed keys (SSE-KMS) for your Amazon S3 buckets. Using customer managed keys requires additional review from your security team. Confirm your key management policies, rotation procedures, and access controls before implementation. Enable automatic key rotation annually. For encryption in transit, enforce TLS by adding a bucket policy that denies non-HTTPS access:
Enable versioning on buckets that store critical data and checkpoints to protect against accidental deletion. For production environments with sensitive data, consider enabling MFA Delete on versioned buckets. Enable S3 server access logging to track requests for security auditing.
Access logging – Enable AWS CloudTrail data events to log Amazon S3 object-level API calls (GetObject, PutObject) and Data Catalog API calls. Store logs in a separate Amazon S3 bucket with restricted access and enable log file integrity validation. Run regular compliance checks using AWS Config.
Operational practices
Set up a continuous integration and continuous deployment (CI/CD) pipeline to automate deployment and testing. Use version control to track schema and code changes. With Apache Iceberg’s schema evolution support, you can add columns without rewriting existing data files. Establish rollback procedures using Apache Iceberg’s snapshot-based architecture, so you can roll back to a previous table state if a bad write corrupts your data.
Troubleshooting
If you run into issues during setup or execution, use the following table to diagnose common errors.
Error
Cause
Solution
ClassNotFoundException
Missing JAR files
Check the dependencies in your lib directory and confirm HADOOP_CLASSPATH points to the correct path
Table not found
Database name mismatch
Check that the database name in t_env.use_database() matches the AWS Glue database where you registered your table
Checkpoint failures
Amazon S3 permissions
Check that your Amazon S3 bucket policy grants s3:PutObject for the checkpoint location
AWS credential errors
Missing AWS IAM configuration
Check that the AWS IAM role attached to your Apache Flink application has glue:GetTable, glue:GetDatabase, and s3:GetObject permissions on the relevant resources
Snapshot not found
Table modified during query
Increase monitor-interval or implement retry logic in your process_record() function
Schema mismatch
Table schema changed between snapshots
Review Apache Iceberg schema evolution settings and confirm backward compatibility
Clean up
To avoid ongoing charges, delete the resources that you created during this walkthrough.
Stop your Amazon Managed Service for Apache Flink application. Open the Amazon Managed Service for Apache Flink console, choose your application name, choose Stop, and confirm the action. Or use the AWS CLI:
If you created an Amazon VPC or Amazon VPC endpoints for testing, delete those resources.
Conclusion
Maintaining separate streaming and batch pipelines doubles your infrastructure costs, creates data synchronization issues, and adds operational complexity that slows your team down. In this post, you replaced that dual-pipeline architecture with a single system built on Apache Iceberg and Amazon Managed Service for Apache Flink. You configured a Flink environment with the required JAR dependencies, connected it to Data Catalog, and implemented streaming queries that read new records incrementally with exactly-once processing semantics. The same data, the same storage layer, the same schema—accessible to both your real-time and batch consumers.
To extend this solution, try these next steps based on your use case:
If you’re processing high volumes (>10,000 records/sec): Start with partition pruning. Add PARTITIONED BY (date_column) to your table definition, this typically reduces query times by 60–80%.
If you need production monitoring: Implement custom Amazon CloudWatch metrics. Track checkpoint duration, records processed per second, and backpressure to catch issues before they impact your pipeline.
Security logs capture essential security-related activities, such as user sign-ins, file access, network traffic, and application usage. These logs are important for monitoring, detecting, and responding to potential security events. The Open Cybersecurity Schema Framework (OCSF) addresses this challenge by providing a standardized format to represent security events, ensuring consistent and efficient data handling across various systems. OCSF enhances interoperability, streamlines analysis, simplifies compliance reporting, and reduces vendor lock-in, fostering greater flexibility and efficiency in security operations.
However, manually transforming diverse security logs into OCSF format at scale can be complex and time-consuming. Amazon Security Lake simplifies this process by automatically centralizing security data from AWS services such as AWS CloudTrail management and data events (Amazon Simple Storage Service (Amazon S3) and AWS Lambda), Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, Amazon Route 53 resolver query logs, AWS Security Hub findings, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, and AWS WAF logs. It also centralizes security logs from software as a service (SaaS) providers, on-premises, and cloud sources into a purpose-built data lake stored in your account. It uses the OCSF format to standardize and normalize this data, ensuring consistency and simplifying analysis. By integrating with analytics tools such as Amazon Athena and Amazon Quick Sight, Security Lake simplifies threat detection, improves security posture monitoring, and streamlines compliance reporting, making it an essential tool for modern security operations.
In this post, we show you how to transform custom security logs into OCSF format after you have the OCSF mappings ready, using a configuration-driven extract, transform, load (ETL) solution.
Accelerating OCSF adoption with AWS ProServe ETL solution
Amazon Security Lake stores security data in OCSF format and so customers looking to use custom log sources in Security Lake must transform their logs into OCSF format. To facilitate this process, the AWS Professional Services (ProServe) team built an ETL solution accelerator that converts custom security logs into OCSF format. This solution bridges existing log formats with the OCSF version 1.1 standard, streamlining data onboarding into Security Lake or other data lakes of security logs coming from multiple security tools.
Prerequisites
To implement this solution, you must have the following resources:
This solution uses serverless AWS services including Amazon S3, Lambda, DynamoDB, Step Functions, and either AWS Glue or Amazon EMR Serverless for ETL. Costs depend on data volume and processing frequency. Key expenses are storage (Amazon S3 and DynamoDB), compute (Lambda, AWS Glue, and Amazon EMR), and orchestration services. The architecture is cost-optimized through serverless components and pay-per-use pricing. Use AWS Pricing Calculator to estimate costs for your specific log volume and retention needs.
Solution overview
The solution uses two input files: a mapping file and a configuration file. These files guide the transformation of source logs into OCSF-compliant Parquet format, which is then partitioned by location/region=region/accountId=accountID/eventDay=yyyyMMdd/ and stored in an Amazon S3 location provided by Security Lake.
The following diagram shows the key architecture components of this solution and data flow between them.
Figure 1: Architecture diagram of ETL solution to transform security logs into OCSF format
The steps mentioned below walks you through the architecture diagram:
Preprocessing steps:
User uploads a mapping file in CSV format that maps custom security logs into OCSF class.
User uploads a metadata file in CSV format that is passed to the solution to transform custom security logs into OCSF format.
An Amazon S3artifact bucket stores the metadata, source-to-target mapping, and Python libraries required for OCSF conversion.
An Amazon S3 event notification invokes the Lambda function that writes the metadata to the asl-etl-framework-ocsf-attribute-metadata DynamoDB table when the metadata files are created or updated.
Metadata and mapping Lambda functions process the respective configuration files and store the required information in DynamoDB tables.
The Reference Lambda function extracts the required OCSF attributes using an API call and stores the results in a DynamoDB table.
Optional enrichment process: The solution reads data from an enrichment database stored on Amazon RDS or an external on-premises database that’s accessible through a JDBC connection from either Amazon EMR or AWS Glue. The credentials of this enrichment database are stored in Secrets Manager.
Source log files are delivered to an S3 bucket by an external process.
An EventBridge schedule or manual invoke initiates the Step Functions workflow, responsible for log conversion.
A Step Functions workflow performs the following tasks:
The preprocessor Lambda function performs checkpointing and invokes the required number of ETL jobs in parallel.
The ETL job converts the source log files to the OCSF-Parquet format using the custom Python libraries stored in the artifact bucket and the mapping information defined in the DynamoDB table.
A separate target S3 bucket stores the converted log data.
An Amazon SNS topic is used to notify users if the Step Functions workflow fails during checkpointing or the ETL process.
Analytics are performed on the converted data.
Deployment
You can find the required resources to deploy this solution in this GitHub repository. It provides detailed instructions in the README on how to deploy the solution. After you have the prerequisites mentioned earlier, see the Environment Setup portion of the repository.
Solution walkthrough
In this section, we walk you through steps to deploy this solution.
Map source log files into OCSF format
Before you start mapping the security logs into OCSF format, check if there are existing mappings available on OCSF mappings Github.
Mapping security logs into the OCSF format typically involves several steps. Here are the high-level steps:
Understand OCSF schema: Familiarize yourself with the OCSF schema, which defines the structure and format for organizing security log data into event classes and attributes. In OCSF, events are organized into event classes, each of which comprises a set of attributes designed to offer comprehensive semantics for the event.
Identify log sources: Determine your security log sources, such as firewalls, intrusion detection systems, or antivirus software. Each log source might have its own format (CSV, JSON, and so on) and structure.
Identify OCSF categories and classes: Analyze the log content and match security events to the appropriate OCSF categories and classes for standardized data organization.
Map fields to OCSF schema: Map the source log data fields in the OCSF schema. Ensure that each field from your logs is mapped to the appropriate field in the OCSF format. If a field in the source log schema isn’t mapped to any OCSF field, you might need to consider mapping it to unmapped object.
Enrichment: Enrich data with additional contextual information, such as standardizing timestamps, converting IP addresses to a common format, or adding supplementary data for better analysis. The enrichment column is added to the final dataset. Each category in OCSF has an optional enrichment column that provides more information about a column. For example, the Authentication OCSF category contains an optional enrichment column that provides more details about the IP addresses. .
Test and validate: Validate mapped log data against the OCSF schema to ensure compliance and accuracy. Test the mapping process with sample log data from different sources to identify any inconsistencies or errors. You can use this open source utility to validate your generated OCSF version 1.1 output file based on mapping.
Contribute OCSF Mapping to the OCSF community: Submit the OCSF mapping to the Github repository and raise a pull request to contribute it to the OCSF community. Iterate on the mapping procedure to improve accuracy, efficiency, and compatibility with the OCSF schema based on the pull request feedback.
By following these steps, you can effectively map security logs into the OCSF format, enabling better interoperability, analysis, and collaboration across security tools and platforms. AWS ProServe has helped many customers map their security logs to OCSF format. If you need guidance to map and transform security logs into OCSF format and want to use AWS ProServe, reach out to your account executive.
Create and transform mapping files
The ETL solution requires a CSV mapping file that maps the custom security log attributes into standardized OCSF attributes based on the specified OCSF class. For detailed instructions on generating this mapping file, see the Solution Usage section, bullet 2, in the README of the code repository. To follow the instructions in this post, you can enable Amazon S3 server access logging to publish source logs to Amazon S3. The following is a sample S3 server access log record:
Because the sample record uses spaces as delimiters and contains an extra space before +0000, you need to wrap each attribute in quotes. Here’s a sample Python code implementation that handles this requirement:
import re
def format_s3_access_log(log_line):
def quote_field(field):
"""Add quotes around a field and handle special cases"""
if field is None or field.strip() == '':
return '"-"'
# If field is already quoted, return as is
if field.startswith('"') and field.endswith('"'):
return field
return f'"{field}"'
try:
# First, protect quoted strings and bracketed content by temporarily replacing them
protected_line = log_line
protected_parts = re.findall(r'(\[.*?\]|".*?")', log_line)
for i, part in enumerate(protected_parts):
protected_line = protected_line.replace(part, f"PROTECTED_{i}_PART")
# Split the protected line
parts = protected_line.split()
# Restore protected parts
restored_parts = []
for part in parts:
if part.startswith('PROTECTED_') and part.endswith('_PART'):
index = int(part.split('_')[1])
restored_parts.append(protected_parts[index])
else:
restored_parts.append(part)
# Quote each field
quoted_fields = [quote_field(field) for field in restored_parts]
# Join with spaces
return ' '.join(quoted_fields)
except Exception as e:
print(f"Error processing line: {e}")
return None
# Example usage
if __name__ == "__main__":
# Example input log line
log_line = '''90de84bb542adb54766fec66ee554475b7e1a56a9d8b30e3598230f9ef6d6ac7 azv-asl-src-logs [29/May/2025:04:35:45 +0000] - arn:aws:sts::768196192565:assumed-role/AwsSecurityAudit/Palisade QS8DSY4SGF8M8SD7 REST.GET.BUCKETPOLICY - "GET /?policy HTTP/1.1" 200 - 255 - 39 - "-" "-" - N9XclJkv6hw/y4yApPyDII2sRoMNbqJqBEXdnmzFndcvhQOpdcc3PNQNQX7NhQaPJ5FKSVPh6hLB0GqsSN4apcbBUHi3rNcPRqa6rFLAYU4= SigV4 TLS_AES_128_GCM_SHA256 AuthHeader azv-asl-src-logs.s3.amazonaws.com TLSv1.3 - -'''
# Process the log line
formatted_output = format_s3_access_log(log_line)
print(formatted_output)
This sample code demonstrates how to wrap quotes around each attribute. You can extend this code to read source Amazon S3 server access log files from an S3 location and write the modified logs to another location. After these logs are available in an S3 bucket in your AWS account, you need to map the S3 server access logs to OCSF format. The following is an example of an S3 server access log CSV mapping file:
Upload the mapping CSV file to the S3 artifact location s3://secure-datalake-artifacts-<account_number>-<aws_region>/config/mapping/. The Lambda function asl-etl-framework_update-mapping-ddb ingests this mapping CSV file, processes its entries, and converts them into the required DynamoDB format. This Lambda function writes the results to the asl-etl-framework-ocsf-attribute-mapping DynamoDB table, which stores the schema and mapping information for all source log files processed by this solution. You can find an example of an S3 server access log CSV metadata file in the GitHub repository.
Upload the completed mapping CSV file into an S3 artifact location s3://secure-datalake-artifacts-<account_number>-<aws_region>/config/metadata/. An upload of a metadata CSV file to S3 invokes a Lambda function asl-etl-framework_insert_metadata_ddb, which stores the configuration in the asl-etl-framework-source-ocsf-metadata DynamoDB table. The following image shows the configuration in DynamoDB table.
Figure 2: Screenshot of metadata configuration in the asl-etl-framework-source-ocsf-metadata DynamoDB table for S3 Access Logs
After inserting the metadata into the asl-etl-framework-source-ocsf-metadata DynamoDB table, the Lambda function asl-etl-framework_update-mapping-ddb is invoked to read the mapping CSV file and inserts mappings into the asl-etl-framework-ocsf-attribute-mapping DynamoDB table. The following image shows the mapping in DynamoDB table.
Figure 3: Screenshot of transformed mapping in the asl-etl-framework-ocsf-attribute-mapping DynamoDB table for S3 Access Logs
Historical load
The ETL solution offers a historical load capability that processes logs from specified date or year ranges based on metadata file inputs. After being converted to OCSF format in Parquet file format, these logs can be integrated into Amazon Security Lake or be used to create a custom data lake. The solution includes checkpointing functionality to handle potential failures during historical data processing.
The checkpointing feature provides process resilience by tracking conversion progress in the asl-etl-framework-ocsf-run-status DynamoDB table. If a conversion process fails during multi-year historical processing, the solution resumes from the point of failure rather than reprocessing previously converted data. For example, if conversion fails while processing the second year’s data, the solution will resume from that point, preserving the first year’s successful conversion. While this feature is enabled by default, you can disable it, in which case any process restart will begin from the initial specified date. The following image shows the load_type as historical along with start_time and end_time for the period you want to transform the logs.
Figure 4: Screenshot of configuration for historical load attributes in the asl-etl-framework-source-ocsf-metadata DynamoDB table
Enrichment
Enterprises often possess valuable contextual data that can enhance their security logs through enrichment. By correlating existing data with security logs and appending relevant information, you can create more comprehensive datasets for advanced analytics and deeper security insights. After the logs are converted to OCSF, you might want to know more about specific columns or attributes so that you can extract meaningful information. To support this, the solution has an option for enrichment. For example, if you want to get additional information, such as the geolocation of each IP address in the logs, you can provide the source database information in the metadata CSV file of the solution. It connects to the source database through a JDBC connection, extracts the requested information associated with the IP address to enrich the dataset, and adds the extracted information as new columns to the converted OCSF log output. In this way, you can have detailed information about each IP address in the converted OCSF log. The following screenshot shows parameters for enabling enrichment by setting the is_enrichment_required flag as true and adding necessary enrichment_attributes to the metadata table.
Figure 5: Screenshot of configuration for enrichment attributes in the asl-etl-framework-source-ocsf-metadata DynamoDB table
ETL transformation using AWS Glue or EMR Serverless
You can use the engine of your choice for the transformation by providing the engine name during the deployment steps as mentioned in the Pre-Deployment Configuration section of the ReadMe. Based on this, the solution uses either AWS Glue or EMR Serverless as mentioned in the Orchestration using Step Functions section.
The process includes the following steps:
The user enters the metadata and mapping information in the respective CSV files and uploads the files to Amazon S3.
A process (Lambda job) converts the metadata and mapping files to a DynamoDB schema and stores them in corresponding DynamoDB tables (metadata and mapping tables).
A preprocessor job is invoked that takes the metadata from the DynamoDB table asl-etl-framework-source-ocsf-metadata and, based on the input parameters passed for the Step Functions workflow shown in the Orchestration using Step Functions section, the Step Functions workflow generates the input arguments for the transformation job (AWS Glue or EMR Serverless based on the user’s choice).
The transformation job (AWS Glue or Amazon EMR based on the user’s choice) is invoked and reads the metadata and mapping tables and converts the data into OCSF format.
The converted OCSF log files are stored to an Amazon S3 location in Parquet format, which is defined in the DynamoDB table asl-etl-framework-source-ocsf-metadata. These custom OCSF logs on S3 can be integrated with Security Lake.
Orchestration using Step Functions
This solution is orchestrated using Step Functions and offers two execution engine options: AWS Glue or EMR Serverless, depending on the services allow-listed in your enterprise. For processing historical loads, we recommend using EMR Serverless; however, AWS Glue is suitable for historical loads less than 100 GB. When invoking the Step Functions workflow, specify the execution engine as either emr-serverless or glue in the input parameters passed using EventBridge.
Figure 6: Screenshot of Step Functions workflow orchestration
To run the workflow, an input must be passed through an EventBridge schedule. The input parameters are as follows:
It’s a best practice to ensure that the generated Parquet files properly map to the various schema definitions specified within the Open Cybersecurity Schema Framework (OCSF). Validating the mapping helps to maintain data integrity and allows the security data to be effectively analyzed and processed by downstream applications and tools, such as Security Lake. You can use OCSF Schema Validator, which was built to provide supplementary validation for Security Lake. Performing this validation step helps detect any schema misalignments or data quality issues early in the process, leading to more reliable and trustworthy security analytics.
If validation of the transformed OCSF Schema fails using the OCSF Schema Validator, you need to validate if your mappings are aligned with the respective OCSF category. Adjust your mappings, rerun the solution, and validate the transformed OCSF logs using OCSF Schema Validator until you get a valid OCSF schema.
When discovering incorrect OCSF mappings or format inconsistencies in converted logs, begin by conducting a thorough validation against OCSF schema specifications to identify specific discrepancies. Update the mappings with correct field mappings, ensuring proper data type conversions and mandatory field requirements are met. Test these corrections using sample data to verify OCSF compliance using the above mentioned tool and data integrity before implementing in production.
Conclusion
In this post, we showed you how the ETL solution accelerator transforms custom security logs into the standardized OCSF format, enabling enhanced security analytics capabilities. This solution, developed by AWS Professional Services (AWS ProServe), addresses common challenges in security log standardization and streamlines the adoption of Amazon Security Lake. While the solution is available as an open source project, engaging with AWS ProServe provides significant advantages, including proven implementation expertise, best practices guidance, and accelerated deployment timelines. Our ProServe team brings extensive experience in security log standardization and can help customize the solution to your specific requirements while ensuring optimal integration with Security Lake. To begin your journey toward standardized security analytics using OCSF, contact your AWS account team to discuss how AWS ProServe can help implement this solution in your environment.
This post is co-authored with Gil Irizarry, Mae Wells-Kress and Craig Harmon from Babel Street.
Can your system tell “John Smith” apart from “John Smith”?
Organizations requiring identity intelligence increasingly face challenges due to complexity of matching names and entities across vast, multilingual, and constantly evolving datasets. Whether helping border security, combating financial crimes, or maintaining regulatory compliance, the accuracy of identity and entity resolution directly determines whether threats are detected, investigations succeed, and regulatory requirements are met. Yet, linguistic diversity, transliterations, inconsistent data formats, and legacy system limitations continue to create friction, leading to false positives, missed matches, and costly manual reviews. As customers ingest and analyze petabytes of unstructured and structured data in Amazon OpenSearch Service, the need for intelligent, scalable, and multilingual matching becomes increasingly important. This is where the integration of Babel Street (an AWS Partner) with OpenSearch Service provides a solution that helps organizations enhance precision, reduce noise, and accelerate insights from their high-volume data environments.
This post explores how combining Babel Street Match with OpenSearch Service provides a solution that helps your organization to handle large-scale, multilingual data.
The growing complexity of identity and entity resolution
As organizations ingest and analyze massive volumes of multilingual and inconsistently formatted data, accurately matching names and entities becomes increasingly difficult. Variations in spelling, transliterations, semantic differences, cultural naming conventions, and incomplete or noisy records can contribute to mismatches. These challenges are compounded by legacy systems, fragmented data pipelines, operational inefficiencies, and evolving regulatory requirements—especially in sectors where precision is a requirement.
Evaluating and enhancing identity in high-volume enterprise environments
Amazon OpenSearch Service is a fully managed, scalable search and analytics service that enables organizations to ingest, search, visualize, and analyze massive volumes of data in near real time. Built to handle structured and unstructured information from diverse sources, it powers use cases ranging from security analytics and log monitoring to enterprise search and advanced analytical applications.
Babel Street delivers risk intelligence trusted by organizations across government, defense, and the private sector. The offering combines access to vast volumes of multilingual data with advanced analytics to uncover hidden identities, secure vendor networks, and identify emerging risks with precision, speed, and scale. From national security to regulatory compliance and enterprise resilience, Babel Street provides the strategic advantage needed to stay ahead of risk, safeguard operations, and protect missions.
Babel Street Match, an offering from Babel Street incorporates advanced identity risk intelligence capabilities, which enhance the precision and reliability of screening processes. This advanced solution uses sophisticated matching techniques to verify identities and identify variations in personal data—including aliases, alternate spellings, and differences in biographical details, helping organizations separate legitimate individuals from potential threats. The ability to screen names, addresses, dates, and other identifiers across different scripts and languages helps reduce false positives and negatives, helps accurately detect critical risks with transparent scoring to meet compliance and audit requirements. Further, Babel Street Match streamlines screening workflows, reduces the burden of manual reviews, and elevates the accuracy of threat detection.
The following diagram shows the details of OpenSearch Service and Babel Street Match Plugin integration.
Babel Street Match integrates directly with the OpenSearch Service domain through a lightweight plugin that runs inside your own AWS account where you have full control of your data. The Match plugin sends encrypted match requests to Babel Street’s fully managed Match engine, where the core matching engine performs the entity-resolution logic. The results return to you in real time, enhancing your existing OpenSearch Service workflows with advanced name- and entity-matching capabilities. Meanwhile, Babel Street’s control plane handles licensing, monitoring, and AWS Marketplace integration behind the scenes, provides continuous validation, automated updates, and a seamless operational experience.
Example use cases
The solution combines enterprise-scale search and analytics with AI-powered, multilingual identity intelligence. This section showcases example use cases where integration has enhanced organizations’ capabilities.
Border Screening: Help agencies identify high-risk travelers, cargo, and networks to strengthen point-of-entry security with faster, automated risk assessment.
Financial Services Compliance: Help Financial institutions and the FinTechs that serve them by offering AI-driven solutions for name screening, adverse media monitoring, and know your customer (KYC)/know your vendor (KYV) due diligence.
Identity and Organization Screening: Help businesses needing identity and organization screening by providing AI, analytics, and advanced matching technologies to assist in addressing complex screening challenges.
Customer and Vendor Onboarding: Help governments and financial institutions by providing research, analytics, and advanced matching technologies needed to quickly and confidently onboard customers and vendors at scale.
Customer Success Stories
Here’s how leading organizations are leveraging Babel Street Match and Amazon OpenSearch Service to solve real-world identity challenges:
A European online brokerage faced AML (anti-money laundering) compliance challenges with its outdated name-matching system, which produced excessive false positives and couldn’t process longer multilingual names. After implementing Babel Street Match on OpenSearch Service, the firm achieved up to 70% better accuracy across 25 languages—significantly reducing manual work and speeding customer payments. Babel Street Match Improves FI’s Name-Matching Accuracy by Up to 70% on OpenSearch
A major border agency struggled with an outdated screening system that flagged 15% of travelers as potential watchlist matches—overwhelming agents and creating long queues. After implementing Babel Street Match, false positives dropped dramatically (from 80,000 to just 100 in one test), hardware needs fell by 70%, and travelers with common names can now pass through faster. As one stakeholder put it: “Name matching is not our biggest problem anymore.” Enabling Stronger, Safer Borders with AI-powered Screening by Babel Street Match
Getting Started with Babel Street Match for Amazon OpenSearch Service
Amazon OpenSearch Service supports third-party plugins like Babel Street Match for OpenSearch. This plugin is supported on OpenSearch version 2.15 or higher and licenses can be obtained through AWS Marketplace.
Installing Babel Street Match for Amazon OpenSearch Service
Prerequisites: Obtain the license file from Babel Street and upload it to an S3 bucket in the same AWS Region as your OpenSearch domain.
Installation Steps:
Create packages – In the OpenSearch Service console, create a package for your license file and select the Babel Street Match plugin from the available options
Associate packages – Link both the license and plugin packages to your OpenSearch domain
Verify – Monitor the domain update and confirm the plugin is active
Together, Babel Street Match and OpenSearch Service help organizations cut through false positives and catch true matches faster. The result? Greater precision, efficiency, and speed—whether protecting entities, maintaining compliance, or securing supply chains. That’s business-critical identity intelligence in action.
Explore how Babel Street Match on Amazon OpenSearch Service can elevate your organization’s identity intelligence capabilities and transform the screening operations through an interactive or customized demo on Babel Street’s website. Portions of this content describing Babel Street products and services are provided by Babel Street. AWS is not responsible for the accuracy of third-party product information.
In this post, we demonstrate how Notebooks in Amazon SageMaker Unified Studio help you get to insights faster by simplifying infrastructure configuration. You’ll see how to analyze housing price data, create scalable data tables, run distributed profiling, and train machine learning (ML) models within a single notebook environment.
Data scientists and analysts often spend days configuring infrastructure and managing authentication across multiple data sources before they can begin analysis. When working with data across Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Snowflake, and local files, teams face repeated authentication setup, manual compute scaling decisions, and tool-switching overhead that delays insights.
Notebooks in Amazon SageMaker Unified Studio provide instant access to 12+ data sources, compute scaling from local to distributed processing, and AI-powered code generation within a single browser-based environment. You’ll learn to use polyglot programming, multi-engine compute, and AI-assisted development to accelerate your path from question to insight.
What are Notebooks in Amazon SageMaker Unified Studio?
Notebooks in Amazon SageMaker Unified Studio provide an interactive environment for data analysis, exploration, engineering, and machine learning workflows. It delivers five integrated capabilities:
Polyglot programming: Write code in Python and SQL interchangeably within the same notebook environment
Unified data access: Connect instantly to data stored in Amazon S3, AWS Glue Data Catalog, Apache Iceberg tables, and third-party sources like Snowflake and BigQuery
Native visualization: Create charts directly from Python and SQL results for immersive data analytics
AI-powered development: Generate code through natural language prompts using SageMaker Data Agent, with an intelligent chat interface for data analytics, data science, and ML tasks
Flexible compute: Scale from basic instances to GPU-powered environments as your needs grow
Architecture
This section covers the architecture of Notebooks, which delivers enterprise-scale analytics with browser-based simplicity through a cloud-native architecture that integrates multiple compute engines, diverse data sources, and AI-powered assistance.
Presentation layer
You access the notebook interface through Amazon SageMaker Unified Studio, interacting with a familiar interface featuring code cells for execution, markdown cells for documentation, and visualization cells for charts and tables.
Compute layer
A dedicated notebook server manages your kernel lifecycle and session state. Key components include a Language Server for code completion, a Python 3.11 runtime with pre-loaded data science libraries, and a Polyglot Kernel that handles your Python, PySpark, and SQL execution within the same notebook. Persistent Amazon Elastic Block Store (Amazon EBS) storage backs each notebook you create.
Execution layer
Notebooks support multiple execution engines, automatically routing your code to the optimal processing engine. In-memory execution handles your smaller datasets and rapid prototyping. Apache Spark via Amazon Athena provides distributed processing for your large-scale analytics via Spark Connect. Native connectivity to Amazon Athena (Trino), Amazon Redshift, Snowflake, and BigQuery processes your SQL queries.
Data Integration
You get unified access to 12+ data sources including AWS-native (Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift) and third-party (Snowflake, BigQuery, PostgreSQL, MySQL) data sources. For the latest supported data sources, see Connect to data sources .
Security is embedded throughout the architecture to protect your work. Data access respects your AWS Identity and Access Management (AWS IAM) permissions. The notebook and the agent can only access data sources you’re authorized to use. Communication between components uses encrypted channels, and your notebook storage is encrypted at rest. The AI agent includes built-in guardrails to help prevent destructive operations and logs interactions for your compliance and auditing purposes.
Prerequisites
Before you begin, you need:
An AWS account with appropriate permissions to create Amazon SageMaker Unified Studio resources. See Set up IAM-based domains for complete permission requirements.
Basic familiarity with Python programming and SQL queries
Understanding of data analysis concepts and ML workflows
Access to the sample housing dataset (provided in the walkthrough)
You will be prompted either to select an existing AWS Identity and Access Management (AWS IAM) role that has access to your data and compute, or to create a new role. For this walkthrough, choose Create a new role and leave the other options at their defaults.
Choose Set up. It takes a few minutes to complete your environment.
Use case
In this post, you’ll use a Notebook and the SageMaker Data Agent to perform the following:
Working with dataset: Upload sample dataset housing.csv and explore with data explorer
Polyglot programming: Query dataframes with SQL via DuckDB
Multi-engine access via AWS Glue: Create an AWS Glue table to unlock Athena SQL/Spark engines for distributed processing
Advanced analytics: Use Athena Spark for data profiling
AI-assisted development: Generate profiling and ML code with Data Agent
ML workflow: Train Random Forest model and evaluate results
First, let’s walk through the interface and explore its core capabilities.
Understanding the interface
The Notebooks interface follows familiar notebook conventions with cells for code execution and markdown for documentation. Within the notebook, you’ll see your current programming environment (such as Python 3.11) and compute profile specifications. The interface allows you to:
Access your data by browsing files, exploring data catalogs, and managing third-party connections
Monitor variables created within your notebook context
Scale compute resources on demand by adjusting virtual CPUs and RAM based on your workload requirements, even scaling up to GPU instances
Manage packages by installing and configuring Python packages as needed
Working with the dataset
For this walkthrough, you’ll use the housing.csv sample dataset which you can download from this page. (the file is named canvas-sample-housing.csv on the linked page). Choose the Files icon in the left panel and choose the Local tab. Upload the CSV file to the notebook on the Local tab.
Notebooks provide you with instant access to your data assets. Using the data explorer, you can browse your AWS Glue Data Catalog, Amazon S3 table catalogs, Amazon S3 buckets, and configured third-party connections.
Choose the three-dot options menu.
Choose Read as dataframe, then run the inserted cell in the notebook to view the results.
import pandas as pd
<<df_csv_xxxx>> = pd.read_csv('housing.csv')
<<df_csv_xxxx>>
When you return a dataframe, Notebooks render it in a rich table format with automatic data profiling.
Polyglot programming: Python and SQL together
One of the most powerful features in Notebooks is the interoperability between Python and SQL. After you load data into a Python dataframe, you can immediately query it using SQL. For example, to calculate total population and household by ocean proximity, you can run:
select sum(population) ,sum(households),ocean_proximity
from<<df_csv_xxxx>>
group byocean_proximity
The notebook’s autocomplete functionality recognizes dataframes in your context, making SQL queries intuitive.
This SQL query runs on DuckDB (an in-memory SQL database engine), which requires no separate installation or server maintenance on your part. DuckDB’s lightweight design integrates into Python, Java, and other environments, making it ideal for your rapid interactive data analysis. For distributed processing needs, you can use engines such as Apache Spark or Trino after creating an AWS Glue table for this dataset.
Create an AWS Glue table for the dataset
After you create an AWS Glue table, you can query the dataset using various AWS Glue catalog-compatible engines, including Amazon Athena SQL (Trino) and Amazon Athena Spark. These engines deliver optimal price-performance for your specific workload requirements.
Start by creating an AWS Glue database. To do that, create a new cell in the notebook by choosing SQL and selecting Amazon Athena (SQL).
Run this SQL to create a database: create database demo;
Next, go to data explorer and choose +Add on the top left, then choose Create table. Choose the database you created earlier and enter a name for the table. Upload the housing.csv dataset file used earlier. Continue by choosing Next in the side panel to create the table.
Next, let’s run a sample SQL query in a new cell using Amazon Athena SQL:
select sum(population) , sum(households), ocean_proximity
fromdemo.housing
group by ocean_proximity
Advanced capabilities with Athena Spark
Before you can build an ML model to predict house prices, let’s analyze the dataset further and run data profiling for additional insights. For advanced exploration, you can use Amazon Athena Spark within your notebook.To do that, you’ll create a new Python cell which has a built-in Spark session. Run the following code to check the Spark version:
# Verify Spark version
spark.version
Using the SageMaker Data Agent for data profiling
Instead of writing boilerplate code manually, you can use the built-in generative AI capability.
Prompt: “Perform data profiling and create visualization for housing table”
The AI assistant generates comprehensive profiling code for you, including basic statistics calculation, column-level profiling, data type analysis, and missing value detection.
The agent accessed your AWS Glue Data Catalog, understood your housing table structure, and generated profiling code tailored to your specific columns and data types. This context awareness reduces the trial-and-error cycle you’d normally face when adapting generic code snippets to your environment. Review the generated code and run it. The fast response times help you iterate on your analysis efficiently.
If you encounter an error, you can resolve it using Fix with AI as shown in the following figure. When errors occur during execution, the “Fix with AI” feature analyzes the traceback, diagnoses the root cause, and generates corrected code, so you can keep your analysis moving forward.
Training ML models
Next, you’ll use the data agent to generate code for training a model that predicts housing prices.
Prompt: “Generate code to train a model that predicts housing prices. Use table housing.”
The AI assistant generates end-to-end code for you that:
Reads housing data from AWS Glue catalog using Amazon Athena Spark and converts to pandas
Converts string columns to numeric, encodes using one-hot encoding and removes missing values
Trains a Random Forest model to predict median house values
Evaluates model performance (RMSE, MAE, R-square)
Displays top 10 most important features for predictions
This multi-step orchestration saves you hours of development time by handling the entire workflow from data access to model evaluation.
If you encounter an error, you can resolve it using Fix with AI available in the results traceback section.
This workflow showcased Notebooks’ unified capabilities: you uploaded files locally, created AWS Glue tables for multi-engine access, used Amazon Athena Spark for distributed profiling, and used AI-assisted ML development to predict housing prices. All of this happened within a single notebook environment without switching tools.
Key benefits and best practices
Notebooks in Amazon SageMaker Unified Studio deliver several advantages:
Faster time to insights: With traditional environments, you might spend hours on configuration before analysis begins. Notebooks bypass this overhead, so you can start work immediately.
Improved collaboration: You can share notebooks with consistent environments, supporting reproducibility and reducing “works on my machine” issues.
Reduced complexity: You can access multiple data sources and compute engines from one interface rather than navigating separate tools for each data source or processing engine.
AI-accelerated development: Generate task-specific code and receive intelligent suggestions, reducing time spent on repetitive coding tasks.
Scalable performance: Handle datasets from megabytes to petabytes with appropriate compute resources. The system scales automatically as data volumes grow.
Best practices
Start with appropriate compute profiles by beginning with smaller instances and scaling up as your needs grow.
Use AI assistance with natural language prompts for your repetitive tasks and complex operations.
Combine engines strategically by using Amazon Athena Spark for your large-scale processing, Amazon Redshift for data warehousing and other specialized engines for your specific workloads.
Document your work using markdown cells to create living documentation alongside your code.
Organize using multiple cells by breaking the complex workflows into logical steps for better readability and debugging.
Cleaning up
To avoid incurring future charges, delete the resources you created in this walkthrough:
In the Amazon SageMaker Unified Studio console, navigate to the Notebook page
Delete the notebook
Delete the demo database and housing table from the AWS Glue Data Catalog
Delete Amazon SageMaker Unified Studio domain created during this walkthrough
If you created a new IAM role specifically for this walkthrough, delete it from the IAM console
Conclusion
In this post, we demonstrated how Notebooks in Amazon SageMaker Unified Studio help you work more efficiently and deliver insights more quickly. By combining familiar notebook interfaces with enterprise-scale compute, multi-engine support, and generative AI assistance, teams can streamline data and AI workflows.
The integration of Python and SQL, instant access to diverse data sources, and intelligent code generation capabilities make Notebooks a valuable tool for modern data teams. Teams can perform exploratory data analysis, build complex data pipelines, or train ML models with the flexibility and power needed within a single, intuitive environment.
Amazon Athena recently added support for reading Parquet Column Indexes in Apache Iceberg tables on November 21, 2025. With this optimization, Athena can perform page-level data pruning to skip unnecessary data within Parquet row groups, potentially reducing the amount of data scanned and improving query runtime for queries with selective filters. For data teams, this may help enable faster insights and help reduce costs when analyzing large-scale data lakes.
Data teams building data lakes often choose Apache Iceberg for its ACID transactions, schema evolution, and metadata management capabilities. Athena is a serverless query engine that allows you to query Amazon S3-based data lakes using SQL, and you don’t need to manage infrastructure. Based on the type of data and query logic, Athena can apply multiple query optimizations to improve performance and reduce costs.
In this blog post, we use Athena and Amazon SageMaker Unified Studio to explore Parquet Column Indexes and demonstrate how they can improve Iceberg query performance. We explain what Parquet Column Indexes are, demonstrate their performance benefits, and show you how to use them in your applications.
Overview of Parquet Column Indexes
Parquet Column Indexes store metadata that query engines can use to skip irrelevant data with greater precision than row group statistics alone. To understand how they work, consider how data is structured within Parquet files and how engines like Athena process them.
Parquet files organize data hierarchically by dividing data into row groups (typically 128-512 MB each) and further subdividing them into pages (typically 1 MB each). Traditionally, Parquet maintains metadata on the contents of each row group level in the form of min/max statistics, allowing engines like Athena to skip row groups that don’t satisfy query predicates. Although this approach reduces the bytes scanned and query runtime, it has limitations. If even a single page within a row group overlaps with the values you are searching for, Athena scans all pages within the row group.
Parquet Column Indexes help address this problem by storing page-level min/max statistics in the Parquet file footer. Row group statistics provide coarse-grained filtering, but Parquet Column Indexes enable finer-grained filtering by allowing query engines like Athena to skip individual pages within a row group. Consider a Parquet file with a single row group containing 5 pages for a column. The row group has min/max statistics of (1, 20), and each page for that column has the following min/max statistics.
When Athena runs a query filtering for values equal to 2, it first checks the row group statistics and confirms that 2 falls within the range (1, 20). Athena will then plan to scan the pages within that row group. Without Parquet Column Indexes, Athena scans each of the 5 pages in the row group. With Parquet Column Indexes, Athena examines the page-level statistics and determines that only page-0 and page-1 need to be read, skipping the remaining 3 pages.
How to use Parquet Column Indexes with Athena
Athena uses Parquet Column Indexes based on table type:
Amazon S3 Tables: Athena automatically uses Parquet Column Indexes by default when they are present.
Iceberg tables in S3 general purpose buckets: Athena does not use Parquet Column Indexes by default. To allow Athena to use Parquet Column Indexes, add an AWS Glue table property named use_iceberg_parquet_column_index and set it to true. Use the AWS Glue console or AWS Glue UpdateTable API to perform these actions.
Measuring Athena performance gains when using Parquet Column Indexes
Now that we understand what Parquet Column Indexes are, we’ll demonstrate the performance benefits of using Parquet Column Indexes by analyzing the catalog_sales table from a 3TB TPC-DS dataset. This table contains ecommerce transaction data including order dates, sales amounts, customer IDs, and product information. This dataset is a good proxy for the types of business analysis that you might perform on your own data, such as identifying sales trends, analyzing customer purchasing patterns, and calculating revenue metrics. We compare query execution statistics with and without Parquet Column Indexes to quantify the performance improvement.
Prerequisites
Before you begin, you must have the following resources:
A SageMaker Unified Studio IAM-based domain.
An Execution IAM Role configured within the SageMaker Unified Studio IAM-based domain with access to S3, AWS Glue Data Catalog, and Athena.
An S3 bucket in your account to store Iceberg table data and Athena query results.
Create catalog_sales Iceberg table
Complete the following steps using SageMaker Unified Studio notebooks. There, you can use SageMaker Unified Studio’s multi-dialect notebook functionality to work with your data using the Athena SQL and Spark engines. To create a catalog_sales Iceberg table in your account, follow these steps:
Navigate to Amazon SageMaker in the AWS Management Console and choose Open under Get started with Amazon SageMaker Unified Studio.
From the side navigation, select Notebooks and choose Create Notebook. The subsequent steps in this post will execute scripts in this notebook.
Create a new SQL cell in the notebook and set the connection type to Athena (Spark). Execute the following query to create a database for the tables in this post.
CREATE DATABASE parquet_column_index_blog;
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to create a Hive table pointing to the location of the TPC-DS catalog_sales table data at the public S3 bucket.
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to add the Hive partitions to the AWS Glue metadata.
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Replace s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/catalog_sales/ with the S3 URI where you want to store your Iceberg table data, then execute the following query to create the catalog_sales Iceberg table from the Hive table.
CREATE TABLE parquet_column_index_blog.catalog_sales
USING iceberg
PARTITIONED BY (cs_sold_date_sk)
LOCATION 's3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/catalog_sales/'
AS
SELECT * FROM parquet_column_index_blog.catalog_sales_hive;
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to delete the catalog_sales_hive table, which was only needed to create the catalog_sales Iceberg table.
DROP TABLE parquet_column_index_blog.catalog_sales_hive;
Run an Athena query without Parquet Column Indexes
After creating the catalog_sales Iceberg table in the preceding steps, we run a simple query that analyzes shipping delays of the top 10 most ordered items. This type of analysis could be critical for ecommerce and retail operations. By identifying which popular items experience the greatest delays, fulfillment teams can focus resources where they matter most. For example, you can adjust inventory placement, change warehouse assignments, or address carrier issues. Additionally, popular items with significant shipping delays are more likely to result in order cancellations or returns, so proactively identifying these issues helps protect revenue.
SELECT cs_item_sk,
SUM(cs_quantity) as total_orders,
AVG(cs_ship_date_sk - cs_sold_date_sk) as avg_ship_delay_days,
MIN(cs_ship_date_sk - cs_sold_date_sk) as min_ship_delay,
MAX(cs_ship_date_sk - cs_sold_date_sk) as max_ship_delay,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) as late_shipments,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) * 100.0 / COUNT(*) as late_shipment_pct,
AVG(cs_ext_ship_cost) as avg_shipping_cost
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IN (
SELECT cs_item_sk
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY SUM(cs_quantity) DESC
LIMIT 10
)
AND cs_ship_date_sk IS NOT NULL
AND cs_sold_date_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY avg_ship_delay_days DESC;
Additionally, this query is a good candidate for demonstrating the effectiveness of using Parquet Column Indexes because it has a selective filter predicate on a single column cs_item_sk. When Athena executes this query, it first identifies row groups whose min/max ranges overlap with the top 10 most ordered items. Without using Parquet Column Indexes, Athena has to scan every page of data within those matched row groups. However, when using Parquet Column Indexes, Athena can prune data further by skipping individual pages within those row groups whose min/max ranges do not overlap with the ids. Complete the following steps to establish baseline query performance when Athena does not use Parquet Column Indexes during the query.
Create a new Python cell in the notebook. Replace s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/query_results/ with the S3 URI where you want to store your Athena query results, then execute the following script. Note the runtime and bytes scanned that will be printed. The script will run the query five times with query result reuse disabled and chooses the minimum runtime and the corresponding bytes scanned among those iterations. See our numbers in the Run Athena query with Parquet Column Indexes section.
import boto3
import time
# Configuration
DATABASE = "parquet_column_index_blog"
OUTPUT_LOCATION = "s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/query_results/"
def run_athena_query(query: str, database: str, output_location: str):
athena_client = boto3.client('athena')
response = athena_client.start_query_execution(
QueryString=query,
QueryExecutionContext={'Database': database},
ResultConfiguration={'OutputLocation': output_location}
)
query_execution_id = response['QueryExecutionId']
while True:
result = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
state = result['QueryExecution']['Status']['State']
if state in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
break
time.sleep(5)
if state != 'SUCCEEDED':
raise Exception(f"Query failed with state: {state}")
stats = result['QueryExecution']['Statistics']
return {
'execution_time_sec': stats['EngineExecutionTimeInMillis'] / 1000,
'data_scanned_gb': stats['DataScannedInBytes'] / (1024 ** 3)
}
def benchmark_query(query: str, database: str, output_location: str, num_runs: int = 5):
results = []
for i in range(num_runs):
stats = run_athena_query(query, database, output_location)
results.append(stats)
best_run = min(results, key=lambda r: r['execution_time_sec'])
execution_time = round(best_run['execution_time_sec'], 1)
data_scanned = round(best_run['data_scanned_gb'], 1)
print(f"Execution time: {execution_time} sec")
print(f"Data scanned: {data_scanned} GB")
QUERY = """
SELECT cs_item_sk,
SUM(cs_quantity) as total_orders,
AVG(cs_ship_date_sk - cs_sold_date_sk) as avg_ship_delay_days,
MIN(cs_ship_date_sk - cs_sold_date_sk) as min_ship_delay,
MAX(cs_ship_date_sk - cs_sold_date_sk) as max_ship_delay,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) as late_shipments,
SUM(
CASE
WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
END
) * 100.0 / COUNT(*) as late_shipment_pct,
AVG(cs_ext_ship_cost) as avg_shipping_cost
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IN (
SELECT cs_item_sk
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY SUM(cs_quantity) DESC
LIMIT 10
)
AND cs_ship_date_sk IS NOT NULL
AND cs_sold_date_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY avg_ship_delay_days DESC;
"""
# Run benchmark
benchmark_query(QUERY, DATABASE, OUTPUT_LOCATION, num_runs=5)
Sort the catalog_sales table
Before rerunning the query with Athena using Parquet Column Indexes, you need to sort the catalog_sales table by the cs_item_sk column. In the preceding query, there is a dynamic filter as a subquery on the cs_item_sk column:
cs_item_sk IN (
SELECT cs_item_sk
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY SUM(cs_quantity) DESC
LIMIT 10
)
When executing this query, Athena pushes down the filter predicate to the data source level, fetching only rows that match the top 10 most ordered items. To maximize page pruning with Parquet Column Indexes, rows with the same cs_item_sk values should be stored near each other in the Parquet file. Without sorting, matching values could be scattered across many pages, forcing Athena to read more data. Sorting the table by cs_item_sk clusters similar values together, enabling Athena to read fewer pages.
Let’s examine the Parquet Column Indexes in one of the Parquet files to understand how the data in the catalog_sales table is currently organized. First, download the Parquet file from the cs_sold_date_sk = 2450815 partition and install the open-source parquet-cli tool on your local machine. Replace <local-path-to-parquet-file> with the path to the downloaded Parquet file, then run the following command on your local machine:
parquet column-index <local-path-to-parquet-file>
This displays Parquet Column Indexes for all columns. For brevity, only the first 11 pages of the cs_item_sk column from the first row group are shown in the following example:
Notice that nearly every page contains a wide range of values. This overlap means Athena cannot eliminate pages when filtering with Parquet Column Indexes on cs_item_sk. For example, searching for cs_item_sk = 100 requires scanning each of the 11 pages because the value 100 falls within every page’s min/max range. With this overlap, enabling Athena to use Parquet Column Indexes would provide no performance benefit. Sorting the data by cs_item_sk eliminates this overlap, creating distinct, non-overlapping ranges for each page. To make Parquet Column Indexes more effective, sort the table by completing the following step:
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the query to sort the cs_item_sk column values of the catalog_sales table in ascending order and to put all the null values in the last few Parquet pages. New Iceberg data files will be generated from this query.
Running the parquet column-index command on the sorted data file from the cs_sold_date_sk = 2450815 partition shows that the Parquet Column Indexes are now sorted and have non-overlapping ranges. The first 11 pages of the cs_item_sk column from the first row group are shown in the following example:
Now when searching for cs_item_sk = 100, Athena only needs to read page-0, skipping the remaining 10 pages entirely.
Run Athena query with Parquet Column Indexes
Now that the data is sorted to eliminate overlapping pages within the row groups for the cs_item_sk column, we run two experiments on the sorted data. The first measures the impact of sorting alone, and the second measures the combined effect of sorting with Parquet Column Indexes.
Create a new Python cell in the notebook. Execute the same script in the section Run Athena query without Parquet Column Indexes and take note of the query runtime and bytes scanned results. This measures the performance of querying sorted data without using Parquet Column Indexes.
Create a new Python cell in the notebook. Execute the following Python script to set the use_iceberg_parquet_column_index table property to true for the catalog_sales table in the AWS Glue Data Catalog.
import boto3
def add_iceberg_parquet_column_index(database_name: str, table_name: str):
glue_client = boto3.client('glue')
# Get current table definition
response = glue_client.get_table(DatabaseName=database_name, Name=table_name)
table = response['Table']
# Build TableInput with only allowed fields
table_input = {'Name': table['Name']}
allowed_fields = [
'Description', 'Owner', 'LastAccessTime', 'LastAnalyzedTime',
'Retention', 'StorageDescriptor', 'PartitionKeys', 'ViewOriginalText',
'ViewExpandedText', 'TableType', 'Parameters', 'TargetTable'
]
for field in allowed_fields:
if field in table:
table_input[field] = table[field]
# Add the property
if 'Parameters' not in table_input:
table_input['Parameters'] = {}
table_input['Parameters']['use_iceberg_parquet_column_index'] = 'true'
# Update the table
glue_client.update_table(DatabaseName=database_name, TableInput=table_input)
# Usage
add_iceberg_parquet_column_index("parquet_column_index_blog", "catalog_sales")
Create a new Python cell in the notebook. Execute the same script in the section Run Athena query without Parquet Column Indexes and take note of the query runtime and bytes scanned results. This measures the performance of querying sorted data using Parquet Column Indexes.
Athena query time and bytes scanned improvement
The following table summarizes the results from each experiment. The percentage improvements for the sorted experiments are measured against the unsorted baseline.
Experiment
Runtime (sec)
Bytes Scanned (GB)
Unsorted without Parquet Column Indexes
20.6
45.2
Sorted without Parquet Column Indexes
15.4 (25.2% faster)
27.8 (38.5% fewer bytes)
Sorted with Parquet Column Indexes
10.3 (50.0% faster)
13.0 (71.2% fewer bytes)
Recommendations
To maximize Athena’s ability to use Parquet Column Indexes and achieve optimal query performance, we recommend the following.
Sort data by frequently filtered columns. This allows Athena to efficiently read Parquet Column Indexes and skip irrelevant pages, potentially reducing scan time. When data is sorted by a filter column, similar values are clustered together within pages. Because Parquet Column Indexes store min/max values for each page, Athena can quickly determine which pages contain matching values and skip the rest.
Sort data by high-cardinality columns. This creates distinct value ranges between pages, maximizing the opportunity for Athena to skip pages during query execution. High-cardinality (many distinct values) columns produce non-overlapping min/max ranges across pages, allowing Athena to more effectively filter out irrelevant pages. In contrast, low-cardinality columns such as boolean or status fields result in overlapping ranges across many pages, reducing the number of skipped pages.
Clean up
When you have finished the steps in this post, complete the following cleanup actions to avoid incurring ongoing charges:
Create a new SQL cell in the notebook and set the connection type to Athena (Spark). Execute the following command to drop the parquet_column_index_blog database and the catalog_sales table.
DROP DATABASE parquet_column_index_blog CASCADE;
Delete the Iceberg table data and the Athena query results from your S3 bucket.
Delete the SageMaker Unified Studio IAM-based domain if it is no longer needed.
Conclusion
In this post, we showed you how Athena uses Parquet Column Indexes to speed up queries and reduce the number of bytes scanned. By using Parquet Column Indexes, Athena can skip irrelevant data pages to improve query performance, especially for queries with selective filters on sorted data. Refer to Optimize Iceberg tables to learn more about this feature and try it out on your own queries.
Developers working with Amazon Managed Streaming for Apache Kafka (Amazon MSK) regularly need to make decisions that require deep operational context—choosing the right instance type, diagnosing consumer lag, or planning for a traffic spike. Answering these questions means piecing together documentation, metrics, and operational know-how.
What if your IDE could guide you through that workflow with built-in domain expertise and tooling? Kiro is an AI-powered agentic IDE that lets you describe what you need in natural language. Whether it’s infrastructure configuration or operational troubleshooting, Kiro guides you through the solution.
In this post, we’ll show you how to use Kiro powers, a new capability that equips Kiro with contextual knowledge and tooling. You can simplify your MSK cluster management, from initial setup to diagnosing common issues, all through natural language conversations.
Challenges operating your MSK Express broker cluster
Amazon MSK Express Brokers are a fully managed offering where AWS handles much of the underlying infrastructure. However, platform teams still need to correctly size clusters based on throughput requirements. They also need to understand the right Amazon CloudWatch metrics during performance issues and investigate when CPU usage or replication lag is higher than expected. MSK best practices documentation spans multiple AWS guides. This makes it time-consuming to find relevant information during production incidents. New team members face a learning curve with MSK operations and can repeat common sizing and configuration mistakes.
Although Express Brokers simplify infrastructure management, you still face operational challenges that require deep Kafka expertise across three areas:
Cluster creation and sizing: You must still select the right instance type, configure networking, and choose authentication methods. These decisions impact cost and performance from day one.
Observability and troubleshooting: Effective operations require correlating broker, partition, and client metrics. Troubleshooting lag or replication issues still requires a solid understanding of Express Brokers’ architecture.
Capacity management: You must monitor CPU usage, understand per-broker throughput limits, and scale before hitting throttling thresholds.
These challenges mean that setting up an MSK cluster, analyzing slow-running clients, or investigating high-CPU load requires pulling together documentation, configuration details, CLI tooling, and operational know-how, which is often spread across multiple sources. Kiro powers address these challenges by bringing best practices, guided workflows, and tooling directly into your IDE, reducing the expertise barrier and the time spent context-switching between documentation, consoles, and the CLI.
Kiro powers
Kiro powers is a feature that combines best practices, specialized context, and tool integrations into a single capability. You can install powers with one click in the Kiro IDE or add them from a public GitHub URL. Each Power combines the following components:
Model Context Protocol (MCP) servers give your Kiro agent direct access to your infrastructure. The AWS MSK MCP server, for example, exposes tools to create clusters, monitor health, and optimize configurations.
Steering files provide persistent knowledge and workflow guides that Kiro loads based on the user’s task, such as monitoring best practices or troubleshooting workflows.
Optional hooks run automated actions when IDE events occur, such as validating configurations before deployment.
The key advantage of Kiro powers is that they load context dynamically based on the user’s task. Instead of configuring every MCP server upfront and re-providing context in each conversation, powers activate the right tools and knowledge on demand. This keeps your agent’s context focused and relevant. In the next section, we look at how these components work together specifically for MSK Express Broker operations.
The MSK Express broker power
The MSK Express broker power packages the AWS MSK MCP server with targeted streaming operations guidance, giving your Kiro agent expertise for MSK Express Broker operations and cluster management. You can use it to build Kafka-based streaming applications through Kiro while maintaining Express broker best practices throughout the development lifecycle.
For cluster operations, you can create Express broker clusters, monitor health metrics, and manage configurations through natural language. You can retrieve cluster metadata, check broker endpoints, and verify replication status. The Power also supports operational monitoring. You can track CPU utilization, throughput limits, partition distribution, and AWS Identity and Access Management (IAM) connection metrics.
To see how this works in practice, here’s what happens when you interact with the Power: When you ask Kiro to create an MSK cluster, the Power recommends appropriate instance sizes based on your throughput requirements. When you’re troubleshooting, it knows to check LeaderCount before diving into network metrics. When you’re troubleshooting authentication failures, it recommends client settings like reconnect.backoff.ms and group.instance.id to resolve connection churn and rebalancing issues against Express broker limits. Use cases include:
Cluster sizing and creation: Describe your throughput requirements (for example, “50 MBps ingress with 3x fan-out”) and the Power calculates the right instance type and broker count, then walks through cluster creation.
Proactive health monitoring: Ask Kiro to review your cluster. It checks CPU against the 60% threshold, compares throughput to instance limits, and flags partition imbalances and throughput bottlenecks before they become incidents.
Incident troubleshooting: Consumer lag spiking? The Power checks the relevant metrics, identifies the root cause (like skewed partition leadership), and guides you through resolution.
Capacity planning: Preparing for a traffic spike? The Power analyzes current utilization against instance limits and recommends whether to scale up or add brokers.
The MSK Express broker power brings together documentation, metrics, and operational context so your Kiro agent can correlate findings and help identify root causes specific to your infrastructure.
Getting started with the MSK Express broker power
Starting with Kiro powers takes only a few clicks in the Kiro IDE. You can install from the built-in marketplace or import from a public GitHub URL. Kiro packages all components and makes them available to the Kiro agent.
To set up the MSK Express broker power, follow these steps:
Choose the Powers icon in the Kiro sidebar
In the AVAILABLE panel, scroll down to Build and Operate MSK Express Broker
The MSK Express broker power streamlines Kafka operations by combining Model Context Protocol (MCP) servers with operational guidance. With natural language interactions, you can create clusters, monitor health, optimize configurations, and troubleshoot issues without reviewing extensive documentation.
Performance issues in analytics environments often remain invisible until they disrupt dashboards, delay ETL jobs, or impact business decisions. For teams running Amazon Redshift Serverless, unmonitored query queues, long-running queries, or unexpected spikes in compute capacity can degrade performance and increase costs if left undetected.
Amazon Redshift Serverless streamlines running analytics at scale by removing the need to provision or manage infrastructure. However, even in a serverless environment, maintaining visibility into performance and usage is essential for efficient operation and predictable costs. While Amazon Redshift Serverless provides advanced built-in dashboards for monitoring performance metrics, delivering notifications directly to platforms like Slack, brings another level of agility. Real-time alerts in the team’s workflow enable faster response times and more informed decision-making without requiring constant dashboard monitoring.
In this post, we show you how to build a serverless, low-cost monitoring solution for Amazon Redshift Serverless that proactively detects performance anomalies and sends actionable alerts directly to your selected Slack channels. This approach helps your analytics team identify and address issues early, often before your users notice a problem.
Solution overview
The solution presented in this post uses AWS services to collect key performance metrics from Amazon Redshift Serverless, evaluate them against thresholds that you can flexibly configure, and notify you when anomalies are detected.
The workflow operates as follows:
Scheduled execution – An Amazon EventBridge rule triggers an AWS Lambda function on a configurable schedule (by default, every 15 minutes during business hours).
Metric collection – The AWS Lambda function gathers metrics including queued queries, running queries, compute capacity (RPUs), data storage usage, table count, database connections, and slow-running queries using Amazon CloudWatch and the Amazon Redshift Data API.
Threshold evaluation – Collected metrics are compared against your predefined thresholds that reflect acceptable performance and usage limits.
Alerting – When a threshold is exceeded, the Lambda function publishes a notification to an Amazon SNS topic.
Observability – Lambda execution logs are stored in Amazon CloudWatch Logs for troubleshooting and auditing.
This architecture is fully serverless and requires no changes to your existing Amazon Redshift Serverless workloads. To simplify deployment, we provide an AWS CloudFormation template that provisions all required resources.
Prerequisites
Before deploying this solution, you must collect information about your existing Amazon Redshift Serverless workgroup and namespace that you want to monitor. To identify your Amazon Redshift Serverless resources:
Provide details for your existing Amazon Redshift Serverless environment. These values connect the monitoring solution to your Redshift environment. Some parameters come with the default values that you can replace with your actual configuration.
Parameter
Default value
Description
Amazon Redshift Workgroup Name
Your Amazon Redshift Serverless workgroup name.
Amazon Redshift Namespace Name
Your Amazon Redshift Serverless namespace name.
Amazon Redshift Workgroup ID
Workgroup ID (UUID) of the Amazon Redshift Serverless workgroup to monitor. Must follow the UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (lowercase hexadecimal with hyphens).
Namespace ID (UUID) of the Amazon Redshift Serverless namespace. Must follow the UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (lowercase hexadecimal with hyphens).
Database Name
dev
Target Amazon Redshift database for SQL-based diagnostic and monitoring queries.
Monitoring schedule
The default schedule runs diagnostic SQL queries every 15 minutes during business hours, balancing responsiveness and cost efficiency. Running more frequently might increase costs, while less frequent monitoring could delay detection of performance issues. You can adjust this schedule to your actual need.
Parameter
Default value
Description
Schedule Expression
cron(0/15 8-17 ? * MON-FRI *)
EventBridge schedule expression for Lambda function execution. Default runs every 15 minutes, Monday through Friday, 8 AM to 5 PM UTC.
Threshold configuration
Thresholds should be tuned based on your workload characteristics.
Parameter
Default value
Description
Queries Queued Threshold
20
Alerts threshold for queued queries.
Queries Running Threshold
20
Alerts threshold for running queries.
Compute Capacity Threshold (RPUs)
64
Alert threshold for compute capacity (RPUs).
Data Storage Threshold (MB)
5242880
Threshold for data storage in MB (default 5 TB).
Table Count Threshold (MB)
1000
Alerts threshold for total table count.
Database Connections Threshold
50
Alert threshold for database connections.
Slow Query Threshold (seconds)
10
Thresholds in seconds for slow query detection.
Query Timeout (Seconds)
30
Timeout for SQL diagnostics queries.
Tip: Start with conservative thresholds and refine them after observing baseline behavior for one to two weeks.
Lambda configuration
Configure the AWS Lambda function settings. The selected default values are appropriate for most monitoring scenarios. You may want to change them only in case of troubleshooting.
If your organization has network isolation requirements, you can optionally enable VPC deployment for the Lambda function. When enabled, the Lambda function runs within your specified VPC subnets, providing network isolation and allowing access to VPC-only resources.
Parameter
Default value
Description
VPC ID
VPC ID for Lambda deployment (required if EnableVPC is true). The Lambda function will be deployed in this VPC. Ensure that the VPC has appropriate routing (NAT Gateway or VPC Endpoints) to allow Lambda to access AWS services like CloudWatch, Amazon Redshift, and Amazon SNS.
VPC Subnet IDs
Comma-separated list of subnet IDs for Lambda deployment (required if EnableVPC is true).
Security Group IDs
Comma-separated list of security group IDs for Lambda (optional). If not provided and EnableVPC is true, a default security group will be created with outbound HTTPS access. Custom security groups must allow outbound HTTPS (port 443) to AWS service endpoints.
Note that VPC deployment might increase cold start times and requires an NAT Gateway or VPC endpoints for AWS service access. We recommend provisioning interface VPC endpoints (through AWS PrivateLink) for the five services the Lambda function calls which keeps all traffic private without the recurring cost of a NAT Gateway.
Security configuration – Encryption
If your organization requires encryption of data at rest, you can optionally enable AWS Key Management Service (AWS KMS) encryption for the Lambda function’s environment variables, CloudWatch Logs, and SNS topic. When enabled, the template encrypts each resource using the AWS KMS keys that you provide, either a single shared key for all three services, or individual keys for granular key management and audit separation.
Parameter
Default value
Description
Shared KMS Key ARN
AWS KMS key ARN to use for all encryption (Lambda, Logs, and SNS) unless service-specific keys are provided. This streamlines key management by using a single key for all services. The key policy must grant encrypt/decrypt permissions to Lambda, CloudWatch Logs, and SNS.
Lambda KMS Key ARN
AWS KMS key ARN for Lambda environment variable encryption (optional, overrides SharedKMSKeyArn). Use this for separate key management per service. The key policy must grant decrypt permissions to the Lambda execution role. If not provided, SharedKMSKeyArn will be used when EnableKMSEncryption is true.
CloudWatch Logs KMS Key ARN
AWS KMS key ARN for CloudWatch Logs encryption (optional, overrides SharedKMSKeyArn). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to the CloudWatch Logs service. If not provided, SharedKMSKeyArn will be used when EnableKMSEncryption is true.
SNS Topic KMS Key ARN
AWS KMS key ARN for SNS topic encryption (optional, overrides SharedKMSKeyArn). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to SNS service and the Lambda execution role. If not provided, SharedKMSKeyArn will be used when EnableKMSEncryption is true.
Enable Dead Letter Queue
False
Optionally enable Dead Letter Queue (DLQ) for failed Lambda invocations to improve reliability and security monitoring. When enabled, events that fail after all retry attempts will be sent to an SQS queue for investigation and potential replay. This helps prevent data loss, provides visibility into failures, and enables security audit trails for monitoring anomalies. The DLQ retains messages for 14 days.
Note that AWS KMS encryption requires the key policy to grant appropriate permissions to each consuming service (Lambda, CloudWatch Logs, and SNS).
On the review page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names.
Choose Submit.
Resources created
The CloudFormation stack creates the following resources:
EventBridge rule for scheduled execution
AWS Lambda function (Python 3.12 runtime)
Amazon SNS topic for alerts
IAM role with permissions for CloudWatch, Amazon Redshift Data API, and SNS
CloudWatch Log Group for Lambda logs
Note: CloudFormation deployment typically takes 10–15 minutes to complete. You can monitor progress in real time under the Events tab of your CloudFormation stack.
Post-deployment configuration
After the CloudFormation stack has been successfully created, complete the following steps.
Note the values for LambdaRoleArn and SNSTopicArn. You will need these in subsequent steps.
Step 2: Grant Amazon Redshift permissions
Grant permissions to the Lambda function to query Amazon Redshift system tables for monitoring data. Complete the following steps to grant the necessary access:
In the left navigation pane, choose Query Editor V2.
Connect to your Amazon Redshift Serverless workgroup.
Execute the following SQL commands, replacing <IAM Role ARN> with the LambdaRoleArn value from your CloudFormation outputs:
CREATE USER "IAMR:<IAM Lambda Role>" WITH PASSWORD DISABLE;
GRANT ROLE "sys:monitor" TO "IAMR:<IAM Role>";
These commands create an AmazonRedshift user associated with the Lambda IAM role and grant it the sys:monitor Amazon Redshift role. This role provides read-only access to catalog and system tables without granting permissions to user data tables.
Step 3: Configure Slack notifications
Amazon Q Developer in chat applications provides native AWS integration and managed authentication, removing custom webhook code and reducing setup complexity. To receive alerts in Slack, configure Amazon Q Developer in Chat Applications to connect your SNS topic to your preferred Slack channel:
When configuring the Slack channel, ensure that you select the correct AWS Region where you deployed the CloudFormation stack.
In the Notifications section, select the SNS topic created by your CloudFormation stack (refer to the SNSTopicArn output value).
Keep the default IAM read-only permissions for the channel configuration.
After configured, alerts automatically appear in Slack whenever thresholds are exceeded.
Cost considerations
With the default configuration, this solution incurs minimal ongoing costs. The Lambda function executes approximately 693 times per month (every 15 minutes during an 8-hour business day, Monday through Friday), resulting in a monthly cost of approximately $0.33 USD. This includes Lambda compute costs ($0.26) and CloudWatch GetMetricData API calls ($0.07). All other services (EventBridge, SNS, CloudWatch Logs, and Amazon Redshift Data API). The Amazon Redshift Data API has no additional charges beyond the minimal Amazon Redshift Serverless RPU consumption for the Amazon Redshift Serverless system table query execution. You can reduce costs by decreasing the monitoring frequency (such as, every 30 minutes) or increase responsiveness by running more frequently (such as, every 5 minutes) with a proportional cost increase.
All costs are estimates and may vary based on your environment. Variations often occur because queries scanning system tables may take longer or require additional resources depending on the system complexity
Security best practices
This solution implements the following security controls:
IAM policies scoped to specific resource ARNs for the Amazon Redshift workgroup, namespace, SNS topic, and log group.
Data API statement access restricted to the Lambda function’s own IAM user ID.
Read-only sys:monitor database role for operational metadata access. Limit to the role created by the CloudFormation template.
Reserved concurrent executions capped at five.
To further strengthen your security posture, consider the following enhancements:
Enable EnableKMSEncryption to encrypt environment variables, logs, and SNS messages at rest.
Enable EnableVPC to deploy the function within a VPC for network isolation.
Audit access through AWS CloudTrail.
Important: This is sample code for non-production usage. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. This solution demonstrates monitoring capabilities but requires additional security hardening for production environments, including encryption configuration, IAM policy scoping, VPC deployment, and comprehensive testing.
Clean up
To remove all resources and avoid ongoing charges if you don’t want to use the solution anymore:
Remove the Slack integration from Amazon Q Developer in chat applications.
Troubleshooting
If no metrics or incomplete SQL diagnostics are returned, verify that the Amazon Redshift Serverless workgroup is active with recent query activity, and ensure the database user has the sys:monitor role (GRANT ROLE sys:monitor TO <user>) in the query editor. Without this role, queries execute successfully but only return data visible to that user’s permissions rather than the full cluster activity.
For VPC-deployed functions that fail to reach AWS services, confirm that VPC endpoints or a NAT Gateway are configured for CloudWatch, Amazon Redshift Data API, Amazon Redshift Serverless, SNS, and CloudWatch Logs.
If the Lambda function times out, increase the LambdaTimeout and QueryTimeoutSeconds parameters. The default timeout of 240 seconds accommodates most workloads, but clusters with many active queries may require additional time for SQL diagnostics to complete.
Conclusion
In this post, we showed how you can build a proactive monitoring solution for Amazon Redshift Serverless using AWS Lambda, Amazon CloudWatch, and Amazon SNS with Slack integration. By automatically collecting metrics, evaluating thresholds, and delivering alerts in near real time to Slack or your preferred collaborative platform, this solution helps detect performance and cost issues early. Because the solution itself is serverless, it aligns with the operational simplicity goals of Amazon Redshift Serverless—scaling automatically, requiring minimal maintenance, and delivering high value at low cost. You can extend this foundation with additional metrics, diagnostic logic, or alternative notification channels to meet your organization’s needs.
To learn more, see the Amazon Redshift documentation on monitoring and performance optimization.
Amazon SageMaker HyperPod offers an end-to-end experience supporting the full lifecycle of AI development—from interactive experimentation and training to inference and post-training workflows. The SageMaker HyperPod Inference Operator is a Kubernetes controller that manages the deployment and lifecycle of models on HyperPod clusters, offering flexible deployment interfaces (kubectl, Python SDK, SageMaker Studio UI, or HyperPod CLI), advanced autoscaling with dynamic resource allocation, and comprehensive observability that tracks critical metrics like time-to-first-token, latency, and GPU utilization.
Deploying inference workloads on Kubernetes-native infrastructure has traditionally required AI teams to navigate a maze of Helm charts, IAM role configurations, dependency management, and manual upgrades — often taking hours before a single model can serve predictions. Today, we’re announcing the Amazon SageMaker HyperPod Inference Operator as a native EKS add-on, enabling one-click installation and managed upgrades directly from the SageMaker console. This eliminates the need for manual Helm charts, complex IAM configuration tweaks, and downtime during upgrades.
In this post, we walk through the new installation experience, demonstrate three deployment methods (console, CLI, and Terraform), and show how features like multi-instance-type deployment and native node affinity give you fine-grained control over inference scheduling
Simplified installation experience
The new installation experience addresses three key customer scenarios with streamlined workflows:
New HyperPod clusters: Automatic installation
When creating new HyperPod clusters through the SageMaker console’s Quick Setup or Custom Setup workflows, the Inference Operator along with necessary dependencies is now installed through EKS add-on automatically as part of the cluster creation process. This eliminates the need for post-deployment configuration and ensures your cluster is ready for model deployments immediately upon creation along with one click upgrades.
Existing clusters: One-click installation
For existing HyperPod clusters, customers can install the Inference Operator with a single click through the SageMaker console. The installation automatically:
Creates required IAM roles with appropriate trust relationships and permissions
The EKS add-on integration provides standardized version management with one-click upgrades through the AWS console or CLI. This ensures customers can easily adopt new features and security updates without complex manual procedures.
The below prerequisite resources are needed to be setup before installing the Inference operator add-on. These prerequisites will be setup if SageMaker AI console is used to setup Inference operator. However, if EKS cli or console is used, these prerequisites will need to be created manually and passed to the add-on through configuration parameters. We discuss these approaches in Installation Methods.
The SageMaker console provides the most streamlined experience with two installation options:
Quick install: Automatically creates all required resources with optimized defaults, including IAM roles, S3 buckets, and dependency add-ons. This option is ideal for getting started quickly with minimal configuration decisions.
Custom install: Provides flexibility to specify existing resources or customize configurations while maintaining the one-click experience. Customers can choose to reuse existing IAM roles, S3 buckets, or dependency add-ons based on their organizational requirements.
Prerequisites
An existing Amazon SageMaker HyperPod cluster with EKS orchestration
IAM permissions for EKS cluster administration
kubectl configured for cluster access
Installation steps
Navigate to the SageMaker Console: Go to HyperPod Clusters → Cluster Management
Select Your Cluster: Choose the cluster where you want to install the Inference Operator
Choose Installation Type: Navigate to Inference tab. Select Quick Install for automated setup or Custom Install for configuration flexibility
Configure Options: If choosing Custom Install, specify existing resources or customize settings as needed
Install: Choose Install to begin the automated installation process
Verify: Check the installation status through the console, or by running kubectl get pods -n hyperpod-inference-system, or by checking the add-on status with aws eks describe-addon --cluster-name CLUSTER-NAME --addon-name amazon-sagemaker-hyperpod-inference --region REGION
Method 2: Install SageMaker HyperPod Inference add-on through EKS APIs
For customers preferring command-line workflows, the Inference Operator can be installed directly using the EKS CLI. Note that all prerequisite resources (IAM roles, S3 buckets, VPC endpoints) and dependency add-ons must be created manually before installing the Inference Operator add-on. For detailed setup instructions, see the installation guide.
Method 3: Install SageMaker HyperPod Inference add-on through Terraform deployment
Organizations utilizing Terraform for Infrastructure as Code (IaC) can deploy HyperPod clusters using the provided modules in the awesome-distributed-training GitHub repository.
To enable the HyperPod inference operator, set the create_hyperpod_inference_operator_module variable to true within your custom.tfvars file, as shown below:
The HyperPod inference operator includes several additional dependencies, which are enabled by default but can be toggled off if they already exist on your EKS cluster:
Teams can now deploy their first inference endpoint within minutes of cluster creation, compared to the previous multi-hour setup process. This acceleration enables faster experimentation and reduces the barrier to adoption for new teams.
Reduced complexity
The new installation experience eliminates the need to manually create and configure multiple AWS resources. Previously, customers needed to create IAM roles, policies, S3 buckets, VPC endpoints, and install multiple Kubernetes operators. Now, a single action handles all these requirements automatically.
Consistent configuration
Automated resource creation ensures consistent, secure configurations across environments. The installation process follows AWS best practices for IAM permissions, network security, and resource naming conventions.
Simplified upgrades
EKS Add-on integration provides standardized upgrade paths with rollback capabilities. Customers can confidently adopt new features and security updates through the familiar AWS console or CLI interfaces.
Advanced features integration
The simplified installation experience seamlessly integrates with advanced HyperPod inference capabilities:
Managed tiered KV cache
During installation, customers can optionally enable managed tiered KV cache with intelligent memory allocation based on instance types. This feature can reduce inference latency by up to 40% for long-context workloads while optimizing memory utilization across the cluster.
Intelligent routing
The installation automatically configures intelligent routing capabilities with multiple strategies (prefix-aware, KV-aware, round-robin) to maximize cache efficiency and minimize inference latency based on workload characteristics.
Observability integration
Built-in integration with HyperPod Observability provides immediate visibility into inference metrics, cache performance, and routing efficiency through Amazon Managed Grafana dashboards.
Deploying your first model
Once the add-on is installed, you can deploy models using the InferenceEndpointConfig or JumpStart models custom resources. Here’s an example configuration for deploying a Llama model:
Multi-Instance Type Deployment HyperPod Inference supports multi-instance type deployment, enhancing deployment reliability and resource utilization. You can specify a prioritized list of instance types in your deployment configuration, and the system automatically selects from available alternatives when your preferred instance type lacks capacity. The Kubernetes scheduler evaluates instance types in priority order using node affinity rules based scheduling, seamlessly placing workloads on the highest-priority available instance type. In the example below, when deploying a model from S3, ml.p4d.24xlarge has the highest priority and will be selected first if memory capacity is available. If ml.p4d.24xlarge is unavailable, the scheduler automatically falls back to ml.g5.24xlarge, and finally to ml.g5.8xlarge as the last resort.
This is implemented using Kubernetes node affinity rules with requiredDuringSchedulingIgnoredDuringExecution to restrict scheduling to the specified instance types, and preferredDuringSchedulingIgnoredDuringExecution with descending weights to enforce priority ordering.
Node affinity For scenarios requiring more granular scheduling control — such as excluding spot instances, preferring specific availability zones, or targeting nodes with custom labels — HyperPod Inference exposes Kubernetes’ native nodeAffinity directly in the InferenceEndpointConfig spec. This gives you the full expressiveness of Kubernetes scheduling primitives.
To clean up your environment after completing this walkthrough, follow these steps to remove the deployed models and uninstall the Inference Operator add-on from your HyperPod cluster.
Removing Inference Operator add-on
Through the SageMaker console:
Navigate to SageMaker Console → HyperPod Clusters → Cluster Management
Select your cluster and go to the Inference tab
Choose Remove to uninstall the Inference Operator add-on and associated resources
Automated migration script is hosted in public GitHub that transitions the HyperPod Inference Operator from Helm to EKS add-on with built-in rollback capabilities if add-on installation fails. Backup files are stored in /tmp/hyperpod-migration-backup-<timestamp>/ for manual rollback if needed.
Key features
Auto-Discovery: Derives configuration from existing Helm deployment (roles, buckets, dependencies)
Safe Migration: Scales down Helm deployments before add-on installation, validates prerequisites
Dependency Handling: Migrates S3/FSx CSI drivers, cert-manager, and metrics-server to add-ons
Rollback Support: Preserves original resources and restores on failure
IAM Roles created
Execution Role (Inference Operator + S3 TLS access)
JumpStart Gated Model Role
ALB Controller Role
KEDA Operator Role
Examples for running the script
# to follow step by step guide
./helm_to_addon.sh --cluster-name <my-cluster> --region us-east-1
# no prompts needed except for initiating rollback in case of failure
./helm_to_addon.sh --cluster-name <my-cluster> --region us-east-1 --auto-approve
# To skip the dependencies FSX, S3, Metricsserver, cert manager migration from Inference operator helm to respective add-ons
./helm_to_addon.sh --cluster-name my-cluster —region us-east-1 --skip-dependencies-migration
Migration flow
Validate existing Helm installation
Auto-derive configuration and create new IAM roles
Tag resources (ALBs, ACM certs, S3 objects) with CreatedBy: HyperPodInference
Scale down Helm deployments for ALB, KEDA and Inference operator
Install Inference Operator add-on with OVERWRITE flag
Clean up old Helm resources
Migrate Helm-installed dependencies that are installed through Inference operator main chart to add-ons. To skip this step provide --skip-dependencies flag.
Benefits
Simplified management through EKS console/APIs
Automated updates via EKS add-on mechanisms
Native EKS integration
Zero downtime migration with rollback safety
Conclusion
The streamlined Inference Operator installation experience for Amazon SageMaker HyperPod eliminates infrastructure complexity and accelerates time to value for machine learning teams. With one-click installation, automated resource management, and seamless upgrade capabilities, teams can focus on deploying and optimizing their inference workloads rather than managing underlying infrastructure.
The EKS Add-on integration provides enterprise-grade lifecycle management while maintaining the flexibility to customize configurations for specific organizational requirements. Combined with advanced features like managed tiered KV cache and intelligent routing, this simplified installation experience makes high-performance inference deployment accessible to teams of all sizes.
Get started today by creating a new HyperPod cluster with the Inference Operator pre-installed, or add it to your existing clusters with a single click through the SageMaker console. For detailed add-on installation instructions and configuration options see this guide and for troubleshooting see this guide.
Traditional business intelligence (BI) integration with enterprise data warehouses has been the established pattern for years. With generative AI, you can now modernize BI workloads with capabilities like interactive chat agents, automated business processes, and using natural language to generate dashboards.
In this post, we provide implementation guidance for building integrated analytics solutions that combine the generative BI features of Amazon Quick with Amazon Redshift and Amazon Athena SQL analytics capabilities. Use this post as a reference for proof-of-concept implementations, production deployment planning, or as a learning resource for understanding Quick integration patterns with Amazon Redshift and Athena.
Common use cases
You can use this integrated approach across several scenarios. The following are some of the most common use cases.
Traditional BI reporting benefits from bundled data warehouse and BI tool pricing, making generative BI the primary use case with significant cost advantages.
Insurance: Automates Solvency II and IFRS 17 regulatory reporting, replacing manual spreadsheet consolidation.
Banking: Accelerates FDIC call report generation and capital adequacy dashboards, cutting month-end close from days to hours.
Interactive dashboards with contextual chat agents give BI teams conversational interfaces alongside their visual metrics.
Gaming: Live ops teams query player retention and monetization KPIs in plain English—no SQL needed.
Financial Services: Trading analysts chat with real-time P&L dashboards to surface anomalies and drill into positions on demand.
Domain-specific analytics workspaces democratize enterprise data exploration through Quick Spaces and natural language queries.
Insurance: Actuarial and underwriting teams query claims and risk data without waiting on data engineering.
Banking: Risk and compliance teams explore credit, market, and operational data through a single natural language interface.
Workflow automation removes repetitive tasks and accelerates self-service analytics.
Financial Services: Automated AR reconciliation flows replace manual ledger matching, shrinking close cycle effort significantly.
Gaming: Telemetry ingestion pipelines trigger reporting refreshes automatically, freeing data engineers from routine work.
Let us examine an end-to-end solution combining these technologies.
Solution flow
AWS offers two native SQL analytics engines for building analytics workloads. Amazon Redshift provides a fully managed data warehouse with columnar storage and massively parallel processing. Amazon Athena delivers serverless interactive query capabilities directly against data in Amazon S3.
You can use either Amazon Redshift or Amazon Athena as a SQL engine while implementing the steps in this post. The following are the steps involved in building an end-to-end solution.
Figure1: Solution steps to integrate SQL Analytics engines with Amazon Quick
Set up your SQL analytics engines: Amazon Redshift or Amazon Athena.
Load data and create business views designed for analytics workloads.
Configure integration between SQL analytics engines and Amazon Quick.
Create data sources in Amazon Quick.
Create datasets and dashboards for visual analytics.
Use Topics and Spaces to provide natural language interfaces to your data.
Deploy chat agents to deliver conversational AI experiences for business users.
Implement business flows to automate repetitive workflows and processes.
Let’s start by walking through steps 1–4 for Amazon Redshift. We then describe the same four steps for Amazon Athena before explaining the Amazon Quick steps 5–8.
Configure and create datasets in Amazon Redshift
Amazon Redshift offers two deployment options to meet your data warehousing needs. Provisioned clusters provide traditional deployment where you manage compute resources by selecting node types and cluster size. Serverless automatically scales compute capacity based on workload demands with pay-per-use pricing. Both options are supported by Amazon Quick. For this walkthrough, we use Redshift Serverless.
Set up SQL analytics engine
To create a Redshift Serverless namespace and workgroup:
On the left navigation pane, select Redshift Serverless.
Follow the steps described in the Creating a workgroup with a namespace documentation page to create a workgroup and a namespace. Note the username and password provided. You will use these details for configuring connections in Amazon Redshift and Quick.
You should see the status as Available for both the workgroup and namespace in the Serverless dashboard.
Figure 2: Amazon Redshift Serverless Workgroup and NamespacesThe deployment will be completed in approximately 3–5 minutes.
Load data and create business views
Now you can load data using the industry-standard TPC-H benchmark dataset, which provides realistic customer, order, and product data for analytics workloads.To load data into Amazon Redshift:
Run the following COPY commands to load data from the public S3 bucket: s3://redshift-downloads/TPC-H/.
Ensure that the IAM role attached to the namespace is set as the default IAM role. If you didn’t set up the default IAM role at the time of namespace creation, you can refer to the Creating an IAM role as default for Amazon Redshift documentation page to set it now.
copy customer from 's3://redshift-downloads/TPC-H/2.18/100GB/customer/' iam_role default delimiter '|' region 'us-east-1';
copy orders from 's3://redshift-downloads/TPC-H/2.18/100GB/orders/' iam_role default delimiter '|' region 'us-east-1';
copy lineitem from 's3://redshift-downloads/TPC-H/2.18/100GB/lineitem/' iam_role default delimiter '|' region 'us-east-1';
Run the following query to validate load status. The status column should show as completed. You can also review the information in other columns to see details about the loads such as record counts, duration, and data source.
select * from SYS_LOAD_HISTORY Where table_name in ('customer','orders','lineitem');
Figure 3: Output of SYS_LOAD_HISTORY showing successful completion of COPY Jobs
Create a materialized view to improve query performance:
Run the following SQL to create a materialized view that pre-compute results set for customer revenues and order volumes by market segment.
CREATE MATERIALIZED VIEW mv_customer_revenue AS
SELECT
c.c_custkey,
c.c_name,
c.c_mktsegment,
SUM(l.l_extendedprice * (1 - l.l_discount)) as total_revenue,
COUNT(DISTINCT o.o_orderkey) as order_count
FROM customer c
JOIN orders o ON c.c_custkey = o.o_custkey
JOIN lineitem l ON o.o_orderkey = l.l_orderkey
GROUP BY c.c_custkey, c.c_name, c.c_mktsegment;
Run the following SQL to review the data in the materialized view.
select * from mv_customer_revenue limit 10;
Configure integration with Amazon Quick
Amazon Quick auto discovers the Amazon Redshift provisioned clusters that are associated with your AWS account. These resources must be in the same AWS Region as your Amazon Quick account. For Amazon Redshift clusters in other accounts or Amazon Redshift Serverless, we recommend that you add a VPC connection following the steps in Enabling access to an Amazon Redshift cluster in a VPC documentation. Usually, these steps are performed by your organization’s cloud security administration team.
For serverless, you will apply the same steps in the workgroup instead of the cluster. You can find the VPC and Security Group settings in the Data Access tab of a workgroup.
Figure 4: Amazon Redshift Serverless workgroup VPC and Security groups
To create a dataset connecting to Amazon Redshift, complete the following steps.
In the Quick left navigation pane, go to Datasets.
Choose the Data sources tab and select Create data source.
Select Amazon Redshift and enter the following:
Data Source Name: Provide customer-rev-datasource as data source name.
Connection type: Select the VPC connection created in the previous step.
Database server: Enter the Amazon Redshift workgroup endpoint (for example, quick-demo-wg.123456789.us-west-2.redshift-serverless.amazonaws.com).
Port: 5439 (default).
Database: dev.
Username/Password: Amazon Redshift credentials with access to the database.
Choose Validateconnection. The validation should be successful.
Figure 5: Amazon Redshift data source configuration
Choose Create Data Source to create a data source.
Now let’s explore how to perform all these four steps to configure Athena in Amazon Quick.
Configure and create datasets in Amazon Athena
Amazon Athena provides immediate query capabilities against petabytes of data with automatic scaling to handle concurrent users. Let’s go through the steps to configure connections between Amazon Quick and Amazon Athena.
For Query result configuration, select Athena managed.
Choose Create workgroup.
Your workgroup is ready immediately for querying data.
Load data and create business views
For Athena, you create tables using the TPC-H benchmark dataset that AWS provides in a public S3 bucket. This approach gives you 1.5 million customer records already optimized in Parquet format without requiring data loading.
To create tables and views in Athena:
Open the Athena Query Editor from the console.
Create a database for your analytics (create S3 bucket if it exists already):
CREATE DATABASE IF NOT EXISTS athena_demo_db
COMMENT 'Analytics database for customer insights'
LOCATION 's3://my-analytics-data-lake-[account-id]/';
Create an external table pointing to the TPC-H public dataset:
CREATE EXTERNAL TABLE IF NOT EXISTS athena_demo_db.customer_csv (
C_CUSTKEY INT,
C_NAME STRING,
C_ADDRESS STRING,
C_NATIONKEY INT,
C_PHONE STRING,
C_ACCTBAL DOUBLE,
C_MKTSEGMENT STRING,
C_COMMENT STRING
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
LOCATION 's3://redshift-downloads/TPC-H/2.18/100GB/customer/'
Create a business-friendly view for analytics:
Run the following SQL to create a view that aggregates customer account balances grouped by market segments.
CREATE VIEW athena_demo_db.customer_deep_analysis AS
SELECT
c_custkey AS customer_id,
c_name AS customer_name,
c_mktsegment AS market_segment,
c_nationkey,
ROUND(c_acctbal, 2) AS account_balance,
CASE
WHEN c_acctbal < 0 THEN 'At-Risk'
WHEN c_acctbal < 2500 THEN 'Low'
WHEN c_acctbal < 5000 THEN 'Mid'
WHEN c_acctbal < 8000 THEN 'High'
ELSE 'Premium'
END
AS balance_tier,
ROUND(AVG(c_acctbal) OVER (PARTITION BY c_mktsegment), 2) AS segment_avg,
ROUND(c_acctbal - AVG(c_acctbal) OVER (PARTITION BY c_mktsegment), 2) AS vs_segment_avg,
ROUND((c_acctbal - AVG(c_acctbal) OVER (PARTITION BY c_mktsegment))
/ NULLIF(STDDEV(c_acctbal) OVER (PARTITION BY c_mktsegment), 0), 2) AS segment_z_score,
RANK() OVER (PARTITION BY c_mktsegment ORDER BY c_acctbal DESC) AS rank_in_segment,
NTILE(5) OVER (ORDER BY c_acctbal DESC) AS global_quintile
FROM athena_demo_db.customer_csv
ORDER BY c_acctbal DESC;
Verify your view from Athena with:
SELECT * FROM athena_demo_db.customer_deep_analysis limit 5;
In the top-right corner, choose your profile icon, then select Manage account.
Under Permissions, choose AWS resources. Figure 7: AWS resource permissions
Enable Athena Access
Under Quick access to AWS services, choose Manage.
Locate Amazon Athena in the list of AWS services.
If Athena is already selected but access issues persist, clear the checkbox and re-select it to re-enable Athena.
Under Amazon S3, select S3 buckets.
Check the boxes next to each S3 bucket that Amazon Quick needs to access—including buckets used for Athena query results and any Redshift COPY source buckets.
Enable Write permission for Athena Workgroup to allow Amazon Quick to write Athena query results to S3 and choose Finish.
Choose Save to update the configuration.
The final step is to grant your Amazon Quick author permissions to query your database, Athena tables, and views. Configuration depends on whether AWS Lake Formation is enabled.
If AWS Lake Formation is not enabled
Permissions are managed at the Quick service role level through standard IAM-based S3 access control. Ensure that the Quick service role (for example, aws-quick-service-role-v0) has the appropriate IAM permissions for the relevant S3 buckets and Athena resources. No additional Lake Formation configuration is required.
If AWS Lake Formation is enabled
Lake Formation acts as the central authorization layer, overriding standard IAM-based S3 permissions. Grant permissions directly to the Amazon Quick author or IAM role.
Choose Permissions, then Data permissions, then Grant.
Select the IAM user or role.
Choose the required databases, tables, and columns.
Grant SELECT at minimum; add DESCRIBE for dataset creation.
Repeat for each user or role that requires access.
Create data source
Follow these steps to create an Athena data source on Amazon Quick.
In the Amazon Quick console, navigate to Datasets and choose Data sources tab.
Choose Create data source, then select the Amazon Athena card.
Enter a Data source name (you can give any name of your choice), select your Athena workgroup (like quick-demo), and choose Validate connection.
Figure 8: Athena data source creation
Choose Create data source.
Your Athena data source is now available for building datasets, dashboards, and Topics.
Use Amazon Quick generative AI features
The next steps, from 5–8, demonstrate Amazon Quick generative AI capabilities using Amazon Redshift as a data source. While we use Amazon Redshift in this example, you can substitute with Amazon Athena based on your specific requirements.
Create dashboards
Let’s start by creating datasets from the Amazon Redshift data source.
In the left navigation pane, choose Datasets.
On the Datasets page, choose Create Dataset.
For the data source, select Amazon Redshift data source customer-rev-datasource.
From the menu, choose mv_customer_revenue.
Figure 9: Select table to visualize
You can choose one of the following query modes. For this post, select Directly query your data option and choose Visualize.
Import to SPICE for quicker analytics – Quick loads a snapshot into its in-memory engine for faster dashboard performance.
Directly queryyour data– Quick runs queries on demand against your query engine.
Select Build icon to open a chat window. Enter “Show me orders by market segments” as the prompt. Note that you need Author Pro access to use this feature.
Figure 10: Build visualization using generative BI feature
You can change the visual type to a pie chart and add it to the analysis.
Choose Publish dashboard. Your dashboard is now available for viewing and sharing.
Create topics and spaces
To fully maximize enterprise data with AI, we must provide the right structure and context. That’s where Topics and Spaces come in. Topics act as natural language interfaces to your structured datasets, automatically analyzing your data, mapping fields, and adding synonyms. Business users can ask “What are total revenues by market segment?” and receive instant, visualized answers without writing a single line of SQL. Spaces bring together all of your related assets into a single collaborative workspace that democratizes data access, reduces context-switching, accelerates team onboarding, so everyone is working from the same trusted, AI-ready data sources.
To create a Quick topic
From the Amazon Quick homepage, choose Topics, then choose Create topic.
Enter a name for your topic. For this post, use Customer Revenue Analytics.
Enter a description. For example:
The Customer Revenue Analytics topic is designed for business users (including analysts, sales operations teams, finance, and market segment owners who need to explore customer and revenue data without SQL expertise. It serves as a natural language interface over the mv_customer_revenue Amazon Redshift dataset, allowing users to ask plain-English questions like “What are total revenues by market segment?” and receive instant, visualized answers. By automatically mapping business language to the underlying schema, it democratizes access to revenue insights across the organization.
Under Dataset, select mv_customer_revenue.
Choose Create. The topic can take 15–30 minutes to enable depending on the data. During this time, Amazon Quick automatically analyzes your data, selects relevant fields, and adds synonyms.
After the topic is enabled, take a few minutes to review and enrich it. The following are some example enrichments.
Add column descriptions to clarify field meaning for business users.
Define preferred aggregations (for example, sum compared to average for revenue fields).
Confirm which fields are Dimensions and which are Measures.
(Optional) To further refine how your topic interprets and responds to queries, add multiple datasets (for example, a customer CSV combined with a database view), custom instructions, filters, and calculated fields.
After your topic is created, its columns are available to add to a Space or to an Agent by selecting it as a data source.
Figure 11: Create a Quick Topic
Create a Space for your team
Spaces bring together dashboards, topics, datasets, documents, and other resources into organized, collaborative workspaces. By centralizing related assets in a single workspace, Spaces reduce context-switching, accelerate onboarding, so everyone is working from the same trusted data sources.
What to include in your Quick Space
Dashboard – Add the dashboard Market Segment Dashboard published from your mv_customer_revenue analysis. This gives team members instant access to visualizations such as revenue by market segment, top customers by order volume, and revenue distribution.
Topic – Connect the Customer Revenue Analytics (built on the mv_customer_revenue materialized view) to enable natural language queries directly against your Amazon Redshift data.
Optionally, you can upload supporting context to ground your team’s analysis:
Data dictionary or field definitions for mv_customer_revenue
Business rules for revenue calculation (for example, how discounts are applied in the TPC-H model)
This implementation guide, so new team members can onboard quickly
To create the Quick Space
From the left navigation menu, choose Spaces, then choose Create space.
Enter a name, for example, Customer Revenue & Segmentation.
Enter a description. For example:
Centralized workspace for customer revenue analysis powered by Amazon Redshift includes interactive dashboards, natural language query access to customer and segment data, and supports documentation for the TPC-H revenue model.
Add knowledge by connecting the Market Segment Dashboard and topic Customer Revenue Analytics.
You can invite team members, such as finance, sales operations, and segment owners, and set appropriate permissions.
Your Space is now ready for collaborative data exploration.
Figure 12: Create a Quick Space
Build chat agents
A custom chat agent delivers conversational AI experiences that understand business context and provide intelligent, grounded responses to user queries. These agents go beyond question-and-answer interactions. They synthesize knowledge from your dashboards, topics, datasets, and documents to explain trends, surface anomalies, guide users through complex analytics workflows, and recommend next steps.
Rather than requiring users to navigate multiple tools or write SQL queries, agents serve as a single conversational interface to your entire analytics environment. Agents can also connect to Actions, pre-built integrations with enterprise tools such as Slack, Microsoft Teams, Outlook, and SharePoint, enabling them to answer questions and trigger real-world workflows, send notifications, create tasks, and interact with external systems directly from the conversation. Custom agents can be tailored to specific business domains, teams, or use cases so that responses align with organizational terminology, data definitions, and business processes. After created, agents can be shared across teams, enabling consistent, actionable, AI-powered data access at scale. For teams working with the mv_customer_revenue dataset, we recommend creating a dedicated Customer Revenue Analysis Agent. This is a purpose-built conversational assistant grounded in your Amazon Redshift data, dashboards, and the Customer Revenue & Segmentation Space.
Create a Quick chat agent
There are two ways that you can use Amazon Quick to create a Quick agent. You can use the navigation menu or directly from Space. The following steps walk you through creating one from the navigation menu.
To create a Quick chat agent
From the left navigation menu, choose Agents, then choose Create agent.
Enter a name for your agent, for example, Customer Revenue Analyst.
Enter a description. For example:
An AI assistant for analyzing customer revenue, market segment performance, and order trends using our Amazon Redshift or data warehouse.
Under Knowledge Sources, add the Customer Revenue & Segmentation Space as a data source. This gives your agent access to the dashboards, topics, and reference documents you’ve already built.
(Optional) Define custom persona instructions to align the agent’s responses with your business context. For example, specifying preferred terminology, response style, or the types of questions it should prioritize.
Choose Launch chat agent.
Start having a conversation with your data. You are welcome to ask any questions. The following are some examples.
Which market segment generated most revenue?
Show me order trends
Figure 13: Create a Quick Chat agent
To share your Quick chat agent
After your agent is published, choose Share and invite team members or share it across your organization. Custom agents can be tailored to specific business contexts so that different teams can get AI assistance that speaks their language, without needing to configure anything themselves.
Create Quick Flows
Quick Flows automate repetitive tasks and orchestrate multi-step workflows across your entire analytics environment. This removes manual effort, reducing human error, and ensuring consistent execution of critical business processes. Flows can be triggered on a schedule or launched on demand, giving you flexible control over when and how automation runs.
You can build flows that span the full analytics lifecycle: monitoring data quality and flagging anomalies, generating and distributing scheduled reports to stakeholders, and triggering downstream actions in integrated systems such as Slack, Outlook. Amazon Quick gives you three ways to create a flow, so whether you prefer a no-code conversation or a visual step-by-step builder, there’s an option that fits how you work.
To create a flow from chat
While conversing with My Assistant or a custom agent, describe the workflow that you want to automate in plain English.
Amazon Quick generates the flow and offers to create it directly from your conversation — no configuration screens required.
To create a flow from a natural language description
From the left navigation menu, choose Flows, then choose Create flow.
Enter a plain-English description of your workflow. For example:
” Query revenue data by market segments. Filter by order count and all dates. Search web for comparable relevant market trends. Generate formatted summary reports providing market summary and look ahead per segment. ”
Amazon Quick automatically generates the complete workflow with all the necessary steps.
Optionally, you can add additional steps.
Choose Run Mode to test the Flow.
After your flow is created, share it with team members or publish it to your organization’s flow library, so everyone benefits from the same automation without having to rebuild it independently.
Figure 14: Create a Quick Flow to generate summaries and publish dashboards
For more complex flow, review weekly customer revenue summary flow as an example.
Queries the mv_customer_revenue materialized view in Amazon Redshift for the latest weekly revenue figures by market segment.
Compares results against the prior week to calculate segment-level variance.
Generates a formatted summary report and publishes it to the Customer Revenue & Segmentation Space.
Sends a notification through email or Slack to finance, sales operations, and segment owners with a direct link to the updated dashboard.
Flags any segment where revenue has declined more than a defined threshold, routing an alert to the appropriate owner for follow-up.
This flow transforms what might otherwise be a manual, multi-step reporting process into a fully automated pipeline, so stakeholders receive consistent, timely revenue insights without analyst intervention and saving analysts an estimated 3–5 hours per week. For detailed guidance on creating and managing flows, see Using Amazon Quick Flows. Also review Create workflows for routine tasks demo.
Cleanup
Consider deleting the following resources created while following this post to avoid incurring costs. We encourage you to use the trials at no cost as much as possible to familiarize yourself with the features described.
Delete the Amazon Quick account used while following this post. If you used an existing account, delete the data sets, dashboards, topics, spaces, agents and flows created.
Conclusion
This integrated approach to business intelligence combines the power of AWS SQL analytics engines with Amazon Quick generative AI capabilities to deliver comprehensive analytics solutions. By following these implementation steps, you establish a foundation for traditional BI reporting, interactive dashboards, natural language data exploration, and intelligent workflow automation. The architecture scales from proof-of-concept implementations to production deployments, transforming how organizations access and act on data insights. For more information about Amazon Quick features and capabilities, see the Amazon Quick documentation. To learn more about Amazon Redshift, visit the Amazon Redshift product page. For Amazon Athena details, see the Amazon Athena product page.
Learn how you can use AWS services like AWS Backup and AWS Elastic Disaster Recovery (AWS DRS), along with AWS Resilience Competency Partner solutions like Arpio to implement powerful and comprehensive Disaster Recovery solutions.
Resilience is the ability of your application to keep running even when “bad stuff” happens. A critical part of your resilience strategy is Disaster Recovery (DR). DR is what protects you against less frequent, but bigger faults like natural disasters, technical faults, and bad actors. To maintain critical business continuity, disaster recovery requires recovering your workload to a new site, such as a different AWS Region or AWS account.
AWS provides powerful tools for all aspects of resilience. However, achieving a comprehensive Disaster Recovery solution for your cloud workloads using native AWS services requires planning and engineering effort. This is because, as the Shared Responsibility Model for Resiliency states, resilience is a shared responsibility between AWS and the customer. This blog post will help you understand your responsibilities and show you how to reduce the work required to access the powerful DR capabilities of AWS.
In this blog post, we take a building blocks approach. Starting with the tools like AWS Backup to protect your data, we then add protection for Amazon Elastic Compute Cloud (Amazon EC2) compute using AWS Elastic Disaster Recovery (AWS DRS). Finally, we show how to use the full capabilities of AWS to restore your entire workload—data, infrastructure, networking, and configuration, using Arpio disaster recovery automation.
Your recovery site
For DR, your recovery site is usually going to be a different AWS Region (cross-Region) or a different AWS account (cross-account) than where your workload runs.
Cross-Region backup and recovery are essential for disaster recovery. This helps to keep your workloads protected if an event causes your source Region to be unable to run your workload. AWS Regions are strong fault isolation boundaries, so the event in your source is highly unlikely to affect your recovery Region.
Cross-account backup is a critical security measure to enable recovery from malware and ransomware. By storing copies of your data in a separate clean room recovery account with distinct credentials, you create an isolated environment that can’t be accessed, even if the source account is compromised.
Protecting your data
We start with your data—your data is the foundation of your workload.
Figure 1. AWS Backup and replication for Amazon RDS
AWS Backup takes this further, tying together many of these disparate backup technologies, giving a single plane of glass to configure data backup plans across resources. AWS Backup also added backup capabilities for AWS resources that previously didn’t have them such as Amazon Elastic File System (Amazon EFS) and Amazon FSx. It also provides the ability to back up your data to a different AWS Region or AWS account. It even enabled cross-Region backup for services like Amazon DynamoDB, which previously didn’t have that capability.
AWS Backup is a powerful tool for protecting your data. With your data protected, you will then need additional automation to get to a fully recovered workload. If you want to build this yourself, AWS offers the tools to do this. In this prior blog post on Backup and Restore, we go more into detail about adding automation using Amazon EventBridge and AWS Lambda functions for automated recovery. For more information, see figures 6 and 7.
With its ability to create vaults for secure storage, define policies for governance, and set schedules for automation, AWS Backup centralizes and streamlines the backup process. Instead of managing backups service by service, you can enforce consistent protection across resources, reduce manual effort, and streamline recovery when it matters most.
Protecting your Amazon EC2 compute
As important as data is, only restoring your data isn’t a complete solution to recovering from disaster. You must also restore your compute resources.
For static Amazon EC2 instances, you can create snapshots of your instances as Amazon Machine Images (AMIs), or use AWS Backup to manage this for you. By static instances, we mean those you create directly and maintain, as opposed to those created by Amazon EC2 Auto Scaling. Such a strategy can deliver a Recovery Points Objective (RPO) and Recovery Time Objective (RTO) of minutes to hours. The size (and growth in size) of your EC2 instances determines the time to back them up. Their size and launch time determines the time to restore them.
If you need real-time RPO (near-zero data loss) and RTO (recovery) in minutes or less, then AWS DRS is the solution here. AWS DRS provides a nearly continuous block-level replication, recovery orchestration, and automated server conversion capabilities. With these, you to achieve a crash-consistent recovery point objective of seconds, and a recovery time objective typically ranging between 5–20 minutes. You can also use AWS DRS to configure your recovery Amazon Virtual Private Cloud (Amazon VPC). So, with the right settings, you can get your EC2 networking to look like your primary environment.
Protecting everything in your entire workload
Restoring data and static EC2 instances is only part of the disaster recovery solution that you need. Modern workloads often rely on a broader range of compute services, including EC2 Auto Scaling, AWS Lambda, Amazon ECS, and Amazon Elastic Kubernetes Service (Amazon EKS). For ECS and EKS, you can run on EC2 instances or go serverless with AWS Fargate. You will need a solution that can restore either of these.
The challenge with these services is making sure that they are recreated with the right configuration and metadata. For example, EC2 instance types and volume sizes, EC2 user data, or AWS Lambda function code, and not everything here is stateless. Both ECS and EKS can rely on persistent Amazon EBS volumes or Amazon Elastic File System (Amazon EFS). In those cases, recovery requires restoring the data and reattaching volumes restored from backup to the correct ECS tasks or EKS pods.
You can build automation to do all of this, or you can rely on an AWS Resilience Software Competency Partner solution to take care of this for you. Arpio is a software as a service (SaaS) product focused on discovering and backing up everything it takes to run your workload on AWS, and recovering it cross-Region and cross-account as a fully functional workload.
Figure 2 illustrates how AWS tools (left) establish a powerful foundation for robust workload recovery. Beyond these foundational building blocks, full recovery requires additional resources (right), including AWS compute options as discussed. Furthermore, complex networking (potentially spanning VPCs and accounts), infrastructure, and IAM principals are critical. Arpio uses and extends AWS Backup, AWS DRS, and other AWS service capabilities to back up and restore a functional AWS workload, including all its necessary components. This unburdens you from the undifferentiated heavy lifting of building your own automation. You still have responsibilities in the shared responsibility model, but Arpio takes on most of the work of getting you backed up and recovered.
Figure 2. AWS tools on the left provide a powerful foundation for full recovery on the right
Even with data, compute, networking, infrastructure, and IAM principals restored, there is another requirement to achieve full recovery: translation of your configuration. For example, an application that accesses the Amazon RDS database requires configuration information about the DB endpoint and credentials. When restoring your RDS instance into your recovery environment, it will have a new endpoint. Arpio addresses this using a two-fold strategy. First Arpio will find all references to the early database endpoint name and translate them to the new database endpoint name. Next, Arpio will also create an Amazon Route 53 private hosted zone in the recovered VPC, mapping the early endpoint to the new one using a CNAME record. This way, applications still using the early name still connect to the newly recovered database. Arpio also securely backs up the credentials in your recovery account, for every database backup taken, ready to be recovered for the point in time that you recover your database from. Figure 3 shows how your recovered application can seamlessly access your restored database.
Figure 3. Arpio automation ensures your applications can access your restored database in the recovery environment
Figure 4 shows a sample AWS workload protected by Arpio. In the standby state, you can see how Arpio is coordinating multiple AWS services. When a disaster or ransomware event occurs, you can launch a recovery. This will create a fully recovered workload as seen in the recovery stage on the right.
Figure 4. Sample AWS workload protected by Arpio
Arpio does all of this in your accounts, on your behalf. To enable this, Arpio applies AWS Well-Architected Tool (AWS WA Tool) best practices for security, using only IAM roles with least-privilege permissions. For example, the IAM role used to access your source AWS account is incapable of changing or mutating your source workload and is explicitly denied from reading or exfiltrating any data.
With its ability to back up over 140 AWS resources and restore them as fully functioning AWS workloads in a cross-Region cross-account recovery environment, Arpio builds on top of the powerful AWS tooling to streamline your complete workload recovery.
Conclusion
Disaster recovery (DR) is essential for a robust resilience strategy. By using the powerful tools offered by AWS and complementing them with AWS Resilience Competency Partner solutions like Arpio, organizations can significantly streamline access to comprehensive and powerful disaster recovery capabilities for their AWS workloads.
Amazon Managed Workflows for Apache Airflow (Amazon MWAA) provides robust orchestration capabilities for data workflows, but managing DAG permissions at scale presents significant operational challenges. As organizations grow their workflow environments and teams, manually assigning and maintaining user permissions becomes a bottleneck that can impact both security and productivity.
Traditional approaches require administrators to manually configure role-based access control (RBAC) for each DAG, leading to:
Inconsistent permission assignments across teams
Delayed access provisioning for new team members
Increased risk of human error in permission management
Significant operational overhead that doesn’t scale
There is another way of doing it by defining custom RBAC roles as mentioned in this Amazon MWAA User Guide. However, it doesn’t use Airflow tags to do so.
In this post, we show you how to use Apache Airflow tags to systematically manage DAG permissions, reducing operational burden while maintaining robust security controls that complement infrastructure-level security measures.
Prerequisites
To implement this solution, you need:
AWS resources:
An Amazon MWAA environment (version 2.7.2 or later, not supported in Airflow 3.0)
IAM roles configured for Amazon MWAA access with appropriate trust relationships
Amazon Simple Storage Service (Amazon S3) bucket for Amazon MWAA DAG storage with proper permissions
Permissions:
IAM permissions to create and modify Amazon MWAA web login tokens
Amazon MWAA execution role with permissions to access the Apache Airflow metadata database
Administrative access to configure Apache Airflow roles and permissions
Solution overview
The automated permission management system consists of four key components that work together to provide scalable, secure access control.The following diagram shows the workflow of how the solution works.
IAM integration layer – AWS IAM roles map directly to Apache Airflow roles. Then, users authenticate through AWS IAM and are automatically assigned corresponding Airflow roles. This supports both individual user roles and group-based access patterns. Note:
IAM Based access control to Amazon MWAA works for Apache Airflow default roles. For custom roles, the Admin user can assign the custom role using the Apache Airflow UI as mentioned in the Knowledge Center post and in the Amazon MWAA User Guide.
If using other authenticators, the tag-based DAG permissions continue to work as stated in the AWS Big Data Blog post.
Tag-based configuration – Apache Airflow tags defined in DAGs are used to declare access requirements. It supports read-only, edit, and delete permissions.
Automated synchronization engine – Scheduled DAG scans all active DAGs for permission tags based on CRON schedule. It then processes tags and updates Apache Airflow RBAC permissions accordingly. Then, it provides a configuration based to control the clean-up of existing permissions.
Role-based access control enforcement – Apache Airflow RBAC enforces the configured permissions by storing on Apache Airflow role and permissions metadata tables. Users see only the DAGs that they have access to. They have granular control over read compared to edit permissions.
Data flow
Amazon MWAA User assumes an IAM role to access the Amazon MWAA UI.
DAG developer adds relevant tags to the DAG definition.
manage_dag_permissions DAG deployed to the Amazon MWAA environment runs on a CRON schedule, for example, daily.
The DAG updates the respective role permissions to the DAG by updating the Apache Airflow metadata on the Apache Airflow DB.
Users gain or lose access based on their assigned roles.
Our solution builds upon the existing IAM integration of Amazon MWAA, while extending functionality through custom automation:
Authentication and role mapping – Users authenticate through AWS IAM roles that map directly to corresponding Airflow roles.
Automated user creation – Upon first login, users are automatically created in the Apache Airflow metadata database with appropriate role assignments.
Tag-based permission control – Each Apache Airflow role contains specific DAG permissions based on tags defined in the DAGs.
Automated synchronization – A scheduled script maintains permissions as DAGs are added or modified.
Step 1: Configure IAM to Airflow role mapping
First, establish the mapping between your IAM principals and Apache Airflow roles. To grant permission using the AWS Management Console, complete the following steps:
Sign in to your AWS account and open the IAM console.
In the left navigation pane, choose Users, then choose your Amazon MWAA IAM user from the users table.
On the user details page, under Summary, choose the Permissions tab, then choose Permissions policies to expand the card and choose Add permissions.
In the Grant permissions section, choose Attach existing policies directly, then choose Create policy to create and attach your own custom permissions policy.
On the Create policy page, choose JSON, then copy and paste the following JSON permissions policy in the policy editor. This policy grants web server access to the user with the default Public Apache Airflow role.
Step 2: Create the automated permission management DAG
Now, create a DAG that will automatically manage permissions based on tags.
from airflow import DAG, settings
from airflow.operators.python import PythonOperator
from sqlalchemy import text
import pendulum
import logging
dag_id = "manage_dag_permissions"
class Constants:
"""
Constants class to hold constant values used throughout the code.
"""
AB_VIEW_MENU = "ab_view_menu"
AB_PERMISSION = "ab_permission"
AB_ROLE = "ab_role"
AB_PERMISSION_VIEW = "ab_permission_view"
AB_PERMISSION_VIEW_ROLE = "ab_permission_view_role"
DAG_TAG = "dag_tag"
CAN_READ = "can_read"
CAN_EDIT = "can_edit"
CAN_DELETE = "can_delete"
def _execute_query(sql_text, params=None, fetch=True):
"""
Execute a parameterized SQL query against the Airflow metadata DB.
All queries use SQLAlchemy text() with bind parameters to prevent SQL injection.
Parameters:
sql_text: SQL string with :named bind parameters
params: dict of parameter values
fetch: If True, return list of first-column values; if False, commit and return None
Returns:
List of values (first column) if fetch=True, else None
Raises:
Re-raises any exception after rollback and logging
"""
session = settings.Session()
try:
stmt = text(sql_text)
if fetch:
result = session.execute(stmt, params or {}).fetchall()
return [row[0] for row in result]
else:
session.execute(stmt, params or {})
session.commit()
return None
except Exception as e:
session.rollback()
logging.error(f"DB query error (fetch={fetch}): {type(e).__name__}: {e}")
raise
finally:
session.close()
def fetch_airflow_role_id(role_name):
"""
Fetch role id of a given role name using parameterized query.
"""
result = _execute_query(
"SELECT id FROM ab_role WHERE name = :role_name",
{"role_name": role_name},
)
if not result:
raise ValueError(f"Airflow role not found: {role_name}")
logging.info("Fetched role ID successfully")
return result[0]
def fetch_airflow_permission_id(permission_name):
"""
Fetch permission id of a given permission using parameterized query.
"""
result = _execute_query(
"SELECT id FROM ab_permission WHERE name = :perm_name",
{"perm_name": permission_name},
)
if not result:
raise ValueError(f"Airflow permission not found: {permission_name}")
logging.info("Fetched permission ID successfully")
return result[0]
def fetch_airflow_menu_object_ids(dag_names):
"""
Fetch view_menu IDs for a list of DAG resource names.
Uses parameterized IN-clause via individual bind params.
Parameters:
dag_names: list of DAG resource names (e.g. ['DAG:my_dag1', 'DAG:my_dag2'])
Returns:
list of view_menu IDs
"""
if not dag_names:
return []
# Build parameterized IN clause: :p0, :p1, :p2, ...
param_names = [f":p{i}" for i in range(len(dag_names))]
params = {f"p{i}": name for i, name in enumerate(dag_names)}
in_clause = ", ".join(param_names)
result = _execute_query(
f"SELECT id FROM ab_view_menu WHERE name IN ({in_clause})",
params,
)
logging.info(f"Fetched {len(result)} view menu IDs")
return result
def fetch_perms_obj_association_ids(perm_id, view_menu_ids):
"""
Fetch permission_view IDs for a permission and list of view_menu IDs.
Uses parameterized query.
"""
if not view_menu_ids:
return []
param_names = [f":vm{i}" for i in range(len(view_menu_ids))]
params = {f"vm{i}": vm_id for i, vm_id in enumerate(view_menu_ids)}
params["perm_id"] = perm_id
in_clause = ", ".join(param_names)
result = _execute_query(
f"SELECT id FROM ab_permission_view WHERE permission_id = :perm_id AND view_menu_id IN ({in_clause})",
params,
)
logging.info(f"Fetched {len(result)} permission-view association IDs")
return result
def fetch_dag_ids_by_tag(tag_name):
"""
Fetch DAG IDs with a given tag name using parameterized query.
"""
result = _execute_query(
"SELECT DISTINCT dag_id FROM dag_tag WHERE name = :tag_name",
{"tag_name": tag_name},
)
logging.info(f"Fetched {len(result)} DAG IDs for tag")
return result
def associate_permission_to_object(perm_id, view_menu_ids):
"""
Associate permission to view_menu objects (DAGs) using parameterized INSERT.
"""
session = settings.Session()
try:
for vm_id in view_menu_ids:
session.execute(
text(
"INSERT INTO ab_permission_view (permission_id, view_menu_id) "
"VALUES (:perm_id, :vm_id) "
"ON CONFLICT (permission_id, view_menu_id) DO NOTHING"
),
{"perm_id": perm_id, "vm_id": vm_id},
)
session.commit()
logging.info(f"Associated permission to {len(view_menu_ids)} view menus")
except Exception as e:
session.rollback()
logging.error(f"Error associating permission to objects: {type(e).__name__}: {e}")
raise
finally:
session.close()
def associate_permission_to_role(permission_view_ids, role_id):
"""
Associate permission_view entries to a role using parameterized INSERT.
"""
session = settings.Session()
try:
for pv_id in permission_view_ids:
session.execute(
text(
"INSERT INTO ab_permission_view_role (permission_view_id, role_id) "
"VALUES (:pv_id, :role_id) "
"ON CONFLICT (permission_view_id, role_id) DO NOTHING"
),
{"pv_id": pv_id, "role_id": role_id},
)
session.commit()
logging.info(f"Associated {len(permission_view_ids)} permissions to role")
except Exception as e:
session.rollback()
logging.error(f"Error associating permissions to role: {type(e).__name__}: {e}")
raise
finally:
session.close()
def validate_if_permission_granted(permission_view_ids, role_id):
"""
Validate if given permissions are associated to given role using parameterized query.
"""
if not permission_view_ids:
return []
param_names = [f":pv{i}" for i in range(len(permission_view_ids))]
params = {f"pv{i}": pv_id for i, pv_id in enumerate(permission_view_ids)}
params["role_id"] = role_id
in_clause = ", ".join(param_names)
result = _execute_query(
f"SELECT id FROM ab_permission_view_role "
f"WHERE permission_view_id IN ({in_clause}) AND role_id = :role_id",
params,
)
logging.info(f"Validated {len(result)} permission grants")
return result
def clean_up_existing_dag_permissions_for_role(role_id):
"""
Clean up existing DAG permissions for a given role using parameterized query.
Note: this creates a brief window where the role has no DAG permissions.
"""
_execute_query(
"DELETE FROM ab_permission_view_role WHERE id IN ("
" SELECT pvr.id"
" FROM ab_permission_view_role pvr"
" INNER JOIN ab_permission_view pv ON pvr.permission_view_id = pv.id"
" INNER JOIN ab_view_menu vm ON pv.view_menu_id = vm.id"
" WHERE pvr.role_id = :role_id AND vm.name LIKE :dag_prefix"
")",
{"role_id": role_id, "dag_prefix": "DAG:%"},
fetch=False,
)
logging.info("Cleaned up existing DAG permissions for role")
def sync_permission(config_data):
"""
Sync permissions based on the config.
Parameters:
config_data: dict with keys:
- airflow_role_name: name of the custom Airflow role
- managed_dags: list of DAG IDs to grant full management permissions on
(can_read, can_edit, can_delete)
- do_cleanup: if True, remove all existing DAG:* permissions first
"""
# Get the role ID for role name
role_id = fetch_airflow_role_id(config_data["airflow_role_name"])
# Clean up existing DAG level permissions if requested
if config_data.get("do_cleanup", False):
clean_up_existing_dag_permissions_for_role(role_id)
managed_dags = config_data.get("managed_dags", [])
if not managed_dags:
logging.info("No managed DAGs found, skipping permission sync")
return
# Determine which permissions to grant (default: can_read only)
permissions = config_data.get("permissions", [Constants.CAN_READ])
# Build DAG resource names (e.g. ["DAG:my_dag1", "DAG:my_dag2"])
dag_resource_names = [f"DAG:{dag.strip()}" for dag in managed_dags]
# Get IDs for DAG view_menu objects
vm_ids = fetch_airflow_menu_object_ids(dag_resource_names)
if not vm_ids:
logging.info("No view_menu entries found for managed DAGs")
return
# Grant the configured permissions on each managed DAG
all_perm_view_ids = []
for perm_name in permissions:
perm_id = fetch_airflow_permission_id(perm_name)
associate_permission_to_object(perm_id, vm_ids)
all_perm_view_ids += fetch_perms_obj_association_ids(perm_id, vm_ids)
# Associate permission_view entries with the role and validate
if all_perm_view_ids and role_id:
associate_permission_to_role(all_perm_view_ids, role_id)
validate_if_permission_granted(all_perm_view_ids, role_id)
def sync_permissions_with_tags(role_mappings):
"""
For each role mapping, fetch DAG IDs by tag and sync permissions.
"""
for role_map in role_mappings:
username = list(role_map.keys())[0]
airflow_role = role_map[username]["airflow_role"]
edit_tag_name = role_map[username]["airflow_edit_tag"]
config_data = {
"airflow_role_name": airflow_role,
"managed_dags": fetch_dag_ids_by_tag(edit_tag_name),
"permissions": role_map[username].get("permissions", [Constants.CAN_READ]),
"do_cleanup": role_map[username].get("do_cleanup", True),
}
logging.info(f"Syncing permissions for airflow role")
sync_permission(config_data)
logging.info("Completed permission sync for role")
"""
Add new roles and permissions here.
Format:
{
"<role_key>": {
"airflow_role": <Custom Airflow role name to grant permissions to>,
"airflow_edit_tag": <Airflow Tag Name - DAGs with this tag will be managed>,
"permissions": <List of permissions to grant on each tagged DAG.
Options: "can_read", "can_edit", "can_delete"
Default: ["can_read"] if omitted>,
"do_cleanup": <Set to True (recommended) to clean up existing DAG permissions>
}
},
IMPORTANT - ROLE SETUP:
When creating a new custom role (e.g. "analytics_reporting", "marketing_analyst")
in the Airflow UI (Security > List Roles), you MUST copy the Viewer role's
permissions into the new role. The Viewer permissions provide base UI access
(browse DAGs, view logs, menu access, etc.). --or-- Assign the viewer role as well.
Without them, users assigned to
the custom role will not be able to log in to the Airflow UI.
This DAG manages DAG-level permissions on DAG:xxx resources.
Which permissions are granted is controlled by the "permissions" list
in each config entry (options: can_read, can_edit, can_delete).
It does NOT manage base UI permissions — those must be set up manually
when creating the role.
Steps to create a new custom role:
1. Go to Security > List Roles > + (Add)
2. Name it to match the "airflow_role" value in the config below
3. Copy all permissions from the "Viewer" role into the new role
4. Save — this DAG will then automatically add DAG-specific permissions
(as configured in the "permissions" list) for each tagged DAG
"""
role_mappings = [
{
"analytics_reporting": {
"airflow_role": "analytics_reporting",
"airflow_edit_tag": "analytics_reporting_edit",
"permissions": ["can_read", "can_edit", "can_delete"],
"do_cleanup": True,
}
},
{
"marketing_analyst": {
"airflow_role": "marketing_analyst",
"airflow_edit_tag": "marketing_analyst_edit",
"permissions": ["can_read", "can_edit", "can_delete"],
"do_cleanup": True,
},
},
]
with DAG(
dag_id=dag_id,
schedule="*/15 * * * *",
catchup=False,
start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
) as dag:
sync_dag_permissions_task = PythonOperator(
task_id="sync_dag_permissions",
python_callable=sync_permissions_with_tags,
op_kwargs={"role_mappings": role_mappings},
)
Step 3: Tag your DAGs for access control
Add appropriate tags to your DAGs to specify which roles should have access. Tags are used to define which roles have access to tagged DAGs.
# Example DAG for analytics_reporting
with DAG(
"analytics_reporting_dag",
description="Daily analytics reporting pipeline",
schedule_interval="@daily",
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
catchup=False,
tags=["reporting", "analytics", "analytics_reporting_edit"]
) as dag:
# DAG tasks here
pass
# Example DAG for marketing_analyst
with DAG(
"marketing_analyst_dag",
description="Daily marketing lead analysis pipeline",
schedule_interval="@daily",
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
catchup=False,
tags=["marketing", "analytics", "marketing_analyst_edit"]
) as dag:
# DAG tasks here
pass
In this example:
The analytics_reporting custom role will have read, edit, and delete access to the DAG analytics_reporting_dag (and other DAGs tagged with analytics_reporting_edit)
The marketing_analyst custom role will have read, edit, and delete access to the DAG marketing_analyst_dag (and other DAGs tagged with marketing_analyst_edit)
The exact permissions granted (can_read, can_edit, can_delete) are configurable per role in the role_mappings config inside the permission management DAG:
Note: Before this DAG can manage permissions for a custom role, the role must be created manually in the Apache Airflow UI (Security > List Roles) with the Viewer role’s permissions copied in. See Step 2 for details.
Step 4: Deploy and test
Upload both the permission management DAG and your tagged DAGs to your Amazon MWAA environment’s S3 bucket.
Wait for Amazon MWAA to detect and process the new DAGs.
Verify that the permission management DAG runs successfully.
Test access with different user roles to confirm proper permission enforcement.
In this section, we cover some common issues and how to troubleshoot them.
Permission sync failures
Symptom: Permission sync DAG fails with database errors.
Cause: Insufficient permissions on MWAA execution role.
Solution: Ensure that the execution role has airflow:CreateWebLoginToken permission and database access.
Tags not being processed
Symptom: DAG tags are present but permissions aren’t updated.
Solution: Check that DAG is active and parsed successfully – Review permission sync DAG logs for processing errors.
Users cannot access expected DAGs
Symptom: Users with correct IAM roles cannot see DAGs
Solution: Confirm that IAM to Apache Airflow role mapping is correct. Verify that the permission sync DAG has run successfully. Check Amazon CloudWatch Logs for permission assignment errors.
Performance issues
Symptom: Permission sync takes too long or times out.
Solution: Reduce sync frequency for large environments. Consider batching permission updates. Monitor DAG execution time and optimize accordingly.
Debugging steps
Check Amazon MWAA environment health and connectivity
Review permission sync DAG execution logs
Verify IAM role configurations and trust relationships
Test with a single DAG to isolate issues
Monitor CloudWatch Logs for detailed error messages
Benefits and considerations
Automated permission management offers you significant operational advantages while enhancing your security. You will benefit from reduced administrative overhead as manual permission assignments are removed, so you can scale seamlessly without additional burden. Your security improves through consistent application of least-privilege principles and reduced human error. You will enhance your developer experience with automatic access provisioning that shortens onboarding time, while your system supports environments with over 500 DAGs without performance degradation.
When you implement these systems, you must adhere to key security practices. You should apply the principle of least privilege, validate tags to make sure that you’re only processing authorized tags, and establish comprehensive audit mechanisms including CloudTrail logging. Your access control measures should restrict permission management functions to administrators while you utilize appropriate role separation for different user personas.
You will need to consider several technical limitations during your implementation. IAM-based access control to Amazon MWAA works only with Apache Airflow default roles, not custom ones, though your tag-based permissions function with alternative authenticators. Permission changes propagate based on DAG schedules, potentially causing delays. You should establish approval processes for your production changes, maintain version control for permissions, and document your rollback procedures to ensure your system’s resilience and security.
Update the IAM role policy or delete the IAM role if not needed.
Conclusion
In this post, you learned how to automate DAG permission management in Amazon MWAA using Apache Airflow’s tagging system. You saw how to implement tag-based access control that scales efficiently, reduces manual errors, and maintains least-privilege security principles across hundreds of DAGs. You also explored the key security practices and technical considerations that you need to keep in mind during implementation.
Try out this solution in your Amazon MWAA environment to streamline your permission management. Start by implementing the tagging system in a development environment, then gradually roll it out to production as your team becomes comfortable with the approach.
High-performance applications such as CPU-intensive processing, memory-heavy analytics, and steady-state data pipelines often require more predictable compute resources than standard AWS Lambda configurations provide. AWS Lambda Managed Instances (LMI) addresses this by letting you run Lambda functions on selected Amazon EC2 instance types while preserving the Lambda programming model. You can choose over 400 Amazon Elastic Compute Cloud (Amazon EC2) instance types from general purpose, compute optimized, or memory optimized instance families to match workload requirements. AWS Lambda continues to manage infrastructure operations such as instance lifecycle management, operating system patching, runtime updates, request routing, and automatic scaling. This approach gives your teams greater control over compute characteristics, EC2 pricing model and reduces operational overhead of managing servers or clusters.
In this post, you will learn how to configure AWS Lambda Managed Instances by creating a Capacity Provider that defines your compute infrastructure, associating your Lambda function with that provider, and publishing a function version to provision the execution environments. We will conclude with production best practices including scaling strategies, thread safety, and observability for reliable performance.
Figure 1. Creating Function on LMI
Creating Capacity Providers
A Capacity Provider defines the infrastructure blueprint for running LMI functions on Amazon EC2. It specifies instance types, network placement, and scaling behavior. To create a Capacity Provider, you need two parameters: an IAM role (Capacity Provider Operator Role) granting Lambda permissions to launch and manage instances and your VPC configuration with subnets and security groups. Create this role in your account with the AWSLambdaManagedEC2ResourceOperator managed policy following the Principle of Least Privilege (granting only the minimum permissions necessary).
This command creates a Capacity Provider with instance types and scaling configuration:
This command returns a Capacity Provider ARN that you’ll use to create your LMI function. Your functions behavior depends on four main configurations in the capacity provider:
Instance selection
Lambda currently supports three Amazon EC2 instance families (.large and up): C (compute optimized) for CPU-heavy work, M (general purpose) for balanced workloads, and R (memory optimized) for large datasets. Choose x86 (Intel/AMD) or ARM (Graviton) architectures. If you don’t specify instance types, Lambda defaults to appropriate instances based on your function’s memory and CPU configuration. This is the recommended starting point unless you have specific performance requirements. When you need more control, use AllowedInstanceTypes to specify only the instance types that Lambda can use or use ExcludedInstanceTypes to exclude specific types while allowing all other instance types. You can’t use both parameters together.
VPC and networking
Configure multiple subnets across Availability Zones. Lambda creates a minimum Amazon EC2 fleet of three instances distributed across your configured Availability Zones to maintain availability and resiliency. Egress traffic from functions, including Amazon CloudWatch Logs, transits through the Amazon EC2 instance’s network interface in your Amazon Virtual Private Cloud (Amazon VPC). As functions send logs and metrics to CloudWatch, you will need internet access through a NAT Gateway or VPC endpoints with AWS PrivateLink for Amazon CloudWatch. This only affects egress traffic; function invoke requests don’t flow through your VPC. Security groups attached to your instances should allow only the traffic your function code needs. With LMI, configure VPC once at the Capacity Provider level instead of per function, simplifying management for multiple LMI functions. Standard Lambda functions continue to use their own VPC configurations. This Capacity Provider VPC configuration applies only to LMI functions.
Figure 2. LMI Networking
Scaling configuration
Set MaxVCpuCount to cap compute capacity and control costs. New invocations throttle when you reach this limit until capacity frees up. Lambda monitors CPU utilization and scales instances automatically. Choose automatic scaling mode where Lambda tunes thresholds based on load patterns, or manual mode where you set a target CPU utilization percentage. Multiple functions can share the same Capacity Provider to reduce costs through better resource utilization, though you might want separate providers for functions with different performance or isolation requirements.
You create an LMI function similarly to creating a standard Lambda function. You package your code, set your runtime, assign an execution role, and configure memory. The difference is specifying a CapacityProviderConfig to tell Lambda which Capacity Provider to use and how to size each execution environment. Specify CapacityProviderConfig during function creation with the Capacity Provider ARN and configure two execution environment settings. ExecutionEnvironmentMemoryGiBPerVCpu sets the memory-to-vCPU ratio (2:1, 4:1, or 8:1) based on your workload type and PerExecutionEnvironmentMaxConcurrency defines how many concurrent requests share each execution environment. This table shows how memory and vCPU allocation maps across supported execution environment ratio.
2:1 Ratio(Compute optimized)
4:1 Ratio(General purpose)
8:1 Ratio(Memory optimized)
Memory (GB)
vCPU(s)
Memory (GB)
vCPU(s)
Memory (GB)
vCPU(s)
2
1
4
1
8
1
4
2
8
2
16
2
6
3
12
3
24
3
8
4
16
4
32
4
10
5
20
5
12
6
24
6
14
7
28
7
16
8
32
8
…
…
32
16
Function Memory-to-CPU configuration
Set the function’s memory size (up to 32 GB for LMI) and ExecutionEnvironmentMemoryGiBPerVCpu ratio. The default ratio is 2:1. A 2:1 ratio map to compute optimized instances for CPU-intensive tasks like video encoding, 4:1 map to a general purpose for balanced workloads, and 8:1 maps to a memory optimized instances for large in-memory datasets or caching. You must set memory in multiples of the ratio. LMI requires a 2 GB minimum as execution environments need sufficient memory to handle multiple concurrent requests. LMI supports up to 32 GB memory per execution environment.
Multi-Concurrency settings
LMI supports multiple concurrent invocations sharing the same execution environment, reducing cost per invocation by maximizing vCPU utilization. This is particularly effective for I/O-bound workloads, where invocations waiting on database queries or API calls yield vCPU usage to other invocations during idle periods. Lambda defaults to max concurrency per execution environment based on your runtime: Node.js (64 per vCPU), Java, and .NET (32 per vCPU), Python (16 per vCPU). Use PerExecutionEnvironmentMaxConcurrency to set a lower limit based on your workload’s resource needs. Decrease it if you’re experiencing memory pressure or CPU contention. When environments reach their configured max concurrency, new invocations throttle until capacity frees up at the execution environment level. This table captures the maximum concurrency per vCPU for each supported programming language.
Language
Default Max Concurrency
Node.js
64 per vCPU
Java
32 per vCPU
.NET
32 per vCPU
Python
16 per vCPU
This command creates a Lambda function and associates it with your Capacity Provider:
Important: publish a function version before invoking an LMI function. Publishing triggers Lambda to provision Amazon EC2 instances and initialize execution environments, so that the configured baseline capacity is ready before you start invoking. Expect a brief delay before your code goes live as Lambda provisions and launches Amazon EC2 instances. With LMI, execution environments pre-warm after publishing and remain invoke-ready, without cold starts for published versions. Standard Lambda environments initialize on first invoke (cold starts).
This command publishes a Lambda function version and provisions capacity:
After publishing, the function works with standard invocation methods including direct invokes, event source mappings, and service integrations with Amazon API Gateway, Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB Streams, and Amazon EventBridge.
Figure 3. LMI Invocation from event sources
Scaling LMI Functions
Lambda monitors CPU utilization at Capacity Provider level. When CPU utilization reaches the target threshold, Lambda automatically provisions additional EC2 instances, and creates more execution environments on those instances, up to the MaxVCpuCount limit you configured for your capacity provider. As demand decreases, Lambda consolidates workloads onto fewer EC2 instances. You can choose automatic scaling mode (Lambda adjusts thresholds based on your patterns) or manual mode (you set a target CPU percentage). Automatic mode works for variable traffic patterns or when getting started. Manual mode fits when you have predictable patterns and want precise control over scaling thresholds for cost optimization.
Min and max execution environments
Control scaling at the function level with min and max execution environments. The default minimum is 3 execution environments to maintain high availability across Availability Zones. Your total function concurrency equals the number of execution environments multiplied by PerExecutionEnvironmentMaxConcurrency. For example, with min set to 3 and PerExecutionEnvironmentMaxConcurrency of 10, you have provided capacity for 30 concurrent invocations. With max set to 20, you can scale up to 200 concurrent invocations with incoming traffic, based on CPU utilization or concurrency saturation per execution environment. Set max to cap total concurrency and prevent noisy neighbor issues when multiple functions share a Capacity Provider. LMI maintains a minimum number of execution environments with a minimum Amazon EC2 fleet, while standard Lambda scales to zero when idle. Set both min and max to 0 to deactivate a function without deleting it.
Figure 4. LMI Scaling
This command updates the minimum and maximum execution environments for your function:
We’ll cover scaling patterns and throughput optimization strategies in depth in a separate blog post.
Best Practices and Production Considerations
Thread Safety
Since LMI supports multiple invocations sharing execution environments, your code must be thread-safe. Code that isn’t thread-safe causes data corruption, security issues, or unpredictable behavior under concurrent load.
Thread safety essentials
Avoid mutating shared objects or global variables. Use thread-local storage for request-specific data. Initialize shared clients (AWS SDK, database connections) outside the function handler and verify that configurations remain immutable during invocations. Write to /tmp using request-specific file names to prevent concurrent writes.
Runtime-specific guidance
Java applications should use immutable objects, thread-safe collections, and proper synchronization. Node.js applications should use async context for request isolation. Python applications run separate processes per execution environment. So, focus on interprocess coordination and file locking for /tmp access.
Workload Optimization
I/O-bound workloads perform better with higher concurrency per environment. Use asynchronous patterns and non-blocking I/O to maximize efficiency. CPU-bound workloads get no benefit from concurrency greater than one per vCPU. Instead, configure more vCPUs per function for true parallelism for compute-heavy tasks like data transformation or image processing.
Testing
Validate your code under concurrent execution. Test with multiple simultaneous invocations to detect race conditions and shared state issues before production deployment. You can use LocalStack for local emulation of LMI. Learn more about LocalStack’s LMI support in their announcement blog.
Compatibility
Tools like Powertools for AWS work with LMI without code changes. However, if you’re reusing existing Lambda function code, layers, or packaged dependencies on LMI, test for thread safety and compatibility with the multi-concurrent execution model before production deployment.
Observability
LMI automatically publishes CloudWatch metrics at two levels: capacity provider (CPU, memory, network, and disk utilization across your Amazon EC2 fleet) and execution environment (concurrency, CPU, and memory per function). Monitor CPUUtilization to understand scaling headroom and right-size your MaxVCpuCount. Track ExecutionEnvironmentConcurrency against ExecutionEnvironmentConcurrencyLimit to catch throttling before it impacts users. Lambda publishes metrics at 5-minute intervals. Use CloudWatch alarms to stay ahead of capacity limits in production.
Conclusion
AWS Lambda Managed Instances combines serverless simplicity with compute flexibility, helping you run high-performance workloads with reduced operational complexity. You maintain the familiar programming model of Lambda while accessing the diverse instance types of Amazon EC2 and predictable pricing, making it well-suited for data processing pipelines, compute intensive operations and cost-sensitive steady-state applications.
Ready to get started with LMI? Deploy our Monte Carlo risk simulation example from GitHub to see LMI in action with a real compute-intensive workload. The sample includes complete infrastructure code and walks you through capacity provider configuration, function setup, and performance optimization.
We want to hear from you. Share your feedback, questions, and use cases on re:Post.
If you’re architecting cloud systems for AI development on AWS, you’ve likely discovered that traditional architectures create friction for AI agents. Many cloud teams are experimenting with AI coding assistants but quickly discover a gap between what these tools promise and what their architectures allow. When an AI agent generates code, it often takes minutes—or hours—before you can validate whether that change actually works. Slow deployment cycles, tightly coupled services, and opaque code bases turn every iteration into a high-friction exercise. As a result, AI agents struggle to operate autonomously, and developers are forced back into manual validation loops.
This article is written for cloud architects who want to remove that friction. It focuses on agentic development, a model where an AI agent does more than suggest snippets—it writes, tests, deploys, and refines code through rapid feedback cycles. To make that possible, both your system architecture and your code base architecture must be designed to support fast validation, safe iteration, and clear intent.
In this post, we demonstrate how to architect AWS systems that enable AI agents to iterate rapidly through design patterns for both system architecture and code base structure. We first examine the architectural problems that limit agentic development today. We then walk through system architecture patterns that support rapid experimentation, followed by codebase patterns that help AI agents understand, modify, and validate your applications with confidence.
Why traditional architectures hinder agentic AI
Most cloud architectures were designed for human-driven development. They assume long-lived environments, manual testing, and infrequent deployments. In an agentic workflow, those assumptions break down.
AI agents must validate changes continuously. When every test requires provisioning cloud resources, waiting for pipelines, or debugging deployment-only failures, feedback loops become too slow. Tight coupling between business logic and cloud services further complicates local testing, while inconsistent project structures make it difficult for an agent to understand where changes belong.
Without architectural support, agentic AI produces more risk than value. The solution is not better prompts, it’s an architecture that treats fast feedback and clear boundaries as first-class concerns. This architectural friction isn’t only inconvenient, it fundamentally limits AI agent effectiveness. Here’s how to redesign your architecture to help unlock the potential of agentic AI.
System architecture for fast agentic feedback loops
Agentic development depends on feedback speed. The faster an agent can observe the impact of a change, the more effectively it can refine its output. System architecture plays a decisive role here.
Figure 1: High-level architecture enabling agentic development: local test loops, ephemeral test stack, and continuous integration and continuous delivery (CI/CD) pipeline triggered by AI
Local emulation as the default feedback path
Whenever possible, your architecture should allow AI agents to test changes locally before touching cloud resources. AWS provides several tools that make this practical.
For example, serverless applications built with AWS Lambda and Amazon API Gateway can be emulated locally using the AWS Serverless Application Model (AWS SAM). With the sam local start-api command, an AI agent can invoke Lambda functions through a locally emulated API Gateway, observe responses immediately, and iterate in seconds rather than minutes.
Containers offer similar benefits for services that run on Amazon Elastic Container Service (Amazon ECS) or AWS Fargate. By building and running the same container images locally, an agent can validate application behavior before deploying to the cloud. For data persistence, Amazon DynamoDB Local allows the agent to test create, read, update, and delete (CRUD) operations against a local database that mirrors the DynamoDB API.
Note: Local emulation reduces iteration time, allowing AI-generated code to be validated in seconds and potentially reducing the cost and risk of experimentation.
Offline development for data and analytics workloads
Many workloads fit neatly into request-response testing, but data processing pipelines often involve large datasets and distributed execution. Even here, agentic workflows benefit from local feedback.
AWS Glue provides Docker images that allow AWS Glue jobs to run locally with the AWS Glue ETL libraries. An AI agent can validate transformations against sample datasets, inspect intermediate results, and only move to the cloud for scale testing. The same pattern applies to other data and machine learning (ML) workloads: isolate logic, test locally with reduced data, and promote validated code to managed services later.
Note: Offline development shortens feedback loops for data workloads and reduces unnecessary cloud runs during early iteration.
Hybrid testing with lightweight cloud resources
Some AWS services cannot be fully emulated locally. In these cases, the goal is not to avoid the cloud, but to keep cloud feedback lightweight.
For event-driven systems using Amazon Simple Notification Service (Amazon SNS) or Amazon Simple Queue Service (Amazon SQS), you can define minimal development stacks using infrastructure as code (IaC) tools such as AWS CloudFormation or the AWS Cloud Development Kit (AWS CDK). An AI agent can deploy small, isolated resources, invoke them through the AWS SDK, and validate behavior without provisioning full environments.
This hybrid approach treats the cloud as another test dependency—used sparingly and predictably.
Note: Hybrid testing confirms real service behavior early while keeping cloud usage focused and controlled.
Preview environments and contract-first design
Fast feedback does not stop at local testing. End-to-end validation still matters, especially when multiple services interact.
Preview environments are short-lived stacks deployed on demand for validation. Defined through IaC, they allow an AI agent to deploy a complete application, run smoke tests, and tear everything down when finished. When combined with contract-first design—where APIs are defined upfront using OpenAPI specifications—agents can validate integrations even before all services are implemented.
Note: Preview environments can reduce integration risk and allow AI-generated changes to be validated safely before reaching production.
Code base architecture for AI-friendly development
System architecture accelerates feedback, but code base architecture determines whether an AI agent can make sense of what it is changing.
Domain-driven structure with explicit boundaries
We recommend agentic development when your repository reflects clear architectural intent. A domain-driven structure inspired by Domain-Driven Design (DDD) separates core business logic from application orchestration and infrastructure concerns.
In practice, this often means organizing code into predictable layers such as /domain, /application, and /infrastructure. The domain layer contains business rules with no Amazon dependencies. Infrastructure code handles integrations with services such as Amazon DynamoDB or Amazon SNS. This separation allows AI agents to modify business logic and validate it locally without touching cloud-specific code.
Patterns like hexagonal architecture reinforce this separation by treating external systems as adapters rather than dependencies.
Note: Clear boundaries can reduce unintended side effects and make AI-generated changes more straightforward to reason about and test.
For example, a rule might state that database access must go through repository classes in the infrastructure layer. The agent consults these rules automatically, reducing the need to restate constraints in every prompt and helping to keep generated code aligned with your architecture.
Note: Project rules reduce architectural drift and help maintain consistency as AI agents operate more autonomously.
Tests as executable specifications
In agentic workflows, tests do more than catch regressions, they define acceptable behavior. A layered testing strategy works particularly well:
Unit tests validate domain logic in isolation and run quickly, making them ideal for frequent AI-driven iterations.
Smoke tests run against deployed environments to surface configuration or permission issues that only appear at runtime, such as missing AWS Identity and Access Management (IAM) permissions.
Well-written tests also act as documentation. When a test fails, the agent can infer what behavior is expected and refine its changes accordingly.
Note: Tests provide fast, objective validation of AI-generated code and reduce the risk of subtle integration failures.
Monorepos and machine-readable documentation
AI agents work more effectively when they have broad context. A monorepo allows the agent to navigate across services, understand shared patterns, and evaluate the impact of changes system-wide. Within that repository, concise and structured documentation is essential. Files such as AGENT.md can explain architectural principles and constraints, while RUNBOOK.md and CONTRIBUTING.md describe operational and development workflows. Machine-readable formats, such as YAML or configuration files, are more straightforward for agents to interpret than lengthy prose.
Kiro can use foundational steering documents—summaries of structure, technology, and product guidelines—to help the agent maintain situational awareness as the project evolves.
Note: Shared context improves the quality of AI-generated changes and reduces the need for manual correction.
Integrating agents safely into delivery pipelines
As AI agents become more capable, governance remains essential. Continuous integration and continuous deliver (CI/CD) pipelines should include guardrails such as required test execution, automated reviews, and branch protections. Over time, as confidence grows, you can expand the agent’s autonomy while keeping humans in the loop for high-impact decisions. This balance allows AI to accelerate routine work without increasing operational risk.
Conclusion
Agentic AI development does not succeed by accident. It requires architectures that prioritize fast feedback, clear boundaries, and explicit intent. Combining local emulation, lightweight cloud testing, and preview environments with domain-driven structure, layered testing, and machine-readable documentation creates an environment where AI agents can operate effectively and safely. Tools like Kiro help bridge the gap between human design decisions and autonomous AI execution. When architecture aligns with agentic workflows, AI agents become true force multipliers, handling iterative development at speed while your team focuses on higher-level design and innovation.
To learn more about how AWS can help your organization implement agentic solutions, visit AWS Agentic AI.
About the authors
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
