Tag Archives: Intermediate (200)

How Moeve standardized dbt runs across data lakes with Amazon Athena

2026-09-22 Rubén Romero Córdoba

Post Syndicated from Rubén Romero Córdoba original https://aws.amazon.com/blogs/big-data/how-moeve-standardized-dbt-runs-across-data-lakes-with-amazon-athena/

As organizations grow, data processing often becomes fragmented across teams, environments, and orchestration tools. This fragmentation leads to inconsistent patterns, duplicated logic, limited cost visibility, and operational overhead.

At Moeve we were no exception. Our analytics teams build their transformations with dbt, an open source tool that defines transformations as SQL models, resolves the references between them, and works out the order in which they run. dbt describes what to transform, but it does not define where or how a project runs. We left that decision to each team, and as the number of projects grew we found fragmented pipelines, inconsistent compute engines, and limited cost visibility slowing every project down. Standardizing our dbt runs on Amazon Athena was how we worked our way out of that.

This post describes the architecture of the centralized, serverless solution we built on Athena, which reduced onboarding for a new dbt project from days to about 15 minutes. The solution centralizes how dbt runs across our data lakes while staying loosely coupled from orchestration. It uses Amazon Athena as the default processing engine, a centralized dbt launcher, and a shared event bus for downstream orchestration.

In the sections that follow we explain how we decoupled dbt runs from orchestration using AWS Step Functions and AWS Fargate, why Athena fits our workloads from a cost and operational perspective, how storing run parameters in Amazon DynamoDB rather than in pipeline code removed the infrastructure deployment step from onboarding, and how publishing results to Amazon EventBridge lets our run and orchestration layers evolve independently.

The challenge of running dbt at scale

Before the dbt launcher, our dbt runs had grown in different directions. Run logic was embedded in project-specific pipelines, orchestration and processing were tightly coupled, teams selected different compute engines for comparable workloads, and we had no consistent governance over run parameters and retries.

As the number of dbt projects increased, this made it difficult to enforce consistent standards and to evolve the solution without touching every pipeline. We needed a way to standardize dbt runs across our data lakes, decouple running a project from deciding what to run, improve cost control and observability, and deliver faster and safer continuous integration and continuous delivery (CI/CD) iterations.

Why Amazon Athena as the default dbt engine

Choosing the processing engine for dbt is a foundational architectural decision, so we made it first.

Serverless processing

Athena is fully serverless. There are no clusters to provision, scale, or maintain. Our teams run their queries with the default Athena pricing, which charges for the data a query scans and gives us elasticity with no capacity planning.

Because each data lake lives in its own account, each team makes its own decision about Athena payment. A team whose workload grows into continuous, high-concurrency usage can move to Athena capacity reservations with no change to the launcher, to their dbt profiles, or to their project configuration. None of our teams have needed to do so yet, and the architecture keeps that choice independent per team.

Our workload is predictable but not continuous. Each dbt project runs for a few minutes when its schedule fires or when its upstream data lands, then stays idle until the next trigger. That shape is what made serverless the right fit for us.

Why Athena fit our solution

For Moeve, the decision to standardize on dbt and Amazon Athena was driven by our goal of creating a common transformation solution that could be adopted across multiple teams and AWS accounts while keeping operations lightweight.

Our data was already stored in Amazon S3 and registered in the AWS Glue Data Catalog, making Athena a natural processing layer. Athena allowed us to run transformations without managing clusters, capacity, or infrastructure, which was particularly important for a small central team supporting multiple domains.

We evaluated alternative processing engines, but for our workload profile and data volumes, Athena provided the best balance between scalability, operational simplicity, and maintainability. Adapter maturity was another important factor. The dbt Athena adapter offered strong integration with testing, CI/CD workflows, and the broader dbt ecosystem, reducing the operational risk of maintaining custom solutions.

Athena also aligned naturally with our architecture. Transformations run in the AWS account that owns the data through cross-account role assumption, while orchestration remains centralized. As a result, we standardized how projects run across teams while keeping compute close to the data.

Finally, the Apache Iceberg support in Athena underpins the idempotent incremental processing model described in this post, so incremental loads and historical reprocessing follow the same path with minimal operational overhead.

Optimized incremental processing

Our largest cost was not reading source data. It was merging into it.

Our fact tables are Apache Iceberg tables in the data lake, and most of our models are incremental. Each run brings in new or corrected records and merges them into a target that can hold several years of history. A merge has to locate the rows it is about to update, and without a predicate on the target the query reads far more of the table than the incoming data can affect. The common approach is a static filter such as the last 30 days, which is wrong in both directions: too wide for an ordinary daily load, and too narrow as soon as a correction arrives for an older partition.

Instead of a fixed window, the platform derives the predicate from the data. Before the merge runs, it reads the distinct values of the partition column present in the incoming dataset and builds the target predicate from them. For a single partition it applies an equality predicate, for a small set an IN list, and for a larger set a bounded range. The merge then reads only the partitions the incoming data can affect.

The same principle applies on the source side. The launcher builds the source filter from the parameters given for that run: an explicit range, an arbitrary SQL condition, or, when neither is supplied, a default window taken from the project configuration. Input is therefore bounded to the subset each run needs.

Two results mattered to us. Because Athena charges for the data a query scans, narrowing both ends of the merge reduces cost without any team hand-tuning individual models. More importantly, a daily load and a full historical reprocess became the same operation with different inputs. Every run is idempotent, so the same input always produces the same result regardless of how many times it runs. That removed the distinction between processing and reprocessing from our runbooks and simplified incident response.

Table design is what makes this pruning possible. Partitioning, columnar formats, and compression all contribute, and the AWS Big Data Blog post Top 10 performance tuning tips for Amazon Athena covers the general techniques. We have deliberately not published a before and after figure here, because the saving depends so heavily on partition design and data distribution that a single number would mislead without extensive context.

Architecture overview

Moeve built a centralized dbt launcher that runs dbt jobs in a uniform way, regardless of the project or the target data lake.

Architecture diagram

Architecture of the centralized dbt launcher running cross-account dbt jobs on Step Functions, Fargate, and Amazon Athena

Figure 1: Centralized dbt launcher and cross-account run flow

At a high level, the architecture consists of:

AWS Step Functions to control the run lifecycle.
AWS Fargate to run dbt in an isolated, ephemeral container.
Amazon Athena as the default dbt processing engine.
Amazon DynamoDB to store dbt project configuration.
Amazon EventBridge to publish run results.

Every dbt run follows the same contract, which gives us consistency and reduces the operational surface we must maintain.

Cross-account processing model

The solution operates in a centralized account while running transformations in domain-specific data lake accounts: corporate, marketing, and manufacturing. The Fargate container assumes a dbt-child IAM role in the target account, so the container processes data where it lives while governance stays centralized.

Each data lake account keeps control of its own IAM permissions and manages its own storage and catalog without affecting the solution. This also puts costs in the right account. We could have attributed Athena spend using Athena workgroups, but Athena is only part of what a query costs. The Amazon Simple Storage Service (Amazon S3) requests it makes and the AWS Key Management Service (AWS KMS) operations it triggers are real costs as well, and running in the owning account attributes all of them to the team that owns the data, per project and per run.

Centralizing dbt runs with the dbt launcher

To stop every team inventing its own way of running dbt, we built a single launcher that all of them go through. Instead of dbt logic living inside multiple pipelines, every run is triggered through one well-defined path.

Run lifecycle

The launcher is an AWS Step Functions state machine. It receives a run request with its parameters, starts an AWS Fargate task from our dbt container image, and the container assumes the IAM role of the target data lake account. dbt then runs its SQL transformations in Athena, reading the source tables and materializing the targets. Alongside the run, Elementary, an open source dbt package, records model-level results and data quality test outcomes. When the run finishes, the launcher publishes a completion event to Amazon EventBridge.

The state machine can be started in different ways depending on the scenario. Some projects run on a schedule, others are triggered when upstream data lands, and teams can request a run on demand. Those decisions are made by our orchestration layer, which submits a standardized run request to the launcher. The launcher therefore stays focused on running dbt projects, regardless of how the run was initiated.

This sequence is identical for all projects and environments. The launcher is responsible only for running the project it was asked to run. It doesn’t decide what should run next, and that responsibility is intentionally delegated to downstream consumers through Amazon EventBridge.

Configuration-driven runs with Amazon DynamoDB

We had to decide where a project’s run parameters would live. In the pipeline definition, changing a timeout would be a code change, a review, a build, and a deployment, for a value we sometimes need to change while an incident is open. We put the parameters in DynamoDB instead, keyed by project, and the launcher reads them at the start of every run. That lets us decouple code deployment from run behavior, update parameters without redeploying services, and enforce consistent defaults across all dbt projects.

The parameters themselves are modest. They cover the target environment and AWS Identity and Access Management (IAM) role, the default processing engine, how long to allow a run to take, how many times to retry on failure, and how many days of data to process by default. These values change for operational reasons rather than logical ones, which is why we did not want them coupled to a release cycle.

The effect on onboarding was larger than we expected. Deploying a new dbt project is now a merge of the dbt models and one configuration entry. There is no Terraform change, no infrastructure review, and nothing to provision, because the compute the project needs already exists and is shared. What used to be a multi-step infrastructure pipeline is now a single CI workflow that validates the project’s SQL and lineage locally, then merges and registers it. We run that local validation with DuckDB, which returns feedback in under two minutes without consuming cloud resources.

Governance did not weaken as a result. Who may change a configuration entry is controlled the same way as any other production change. What changed is that the change no longer has to travel through an infrastructure deployment to take effect.

CI/CD pipeline diagram

A pull request triggers local validation of the project’s SQL and lineage. On merge, the dbt models are deployed and the project’s configuration entry is registered in DynamoDB, after which the launcher can run the project.

Figure 2: CI workflow for onboarding and updating a dbt project

Athena remains the engine of record. Local validation catches Jinja errors, unresolved references, and obvious SQL mistakes, but Athena-specific behavior, cross-account permissions, and AWS Glue Data Catalog interactions are only proven in the target environment. It’s important to be explicit about that boundary with the teams, so that a green CI run is not read as a guarantee.

Publishing run results with Amazon EventBridge

After a dbt run finishes, the launcher publishes a structured event to a central Amazon EventBridge bus recording whether the run succeeded, which project and source it covered, when it started and finished, how long it took, and which datasets it updated. The launcher does not know which consumers are subscribed.

This event-driven approach gives us loose coupling between running a project and orchestrating what comes next, multiple downstream consumers for the same signal, and independent evolution of both layers.

At Moeve the main consumer is our orchestration layer, which models the dependencies between datasets as a graph. Each completion event tells it that a node is now up to date, so it can determine which downstream projects have all their inputs ready and start them. That consumer has no special status. An AWS Lambda function, a monitoring dashboard, or a notification integration can subscribe to the same events without any change to the launcher.

Validation and observability

Because every project follows the same lifecycle, we get validation and observability in one place instead of per pipeline. Step Functions shows the state of any run and the step at which it failed, and error handling and retries are defined once. Fargate logs carry the container runtime detail. dbt and Elementary report model-level results and data quality test outcomes. The completion event on the bus is the auditable record that a project finished and what it produced. Together these layers give us operational visibility without coupling the components to each other.

Cost control and resource cleanup

The platform is serverless end to end, which keeps idle cost close to zero and removes a class of operational mistake. Fargate tasks are created for a run and destroyed when it ends, so no long-running container needs maintenance. Athena has no persistent compute. An idle Step Functions state machine costs nothing. Nothing is left running unintentionally, which matters when the number of projects on the platform keeps growing.

Results

The metrics in the following table are the ones our teams notice day to day, and the reason the solution is maintainable by a small central team.

Metric	Before	After
Onboarding time for a new dbt project	Days, including pipeline and infrastructure setup	About 15 minutes, configuration only
Run consistency	Varied by team	Same lifecycle and contract for every project
CI feedback time	8 to 10 minutes on Jenkins	Under 2 minutes with local validation
Cost visibility	Per-account aggregate	Per-project and per-run attribution
Operational overhead	One pipeline per project	One solution for all projects

Conclusion

Standardizing how we run dbt turned out to depend less on dbt than on defining two boundaries clearly.

The first is the contract of the launcher: parameters in, event out. We defined that interface before building the internals, and it has stayed stable while the implementation changed several times. The second is the separation between running a project and deciding what to run next. Making the launcher publish events without knowing its consumers is why our orchestration layer could be rebuilt while the launcher stayed as it was, and the launcher has never been modified to accommodate a new orchestration requirement.

Two smaller decisions carried more weight than we expected. Keeping run parameters in DynamoDB rather than in pipeline definitions means timeouts, retries, and engine selection can be changed without a deployment, which is valuable during incident response. Making every run idempotent removed the distinction between processing and reprocessing, so a daily load and a full historical reprocess are the same operation with different inputs.

Athena is serverless, so our teams did not need to set up infrastructure of their own. That is what made it practical for everyone to run dbt in a standardized way, and why onboarding a new project went from days to about 15 minutes.

If your organization runs dbt across multiple accounts and teams, pair Amazon Athena with a clear contract for the component that runs your projects: fixed parameters in, a published event out.

Resources

About the authors

Achieving CNIL/EU ePrivacy compliance for email tracking with Amazon SES

2026-09-19 Toni Pivcevic

Post Syndicated from Toni Pivcevic original https://aws.amazon.com/blogs/messaging-and-targeting/achieving-cnil-eu-eprivacy-compliance-for-email-tracking-with-amazon-ses/

Open and click tracking have long been foundational email metrics. Multiple data protection authorities, including in EU and Canada, have issued guidance requiring explicit opt-in consent before deploying open tracking pixels or click link wrapping in email. As privacy expectations evolve, a growing number of jurisdictions now require senders to obtain explicit consent before tracking whether a recipient opened an email or clicked a link. In this post, you learn how to use Amazon Simple Email Service (Amazon SES) configuration set overrides to control open and click tracking per request.

How Amazon SES tracking works

Amazon SES enables open and click tracking only when explicitly configured by you. Amazon SES does not inject tracking pixels or wrap links by default.

Open tracking — When you add an event destination publishing OPEN events to a configuration set, Amazon SES inserts a 1×1 tracking pixel (served from awstrack.me) into HTML email sent using that configuration set. When a recipient opens the email and their client loads images, Amazon SES records the open event and publishes it to the configured destination.

Click tracking — When you include CLICK events in a configuration set’s event destination, Amazon SES rewrites links in HTML email to redirect through awstrack.me, recording click events before sending the user to the original URL.

Configuration sets — Configuration sets are the control surface for both features. Tracking activates only when an event destination explicitly includes OPEN or CLICK in its matching event types. You can have multiple configuration sets with different tracking configurations and select the appropriate one at send time.

Per-request tracking overrides — Amazon SES now supports open and click tracking override parameters directly in the SendEmail and SendBulkEmail APIs through the ConfigurationOverrides object. You can enable or disable open tracking and click tracking on an individual API call, without maintaining separate configuration sets. The override takes precedence over the tracking behavior defined in the associated configuration set, giving you fine-grained, per-recipient control at send time. This is the most direct way to honor recipient-level consent choices.

Event publishing and metrics — Open and click events are published to the destinations you configure: Amazon CloudWatch, Amazon Data Firehose, or Amazon EventBridge. Plan for reduced fidelity in open-rate dashboards for recipients in jurisdictions where you cannot track without consent.

Note on Amazon SES Contact Lists: Contact Lists manage topic-level subscription preferences (for example, marketing versus transactional) and control whether Amazon SES delivers to a contact. They do not control tracking behavior. There is no native mapping between a contact’s subscription status and tracking pixel injection. Tracking consent must be managed separately in your application.

Prerequisites

To follow the steps in this post, you should be familiar with Amazon SES configuration sets and basic email sending concepts. You also need the following:

An AWS account with Amazon SES out of sandbox mode.
AWS Identity and Access Management (IAM) permissions to create and manage Amazon SES configuration sets (ses:CreateConfigurationSet, ses:CreateConfigurationSetEventDestination, ses:DeleteConfigurationSet).
The AWS Command Line Interface (AWS CLI) version 2 installed and configured, or access to the AWS SDK for Python (Boto3).

Option A: Use per-request tracking overrides (recommended)

The most direct approach to consent-based tracking uses the ConfigurationOverrides parameter in the SendEmail API. This approach requires only a single configuration set with tracking-enabled event destinations. At send time, you override the tracking behavior based on each recipient’s consent status.

AWS SDK:

import boto3

ses_client = boto3.client("sesv2", region_name="us-east-1")

def send_campaign_email(subscriber: dict, subject: str, html_body: str):
    # Determine tracking behavior based on recipient consent
    tracking_consent = subscriber.get("tracking_consent", False)

    # Use ConfigurationOverrides.Tracking to control per-request behavior
    ses_client.send_email(
        FromEmailAddress="[email protected]",
        Destination={"ToAddresses": [subscriber["email"]]},
        Content={
            "Simple": {
                "Subject": {"Data": subject},
                "Body": {"Html": {"Data": html_body}},
            }
        },
        ConfigurationSetName="my-config-set",
        ConfigurationOverrides={
            "Tracking": {
                "OpenTrackingEnabled": tracking_consent,
                "ClickTrackingEnabled": tracking_consent,
            }
        },
    )

The ConfigurationOverrides.Tracking object takes precedence over the configuration set’s event destination settings for that individual send. If the recipient has not consented, Amazon SES does not inject the tracking pixel or wrap links, regardless of whether the configuration set has OPEN and CLICK events enabled.

Advantages of per-request overrides:

No need to maintain separate configuration sets for tracked versus untracked sends.
A single configuration set can handle all recipients, which simplifies event destination management, suppression, and DomainKeys Identified Mail (DKIM) and domain settings.
Per-recipient control without branching logic for configuration set selection.
Works with SendBulkEmail as well, setting tracking overrides per recipient in the bulk request.

SMTP interface — If you send through SMTP, per-request overrides are not available. Use Option B (separate configuration sets) instead.

Option B: Use separate configuration sets

If you send through SMTP or prefer to separate tracking behavior at the configuration set level, create two configuration sets: one with tracking enabled for consented recipients, and one with no tracking for non-consented recipients.

Console: In the Amazon SES console, choose Configuration sets, and then choose Create configuration set. To enable tracking on the first set, add an event destination and include OPEN and CLICK in the matching event types. Create a second set with no OPEN or CLICK event destinations, or omit event destinations entirely.

AWS CLI:

# Configuration set with tracking enabled (for consented recipients)
aws sesv2 create-configuration-set \
    --configuration-set-name tracking-enabled

aws sesv2 create-configuration-set-event-destination \
    --configuration-set-name tracking-enabled \
    --event-destination-name open-click-destination \
    --event-destination '{
    "Enabled": true,
    "MatchingEventTypes": ["SEND","DELIVERY","BOUNCE","COMPLAINT","OPEN","CLICK"],
    "CloudWatchDestination": { ... }
}'

# Configuration set with no tracking (for non-consented recipients)
aws sesv2 create-configuration-set \
    --configuration-set-name no-tracking

Replace { … } with your CloudWatch destination configuration. For the full parameter structure, see Managing Amazon SES event destinations.

No event destination for OPEN or CLICK means no pixel is injected and no links are wrapped.

Send-time selection:

Your application selects the configuration set for each message based on the recipient’s consent status.

import boto3

ses_client = boto3.client("sesv2", region_name="us-east-1")

def send_campaign_email(subscriber: dict, subject: str, html_body: str):
    config_set = (
        "tracking-enabled"
        if subscriber.get("tracking_consent")
        else "no-tracking"
    )
    ses_client.send_email(
        FromEmailAddress="[email protected]",
        Destination={"ToAddresses": [subscriber["email"]]},
        Content={
            "Simple": {
                "Subject": {"Data": subject},
                "Body": {"Html": {"Data": html_body}},
            }
        },
        ConfigurationSetName=config_set,
    )

SMTP interface — Set the configuration set through the X-SES-CONFIGURATION-SET header:

X-SES-CONFIGURATION-SET: no-tracking

Disabling tracking on individual links

To disable click tracking on individual links within a tracked email (for example, your unsubscribe link), use the ses:no-track attribute:

<a ses:no-track href="https://anycompany.example.com/unsubscribe">Unsubscribe</a>

Amazon SES strips the ses:no-track attribute before delivery, so recipients never see it. This works with both Option A and Option B.

Regardless of which option you choose, Amazon SES provides the mechanisms to enable or disable tracking at send time. You are responsible for:

Maintaining a consent database recording each recipient’s tracking consent status.
Determining at send time whether a recipient has consented to open and click tracking.
Passing the correct override parameter (Option A) or selecting the appropriate configuration set (Option B) based on that determination.

Important: Once an email is delivered with a tracking pixel, it cannot be retroactively deactivated.

Capture consent at sign-up. France’s data protection authority (CNIL) recommends collecting tracking consent at the point of email address collection. Include a clearly labeled, unchecked checkbox. For example: “I agree to allow AnyCompany to track whether I open or click email to improve future communications.” This is a separate checkbox from consent to receive marketing email.

Store consent with proof. You must be able to demonstrate valid consent for each individual. Store a consent timestamp and source alongside the subscriber record:

import boto3
from datetime import datetime, timezone

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("Subscribers")

def save_subscriber(email: str, marketing_consent: bool, tracking_consent: bool):
    table.put_item(
        Item={
            "email": email,
            "marketing_consent": marketing_consent,
            "tracking_consent": tracking_consent,
            "tracking_consent_timestamp": datetime.now(timezone.utc).isoformat(),
            "tracking_consent_source": "sign-up-form-v2",
        }
    )

Make refusal as straightforward as acceptance. Do not use pre-checked boxes or dark patterns that make opting out harder than opting in.

Provide withdrawal at any time. Every email sent with tracking enabled must include a link that recipients can use to withdraw tracking consent independently of unsubscribing. A “Manage email preferences” link in the footer, separate from the unsubscribe link, satisfies this requirement:

<p style="font-size:12px; color:#666;">
  <a href="https://anycompany.example.com/email-preferences?token={SUBSCRIBER_TOKEN}">Manage email preferences</a>
  &nbsp;|&nbsp;
  <a href="{UNSUBSCRIBE_URL}">Unsubscribe</a>
</p>

Replace {SUBSCRIBER_TOKEN} with a signed token that identifies the subscriber’s record in your consent database. Replace {UNSUBSCRIBE_URL} with your Amazon SES one-click unsubscribe URL or list-unsubscribe endpoint.

Monitoring and governance

Audit event volumes. Use Amazon SES event publishing to monitor open and click event counts. If you use Option B, compare volumes between tracking-enabled and no-tracking. If you use Option A, monitor the ratio of sends with tracking enabled versus disabled. A sudden drop in consented opens might indicate an issue with your consent capture flow.

Review data retention. If you rely on the deliverability-only exemption for any segment, verify that your data pipeline retains only the date of last open, not the time, IP address, or user-agent string. Collecting those fields, even temporarily, voids the exemption per CNIL guidance.

Document your consent architecture. Maintain a record of how and when each subscriber provided consent, which form version was in use, and where the consent data is stored.

Clean up

If you created test configuration sets while following this post and do not intend to use them, delete them to avoid unintended configuration being applied to future sends.

aws sesv2 delete-configuration-set --configuration-set-name tracking-enabled
aws sesv2 delete-configuration-set --configuration-set-name no-tracking

Conclusion

Amazon SES gives you the tools to obtain prior consent before tracking opens and clicks. With per-request tracking overrides, you can honor consent decisions inline with each API call, with no additional configuration sets required. For senders using SMTP, separate configuration sets achieve the same outcome. Combined with consent capture at sign-up and a preferences management link in every tracked email, you can maintain meaningful analytics for consented recipients while respecting the privacy of those who have not opted in.

For more information, see the following resources in the Amazon SES Developer Guide:

About the authors

Connect Amazon SageMaker Unified Studio to Microsoft Power BI – Part 1: IAM Identity Center (IDC)-based domains

2026-09-15 Ramesh H Singh

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/connect-amazon-sagemaker-unified-studio-to-microsoft-power-bi-part-1-iam-identity-center-idc-based-domains/

Connecting Power BI to your Amazon SageMaker Unified Studio data catalogs typically required third-party bridges. These bridges added complexity and licensing costs. In this post, you create a direct connection using new authentication modes in the Amazon Athena ODBC driver, removing those dependencies entirely. If your organization uses Power BI as its business intelligence (BI) tool, your analysts can configure access to governed data in Amazon SageMaker Unified Studio without changing their tools or workflows. As an AWS alternative, Amazon Quick Sight provides serverless BI integration with Amazon SageMaker Unified Studio at pay-per-session pricing.

A previous post showed the connection method using a third-party ODBC-JDBC bridge. The Amazon Athena ODBC driver (version 2.2.0 and later) now supports Amazon SageMaker Unified Studio authentication directly, eliminating the need for customers to configure third-party bridge components previously required for this connection. This bridge also created additional components and required ongoing maintenance. The native connection simplifies the architecture by reducing these requirements.

UC Irvine, a top-ten U.S. public research university, consolidates student data from systems across multiple departments into a single governed repository that supports reporting, research, and analytics for decision-making at the strategic, tactical, and operational levels. Many of their analysts rely on Power BI to explore and visualize this governed data.

“Our users rely on Power BI for data visualization and reporting, but connecting to governed data in AWS previously required workarounds. The ODBC connection feature gives a direct path from Power BI into our SageMaker Unified Studio projects—no bridge software, no extra licensing, just a connection string and we’re ready to go.”

— Bernadette Theologidy, Manager, Student Analytics, UC Irvine

The Athena ODBC driver introduces two new authentication modes for SageMaker Unified Studio:

SageMakerBrowserIdc (for IDC-based domains): The driver opens a browser window and authenticates through AWS IAM Identity Center (and your external identity provider, if configured). No local AWS credentials are needed.
SageMakerIam (for AWS Identity and Access Management (IAM)-based and IDC-based domains): The driver uses AWS credentials from the default credential provider chain. For this walkthrough, we use AWS IAM Identity Center to provide those credentials.

You connect Microsoft Power BI to Amazon SageMaker Unified Studio through Athena. The Athena ODBC driver supports using two connection methods that use these authentication modes:

Method 1: DSN-based (Athena Power BI connector): You configure an ODBC Data Source Name (DSN) and use the Athena connector in Power BI. This method supports DirectQuery and Import mode with both SageMakerBrowserIdc and SageMakerIam authentication.

Method 2: DSN-less (Power BI ODBC connector): You use the Power BI ODBC connector with a connection string, requiring no DSN configuration. This method supports Import mode only with SageMakerIam authentication. DirectQuery isn’t available because the Power BI ODBC connector doesn’t support it. The connection string in Power BI Desktop must match exactly the one on Power BI Service. Because the gateway runs as a Windows service without interactive browser access, both ends must use SageMakerIam.

Feature	Method 1: DSN-based	Method 2: DSN-less
Power BI Connector	Amazon Athena connector	ODBC connector
Data connectivity mode	DirectQuery and Import	Import only
Requires DSN configuration	Yes	No
Data freshness	Real-time (DirectQuery) or scheduled (Import)	Scheduled refresh only
Authentication types	SageMakerIam and SageMakerBrowserIdc	SageMakerIam only
Domain types supported	IAM-based and IDC-based	IAM-based and IDC-based
Best for	Dashboards requiring live data	Scenarios where DSN management is not possible or scheduled refresh is acceptable

This is Part 1 of a two-part series. This post covers IDC-based domains using both connection methods. Part 2 covers IAM-based domains.

Solution overview

In this walkthrough, you take the role of a data analyst at an energy company. You need to understand the current state and future direction of the U.S. power generation fleet using the Public Utility Data Liberation Project, available on the Registry of Open Data on AWS. Our goal is to analyze generation capacity and identify where new investment is flowing. We connect Power BI to Athena through Amazon SageMaker Unified Studio and query the EIA-860 generators dataset directly from our data catalog. The result is a single visualization that reveals the energy transition.

The following diagram illustrates the solution architecture for connecting Power BI to Amazon SageMaker Unified Studio through Amazon Athena.

Architecture diagram showing Power BI connecting to Amazon Athena through Amazon SageMaker Unified Studio, with a Microsoft on-premises data gateway on Amazon EC2

Figure 1: Architecture diagram

The following architecture demonstrates a six-step workflow.

Data engineers and analysts connect Power BI Desktop to Athena as a data source.
They build their reports locally.
They then publish them to the Power BI Service.
Microsoft On-Premises Data Gateway on an Amazon Elastic Compute Cloud (Amazon EC2) instance connects to Athena using the instance’s attached IAM role.
The Power BI Service then uses this gateway connection.
Report viewers access the published reports through Power BI Service to make data-driven decisions.

On the AWS side, Athena queries the data catalog managed by AWS Glue Data Catalog. The catalog references data stored in Amazon Simple Storage Service (Amazon S3). An Amazon SageMaker Unified Studio project governs all access.

In an IDC-based domain (covered in this post), Power BI Desktop uses SageMakerBrowserIdc for Method 1 and SageMakerIam for Method 2. Power BI Desktop can run on-premises or on an EC2 instance. The gateway always uses SageMakerIam (it runs as a Windows service without browser access) and authenticates using instance profile credentials, which rotate automatically. The gateway can only query data within projects where its IAM role has been added as a member. For IAM-based domains, see Part 2.

Prerequisites

Before connecting Power BI to Amazon SageMaker Unified Studio, verify that your environment meets these requirements:

Athena ODBC driver – The latest Amazon Athena ODBC driver (version 2.2.0 or more recent) for Windows 64-bit.
Microsoft Power BI Desktop – The latest version installed on your Windows machine.
Microsoft Power BI Pro License – Required for publishing reports and configuring the on-premises data gateway.
Microsoft Power BI on-premises data gateway – The latest version installed on the EC2 instance.
Amazon SageMaker Unified Studio – An Amazon SageMaker Unified Studio IDC-based domain.

You need an Amazon SageMaker Unified Studio project with data assets. For detailed instructions, refer to the Amazon SageMaker Unified Studio User Guide.

The following screenshot shows the Amazon SageMaker Unified Studio project Query Editor interface, which runs a preview query against the EIA-860 generators dataset.

SageMaker Unified Studio Query Editor previewing the EIA-860 generators dataset

Figure 2: SageMaker Unified Studio project with the EIA-860 generators dataset available in the data catalog

Method 1: DSN-based connection (Athena Power BI connector)

This method uses the Amazon Athena Power BI connector with an ODBC Data Source Name (DSN), supporting DirectQuery and Import mode.

You configure Power BI Desktop to connect to your data assets in Amazon SageMaker Unified Studio using the SageMakerBrowserIdc authentication mode. The driver opens a browser window and authenticates through IAM Identity Center (and your external identity provider, if configured).

Add your SSO user as a member of your SageMaker Unified Studio project

Your single sign-on (SSO) user needs project-level access to query data with Athena. Verify your user is listed as a project member or add it by following Add project members in the Amazon SageMaker Unified Studio User Guide.

The following screenshot shows the SageMaker Unified Studio project user management page, where project owners can add or remove project users and roles.

SageMaker Unified Studio project members page listing users and roles

Figure 3: Members of a SageMaker Unified Studio project

Gather configuration values to configure your Amazon Athena ODBC DSN

Gather the following values from your Amazon SageMaker Unified Studio project:

Open your Amazon SageMaker Unified Studio project.
In the top right, select the three dots.
Choose Project details.
Select JDBC and ODBC details.
Under ODBC connection details copy the following information: IDC issuer URL, domain ID, project ID, Athena workgroup name and AWS Region.

The following screenshot shows the Amazon SageMaker Unified Studio project overview page, where you can copy these details.

SageMaker Unified Studio project overview showing ODBC connection details

Figure 4: ODBC connection details

Configure the ODBC DSN

Create a System DSN using the Amazon Athena ODBC driver. For the general DSN creation steps, see Configuring a data source name on Windows in the Amazon Athena User Guide.

Enter the following values:

Field	Value
Data Source Name	Name your datasource (for example, `pbi-idcdomain`)
Region	The AWS Region where your Amazon SageMaker domain is provisioned (for example, `us-east-1`)
Catalog	`AwsDataCatalog`
Database	`default`
Workgroup	Your Athena workgroup name (for example, `workgroup-abcdefghij-klmexample`)

In the Authentication Options, configure the following values:

Field	Value
Authentication Type	`SageMakerBrowserIdc`
SSO Start URL	IAM Identity Center entry point (for example, `https://identitycenter.amazonaws.com/ssoins-0example`)
SSO Region	Region of IAM Identity Center (for example, `us-east-1`)
SageMaker Domain ID	`dzd-123456example`
SageMaker Project ID	`abcd12example`
SageMaker Domain Region	Region of your Amazon SageMaker Unified Studio project (for example, `us-east-1`)

Choose OK, then Test to verify the connection. Choose Allow Access when prompted by the browser.

The following screenshot shows the consent prompt.

Browser consent prompt requesting access approval during authentication

Figure 5: Browser consent prompt

The following screenshot shows the successful connection test.

ODBC DSN configuration showing a successful connection test with SageMakerBrowserIdc

Figure 6: Successful connection test in the ODBC DSN configuration with SageMakerBrowserIdc authentication

Connect Power BI Desktop to your data

With the DSN configured, you can connect Power BI Desktop to your data catalog and load the generators dataset.

Open Power BI Desktop.
Open the Get Data menu and select More.
Search for and select Amazon Athena and choose Connect.
For Data Source Name (DSN), enter pbi-idcdomain.
Select DirectQuery.
Choose OK.
Choose Use Data Source Configuration and then Connect.
In the AwsDataCatalog folder, navigate to your database.
Select the core_eia860__scd_generators table.
Choose Load.

The following screenshot shows Power BI Desktop successfully connected to the AWS data catalog.

Figure 7: Power BI Desktop connected to the data catalog with the generators table loaded using SageMakerBrowserIdc authentication

Create your dashboard and publish it

You can create a dashboard to visualize U.S. power generation data. To create a visualization, complete the following steps:

In the Visualizations pane, choose the Stacked bar chart.
Assign the Y-Axis: Drag technology_description to the Y-Axis.
Assign the X-Axis (Values): Drag capacity_mw to the X-Axis (automatically summed).
Assign the Legend (Stack): Drag operational_status to the Legend field.
Choose Publish.
Give your report a name (for example, generation-idcdomain) and choose Save.
Sign in and choose a destination workspace.

Power BI Desktop stacked bar chart of generation capacity by technology and operational status

Figure 8: Power BI Desktop report using the EIA-860 generators dataset

After publishing, the report structure is available on Power BI Service.

Method 2: DSN-less connection (Power BI ODBC connector)

In this method, you use the Power BI ODBC connector with a connection string (no DSN required). This method supports Import mode only and SageMakerIam authentication. Because the gateway cannot perform browser authentication, both Desktop and gateway must use SageMakerIam. If your workflow requires SageMakerBrowserIdc, use Method 1.

If your machine already has AWS credentials through another method in the default credential provider chain, skip the following setup.

Administrator setup

Create a custom permission set named SageMakerDataAnalyst in IAM Identity Center with the following inline policy. For detailed steps, see Create a permission set in the AWS IAM Identity Center User Guide.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Assign your user to this permission set for the AWS account containing your SageMaker Unified Studio domain. Then configure your AWS Command Line Interface (AWS CLI) SSO profile by running aws configure sso. For the full CLI configuration walkthrough with detailed steps, see Part 2. After your profile is configured, run aws sso login to authenticate.

Add the IAM identity as a member of SageMaker Unified Studio project

The IAM identity providing credentials needs both domain-level and project-level access to query data through Athena.

Add AWSReservedSSO_SageMakerDataAnalyst_1234example as a domain IAM user: see Managing users in the Amazon SageMaker Unified Studio Admin Guide. Choose Current account.

SageMaker Unified Studio domain users list including the IAM identity

Figure 9: List of users of your SageMaker Unified Studio domain including the IAM identity

Add AWSReservedSSO_SageMakerDataAnalyst_1234example as a project member: see Add project members in the Amazon SageMaker Unified Studio User Guide.

SageMaker Unified Studio project members list including the IAM identity

Figure 10: Members of a SageMaker Unified Studio project including the IAM identity

Gather configuration values

Gather the following connection values from your Amazon SageMaker Unified Studio project:

Open your Amazon SageMaker Unified Studio Project.
On the navigation pane, choose Overview.
Select JDBC and ODBC details.
Select the Using IAM auth toggle.
Copy the ODBC connection string.

SageMaker Unified Studio project overview showing the ODBC connection string for IAM auth

Figure 11: ODBC connection string on the SageMaker Unified Studio project overview

Connect Power BI Desktop to your data and publish

With the configuration parameters of your project, you can connect Power BI Desktop to your data catalog and load the generators dataset.

Open Power BI Desktop.
Open the Get Data menu and select More.
Search for and select ODBC and choose Connect.
For Data Source Name (DSN), select (None).
Expand Advanced Options.
In the Connection string field, enter your connection string. For example, Driver={Amazon Athena ODBC (x64)};AwsRegion=us-east-1;Catalog=AwsDataCatalog;Schema=default;Workgroup=workgroup-abcdefghij-klmexample;SageMakerDomainId= dzd-123456example;SageMakerProjectId= abcd12example;SageMakerDomainRegion=us-east-1;AuthenticationType=SageMakerIam;
Choose OK.
Choose Default or Custom and then Connect.
In the AwsDataCatalog folder, navigate to your database.
Select the core_eia860__scd_generators table.
Choose Load.

When publishing, name your report generation-idcdomain-dsnless.

Configure the on-premises data gateway and view your report on Power BI Service

After creating your reports in Power BI Desktop, configure the on-premises data gateway to view your report on Power BI Service.

You can configure the gateway using either a DSN or a DSN-less connection string, matching the method you used in Power BI Desktop.

Create and attach an IAM role to the Power BI Gateway EC2 instance

Create an IAM role for the EC2 instance that will host your Power BI gateway. Name the role pbi-gateway-role (or a name of your choice). The role must use EC2 as the trusted entity and include the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Attach this role to your Power BI Gateway EC2 instance. For detailed steps on creating and attaching an IAM role to an EC2 instance, refer to IAM roles for Amazon EC2 in the Amazon EC2 User Guide.

Add the Power BI Gateway IAM role as a member of SageMaker Unified Studio project

The gateway IAM role needs project-level access to query data through Athena.

Add the IAM pbi-gateway-role role as a domain IAM user: see Managing users in the Amazon SageMaker Unified Studio Admin Guide. Choose Current account (or Associated account if your gateway is deployed in a different account).

The following screenshot, from the Amazon SageMaker page of the AWS Management Console, shows the list of users of your Amazon SageMaker Unified Studio domain, including the IAM gateway role.

SageMaker Unified Studio domain users list including the Power BI gateway IAM role

Figure 12: List of users of your SageMaker Unified Studio domain including the IAM gateway role

Add the IAM pbi-gateway-role role as a project member: see Add project members in the Amazon SageMaker Unified Studio User Guide.

The following screenshot shows the Amazon SageMaker Unified Studio project user management page listing the project members.

SageMaker Unified Studio project members list including the Power BI gateway IAM role

Figure 13: Members of a SageMaker Unified Studio project including the IAM gateway role

Configure the data source on Power BI Gateway

How you configure the data source depends on the method you used in Power BI Desktop.

Method 1 (DSN-based)

Configure a System DSN on the gateway EC2 instance following the same ODBC DSN steps described in Method 1. When configuring, make sure that:

You use the System DSN tab (not User DSN) because the gateway runs as a Windows service under a separate account.
The authentication type is set to SageMakerIam regardless of what you used on Desktop.
The DSN name matches exactly the one configured on Power BI Desktop (for example, pbi-idcdomain)

Method 2 (DSN-less)

No configuration is needed on the gateway machine itself. You configure the data source directly in Power BI Service.

Configure the data source and view your report on Power BI Service

To view your report, complete the following steps:

Open the workspace where you saved your report.
Search the Semantic Model which has the same name as your report (for example, generation-idcdomain) and choose the More options icon (three dots).
Choose Settings.
Expand Gateway and Cloud Connection.
Choose View Datasources (play icon) on your gateway.
Choose Manually add to gateway.
Add a connection name (for example, pbi-idcdomain).

The next step depends on the method that you chose:

Method 1 (DSN-based)

Add the DSN (for example, pbi-idcdomain) that matches exactly the one configured on Power BI Desktop.

Method 2 (DSN-less)

In the Connection string field, enter the connection string that matches exactly the one used in Power BI Desktop.

Next, continue with the configuration:

Select Anonymous as Authentication Method.
Choose Create.
Expand again Gateway and Cloud Connection.
For Maps to, choose the connection that you created (for example, pbi-idcdomain).
Choose Apply.
Return to the workspace where you saved your report.
On the Content section, choose your report (for example, generation-idcdomain).

The following screenshot shows a Power BI report on Power BI Service.

Published Power BI report rendering on Power BI Service

Figure 14: Power BI report on Power BI Service

You can now see your report online with the data from your Amazon SageMaker Unified Studio project.

Clean up

To avoid additional charges after testing, delete the Amazon SageMaker Unified Studio domain and EC2 instances. Refer to Delete domains and Terminate Instances for instructions.

Conclusion

In this post, you connected Microsoft Power BI to Amazon SageMaker Unified Studio using an IDC-based domain with both DSN-based and DSN-less methods. This provides a direct connection, with no third-party licensing, that maintains data governance. In Part 2, we cover IAM-based domains.

You can automate many steps of this process. For information about automating DSN creation on the Power BI Gateway or Service, refer to How ENGIE automates the deployment of Amazon Athena data sources on Microsoft Power BI. If you don’t want users adding the gateway IAM role directly, you can create a custom blueprint as a self-service tool for gateway role addition. The blueprint uses a ProjectMembership resource with a configurable parameter that project owners can activate at project creation, automatically adding the gateway role as a project contributor.

For additional best practices, refer to the Using Microsoft Power BI with the AWS Cloud Whitepaper. To learn more, visit Amazon SageMaker Unified Studio and Amazon Athena.

About the authors

Connect Amazon SageMaker Unified Studio to Microsoft Power BI – Part 2: IAM-based domains

2026-09-15 Ramesh H Singh

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/connect-amazon-sagemaker-unified-studio-to-microsoft-power-bi-part-2-iam-based-domains/

In Part 1 of this series, we connected Microsoft Power BI to Amazon SageMaker Unified Studio using an IAM Identity Center (IDC)-based domain. The Amazon Athena ODBC driver (version 2.2.0 and later) supports Amazon SageMaker Unified Studio authentication natively, removing the third-party ODBC-JDBC bridge previously required. We walked through both the DSN-based connection and the DSN-less connection, from Power BI Desktop through the on-premises data gateway to Power BI Service, where report viewers access published dashboards.

In this post, you create the same direct connection using an AWS Identity and Access Management (IAM)-based domain. The walkthrough covers the same two connection methods. The differences are the Amazon SageMaker Unified Studio console navigation paths, the configuration values, and an additional administrator setup that provides AWS credentials through AWS IAM Identity Center. This is Part 2 of a two-part series. For a detailed comparison of the two connection methods, see Part 1.

Solution overview

The architecture is the same as the previous post (see the architecture diagram and walkthrough scenario in Part 1). Power BI Desktop connects to Amazon Athena through the ODBC driver and the Amazon SageMaker Unified Studio project governs all data access. At the same time, the on-premises data gateway on an Amazon Elastic Compute Cloud (Amazon EC2) instance bridges the connection to Power BI Service so report viewers can access published dashboards.

The difference is in authentication: An IAM-based domain uses SageMakerIam authentication for both connection methods. The driver retrieves credentials from the AWS default credential provider chain. For this walkthrough, AWS IAM Identity Center provides those credentials through a custom permission set. Power BI Desktop can run on-premises or on an EC2 instance in the AWS Cloud. The gateway EC2 instance authenticates using its attached IAM role.

Prerequisites

Complete the prerequisites from Part 1. Additionally, you need:

AWS Command Line Interface (AWS CLI) – The latest version of the AWS CLI installed on your Windows machine. In this post series, the ODBC driver uses the AWS IAM Identity Center profile configured through the CLI for authentication.
Amazon SageMaker Unified Studio – An Amazon SageMaker Unified Studio IAM-based domain with AWS IAM Identity Center single sign-on (SSO) enabled.

The following screenshot shows the Amazon SageMaker Unified Studio (IAM-based domain) project Query Editor interface. It runs a preview query on the EIA-860 generators dataset.

SageMaker Unified Studio Query Editor previewing the EIA-860 generators dataset in an IAM-based domain

Figure 1: SageMaker Unified Studio (IAM-based domain) project with the EIA-860 generators dataset available in the data catalog

Administrator setup

This section configures AWS IAM Identity Center to provide credentials for the SageMakerIam authentication mode. It applies to Method 1 (IAM-based domain) and Method 2 (both domain types). If your machine already has AWS credentials available through another method in the default credential provider chain, you can skip this section and proceed directly to the method of your choice. For the full list of credential sources, refer to Credential providers in the AWS SDKs and Tools Reference Guide.

Create a permission set in IAM Identity Center

Create a custom permission set named SageMakerDataAnalyst in IAM Identity Center with the following inline policy. For detailed steps, see Create a permission set in the AWS IAM Identity Center User Guide.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

The "Resource": "*" is required because these API actions do not support resource-level permissions. For more information, see Actions, resources, and condition keys for Amazon DataZone.

This doesn’t grant broad access to your data. These are read-only metadata actions that allow the ODBC driver to discover connection details and retrieve temporary Athena credentials. The actual data access is governed by Amazon SageMaker Unified Studio project membership: Users can only query data within projects where they have been explicitly added as members. The Amazon SageMaker Unified Studio project IAM role provides Athena and Amazon S3 permissions separately.

Assign users to the permission set

To assign users or groups to the target AWS account, complete the following steps:

In the IAM Identity Center console, choose AWS accounts.
Select the target account where your Amazon SageMaker Unified Studio IAM-based domain is deployed.
Choose Assign users or groups.
Select the SSO users or groups that need access.
Select the SageMakerDataAnalyst permission set.
Choose Submit.

Configure AWS IAM Identity Center profile

To configure the AWS IAM Identity Center profile, run the following command in your terminal on Windows:

aws configure sso

When prompted, enter the following values:

Prompt	Value
SSO session name	For example, `smus`
SSO start URL	The IDC issuer URL. For example, `https://identitycenter.amazonaws.com/ssoins-0example`
SSO region	The SSO Region. For example, `us-east-1`
SSO registration scopes	`sso:account:access`

A browser window opens for authentication. After authentication, select your account and the SageMakerDataAnalyst role.

The following screenshots show the consent window and the successful authentication message.

Browser consent prompt requesting access approval during AWS CLI SSO authentication

Figure 2: Browser consent prompt

Browser page confirming successful AWS CLI SSO authentication

Figure 3: Browser authentication successful message

When prompted, enter the following values:

Prompt	Value
Default client Region	None
CLI default output format	None
Profile Name	Change value by default

The resulting ~/.aws/config file should look like the following:

[default]
sso_session = smus
sso_account_id = 1234example
sso_role_name = SageMakerDataAnalyst

[sso-session smus]
sso_start_url = https://identitycenter.amazonaws.com/ssoins-0example
sso_region = us-east-1
sso_registration_scopes = sso:account:access

Verify authentication and daily use

To verify that your SSO profile is working correctly, run the following command:

aws sts get-caller-identity

You should receive a response like the following:

{
    "UserId": "AROARHJJNFBQD6EXAMPLE:[email protected]",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_SageMakerDataAnalyst_1234example/[email protected]"
}

For daily use, no passwords or EC2 instance roles are required. When your SSO session expires, run the following command to quickly refresh it:

aws sso login

Add your IAM identity as a member of your Amazon SageMaker Unified Studio project

The IAM identity providing credentials to the ODBC driver needs project-level access to query data through Athena. If you completed the administrator setup, this is the SSO role associated with your permission set (for example, AWSReservedSSO_SageMakerDataAnalyst_1234example). If you’re using another credential source, add the IAM role or user that provides those credentials. For detailed steps, see Managing users for IAM-based domains in the Amazon SageMaker Unified Studio Administrator Guide.

The following screenshot shows the Amazon SageMaker Unified Studio domain management page, which lists the members in a project.

SageMaker Unified Studio project members list

Figure 4: List of members of your SageMaker Unified Studio project

Gather the information to authenticate

To get the parameters that you need to authenticate, complete these steps:

Open your Amazon SageMaker Unified Studio Project.
Open Domain Management.
Choose Users.
Choose View SSO connection.
Copy the end of the Instance ARN, so we can build the Instance URL like https://identitycenter.amazonaws.com/ssoins-0example

The following screenshot shows the Amazon SageMaker Unified Studio domain management page with SSO connection details.

SageMaker Unified Studio domain SSO connection details showing the IAM Identity Center instance ARN

Figure 5: AWS IAM Identity Center information

Choose the user icon and copy the Region as shown in the following screenshot.

SageMaker Unified Studio user menu showing the Region

Figure 6: User icon with the Region information

Method 1: DSN-based connection (Athena Power BI connector)

In this method, you configure an ODBC Data Source Name (DSN) and use the Amazon Athena connector in Power BI. This method uses SageMakerIam authentication mode and supports both DirectQuery and Import mode.

This section covers IAM-based domains. For IDC-based domains, see Part 1.

Gather configuration values to configure your Amazon Athena ODBC DSN

Before configuring the ODBC DSN, gather the following connection values from your Amazon SageMaker Unified Studio project:

Open your Amazon SageMaker Unified Studio Project.
Top right, select the three dots.
Choose Project details.
Select JDBC and ODBC details.
Copy the following values: domain ID, Amazon SageMaker project ID, AWS Region, and Athena workgroup.

The following screenshot shows the Amazon SageMaker Unified Studio project overview page, which provides the project details to copy.

SageMaker Unified Studio project details showing domain ID, project ID, Region, and Athena workgroup

Figure 7: Project details with SageMaker domain ID, SageMaker project ID, Region, and Athena workgroup

Configure the ODBC DSN

Create a System DSN using the Amazon Athena ODBC driver. For the general DSN creation steps, see Configuring a data source name on Windows in the Amazon Athena User Guide. Enter the following values:

Field	Value
Data Source Name	Name your datasource (for example, `pbi-iamdomain`)
Region	The AWS Region where your Amazon SageMaker domain is provisioned (for example, `us-east-1`)
Catalog	`AwsDataCatalog`
Database	`default`
Workgroup	Your Athena workgroup name (for example, `workgroup-abcdefghij-klmexample`)

In the Authentication Options, configure the following values:

Field	Value
Authentication Type	`SageMakerIam`
SageMaker Domain ID	`dzd-123456example`
SageMaker Project ID	`abcd12example`
SageMaker Region	Region of your SageMaker Unified Studio project (for example, `us-east-1`)

Choose OK, then Test to verify the connection. Choose Allow Access when prompted by the browser.

The following screenshot shows the successful connection test.

ODBC DSN configuration showing a successful connection test with SageMakerIam

Figure 8: Successful connection test in the ODBC DSN configuration with SageMakerIam authentication

Connect Power BI Desktop to your data

With the DSN configured, you can connect Power BI Desktop to your data catalog and load the generators dataset.

Open Microsoft Power BI Desktop.
Open the Get Data menu and select More.
Search for and select Amazon Athena and choose Connect.
For Data Source Name (DSN), enter pbi-iamdomain.
Select DirectQuery.
Choose OK.
Choose Use Data Source Configuration and then Connect.
In the AwsDataCatalog folder, navigate to your database.
Select the core_eia860__scd_generators table.
Choose Load.

The following screenshot shows Power BI Desktop successfully connected to the data catalog.

Figure 9: Power BI Desktop connected to the data catalog with the generators table loaded using SageMakerIam authentication

Create your dashboard and publish it

You can create a dashboard to visualize U.S. power generation data. To create a visualization, complete the following steps:

In the Visualizations pane, choose the Stacked bar chart.
Assign the Y-Axis: Drag technology_description to the Y-Axis.
Assign the X-Axis (Values): Drag capacity_mw to the X-Axis (automatically summed).
Assign the Legend (Stack): Drag operational_status to the Legend field.
Choose Publish.
Give your report a name (for example, generation-iamdomain) and choose Save.
Sign in and choose a destination workspace.

The following screenshot shows the Power BI dashboard with U.S. power generation data.

Power BI stacked bar chart of U.S. generation capacity by technology and operational status

Figure 10: Power BI dashboard with U.S. power generation data

After you publish, the report structure becomes available on Microsoft Power BI Service.

Method 2: DSN-less connection (Power BI ODBC connector)

In this method, you use the Power BI ODBC connector with a connection string (no DSN required). This method supports Import mode only and SageMakerIam authentication. Because the gateway can’t perform browser authentication and connection strings need to match, both Desktop and gateway must use SageMakerIam.

This section covers IAM-based domains. For IDC-based domains, see Part 1.

Gather configuration values to configure your DSN-less connection

Gather the following connection values from your Amazon SageMaker Unified Studio project:

Open your Amazon SageMaker Unified Studio Project.
Top right, select the three dots.
Choose Project details.
Select JDBC and ODBC details.
Copy the ODBC connection string.

The following screenshot shows the Amazon SageMaker Unified Studio project overview page with the ODBC connection string to copy.

SageMaker Unified Studio project overview showing the ODBC connection string

Figure 11: Project details with ODBC connection string

Connect Power BI Desktop to your data and publish

With the configuration parameters of your project, you can connect Power BI Desktop to your data catalog and load the generators dataset.

Open Power BI Desktop.
Open the Get Data menu and select More.
Search for and select ODBC and choose Connect.
For Data Source Name (DSN), select (None).
Expand Advanced Options.
In the Connection string field, enter your connection string. For example, Driver={Amazon Athena ODBC (x64)};AwsRegion=us-east-1;Catalog=AwsDataCatalog;Schema=default;Workgroup=workgroup-abcdefghij-klmexample;SageMakerDomainId= dzd-123456example;SageMakerProjectId= abcd12example;SageMakerDomainRegion=us-east-1;AuthenticationType=SageMakerIam;
Choose OK.
Choose Default or Custom and then Connect.
In the AwsDataCatalog folder, navigate to your database.
Select the core_eia860__scd_generators table.
Choose Load.

When publishing, name your report generation-iamdomain-dsnless.

Configure the gateway and view your report on Power BI Service

After creating your reports in Power BI Desktop, configure the on-premises data gateway to view your report on Power BI Service.

You can configure the gateway using either a DSN or a DSN-less connection string, matching the method you used in Power BI Desktop.

Create and attach an IAM role to the Power BI Gateway EC2 instance

Create an IAM role for the EC2 instance that will host your Power BI gateway. Name the role pbi-gateway-role (or a name of your choice). The role must use EC2 as the trusted entity and include the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SageMakerAccess",
            "Effect": "Allow",
            "Action": [
                "datazone:GetConnection",
                "datazone:ListConnections",
                "datazone:GetDomain",
                "datazone:GetProject"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSForDriver",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Attach this role to your Power BI Gateway EC2 instance. For detailed steps on creating and attaching an IAM role to an EC2 instance, refer to IAM roles for Amazon EC2 in the Amazon EC2 User Guide.

Add the Power BI Gateway IAM role as a member of SageMaker Unified Studio project

The gateway IAM role needs project-level access to query data through Athena. The steps to add the role differ depending on your domain type.

IAM-based domain

Open your Amazon SageMaker Unified Studio Project.
Open Domain Management.
Choose your Project Name.
Choose Members.
Choose Add members.
Select the IAM role of your Power BI gateway (for example, pbi-gateway-role).
Choose Add.

The following screenshot shows the Amazon SageMaker Unified Studio project domain management page with options to add members to a project.

Figure 12: List of members of a SageMaker Unified Studio project with the IAM gateway role

Configure the data source on Power BI Gateway

How you configure the data source depends on the method you used in Power BI Desktop.

Method 1 (DSN-based)

Configure a System DSN on the gateway EC2 instance following the same ODBC DSN steps described in Method 1. When configuring, make sure that:

You use the System DSN tab (not User DSN) because the gateway runs as a Windows service under a separate account.
The authentication type is set to SageMakerIam.
The DSN name matches exactly the one configured on Power BI Desktop (for example, pbi-iamdomain).

Method 2 (DSN-less)

No configuration is needed on the gateway machine itself. You configure the data source directly in Power BI Service.

Configure the data source and view your report on Power BI Service

To view your report, complete the following steps:

Open the workspace where you saved your report.
Search the Semantic Model which has the same name as your report (for example, generation-iamdomain) and choose the More options icon (three dots).
Choose Settings.
Expand Gateway and Cloud Connection.
Choose View Datasources (play icon) on your gateway.
Choose Manually add to gateway.
Add a connection name (for example, pbi-iamdomain).

The next step depends on the method that you chose:

Method 1 (DSN-based)

Add the DSN (for example, pbi-iamdomain) that matches exactly the one configured on Power BI Desktop.

Method 2 (DSN-less)

In the Connection string field, enter the connection string that matches exactly the one used in Power BI Desktop.

Next, continue with the configuration:

Select Anonymous as Authentication Method.
Choose Create.
Expand again Gateway and Cloud Connection.
For Maps to, choose the connection that you created (for example, pbi-iamdomain).
Choose Apply.
Return to the workspace where you saved your report.
On the Content section, choose your report (for example, generation-iamdomain).

The following screenshot shows a report on Power BI Service.

Figure 13: Power BI report on Power BI Service

You can now see your report online with the data from your Amazon SageMaker Unified Studio project.

Clean up

To avoid additional charges after testing, delete the Amazon SageMaker Unified Studio domain and EC2 instances. Refer to Delete domains and Terminate Instances for instructions.

Conclusion

In this two-part series, you connected Power BI to Amazon SageMaker Unified Studio through Amazon Athena. Part 1 covered IDC-based domains. This post covered IAM-based domains using SageMakerIam authentication. This provides a direct connection path, with no third-party licensing, while maintaining data governance and security.

For additional best practices, refer to the Using Microsoft Power BI with the AWS Cloud Whitepaper. To learn more, visit Amazon SageMaker Unified Studio and Amazon Athena.

About the authors

Customize Amazon API Gateway destinations for execution logs

2026-09-10 Giedrius Praspaliauskas

Post Syndicated from Giedrius Praspaliauskas original https://aws.amazon.com/blogs/compute/customize-amazon-api-gateway-destinations-for-execution-logs/

Amazon API Gateway execution logs help you trace request processing step by step through your REST API stages. They capture authorization results, integration latency, mapping template output, and error details that are otherwise invisible at the API surface. When a production request fails in a way the access log cannot explain, the execution log is usually where you find the explanation.

Until now, execution logs had two constraints. Every log event was truncated at 1 KB, so a request carrying a moderately sized JSON body would exceed that limit and the remainder was dropped. Logs could only go to the auto-managed log group that API Gateway creates for you (API-Gateway-Execution-Logs_{rest-api-id}/{stage_name}).

With Amazon CloudWatch Logs delivery for REST API execution logs, you can now route execution logs to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose. Log events can be up to 1 MB per entry, and you benefit from vended logs pricing.

In this post, you learn how CloudWatch Logs delivery works with API Gateway execution logs, how to configure it, and what patterns work best for common observability scenarios.

Understanding API Gateway execution logs

API Gateway produces two categories of logs: access logs and execution logs. Access logs record a summary line per request, similar to an HTTP server access log. You configure the format and destination yourself.

Execution logs are different. They capture the internal processing of each request as it moves through the API Gateway pipeline: authorizer evaluation, request validation, integration dispatch, response mapping, and error handling. These logs exist so you can answer questions such as “why did my authorizer reject this token?” or “what did the mapping template produce before it reached my backend integration?”

API Gateway manages execution log creation automatically. When you set loggingLevel to INFO or ERROR in your stage’s method settings, the service writes execution log events to a CloudWatch Logs log group it manages on your behalf. You do not choose the log group name or configure retention directly on it.

The auto-managed model works for many customers but may create friction for teams with specific observability requirements. Compliance frameworks that require logs in S3 with a particular prefix structure need an extra subscription filter and delivery mechanism. Sending execution logs into a security information and event management (SIEM) tool through a Firehose stream requires a forwarding layer.

Configurable log delivery with CloudWatch Logs

CloudWatch Logs delivery separates log routing from log content. Two concepts control the behavior:

DeliverySource is scoped to your API Gateway stage ARN. It defines where logs go. You create a delivery source, then attach one or more delivery destinations (CloudWatch Logs log group, S3 bucket, or Firehose stream).

MethodSettings controls what gets logged. The loggingLevel setting (INFO, ERROR, or OFF) and dataTraceEnabled flag still determine which log events API Gateway produces. These settings work the same way regardless of whether you use the auto-managed log group or CloudWatch Logs delivery.

When you create a delivery using the CloudWatch Logs APIs, CloudWatch Logs activates your log delivery on your API Gateway stage. When you delete the delivery, CloudWatch Logs disables it accordingly. You do not need to flip any flags on the API Gateway side, and the execution logs automatically resume flowing to the auto-managed log group.

Your existing method settings keep their meaning. The loggingLevel and dataTraceEnabled values continue to control log content. If loggingLevel is already INFO or ERROR, creating a delivery redirects those logs to your chosen destination with no further configuration.

The following diagram shows how the pieces fit together.

Diagram showing one API Gateway stage delivery source fanning out to CloudWatch Logs, Amazon S3, and Firehose destinations.

Figure 1 — A single delivery source scoped to an API Gateway stage feeds one or more deliveries, each of which writes to a delivery destination backed by CloudWatch Logs, Amazon S3, or Amazon Data Firehose

The following table summarizes what changes when log delivery is active.

Aspect	Standard execution logging	Log delivery
Destination	Auto-managed CloudWatch Logs log group	CloudWatch Logs, Amazon S3, or Firehose
Multi-destination	No	Yes
Pricing	Standard CloudWatch Logs ingestion	Vended logs pricing
Log event size	Truncated at 1 KB	Up to 1 MB
Setup	Set `loggingLevel` in MethodSettings	Create delivery through CloudWatch Logs APIs
Teardown	Set `loggingLevel` to `OFF`	Delete delivery

What stays the same

Only execution log routing changes. Access logs continue to flow through accessLogSettings to whatever log group you configure, and unrelated stage features such as AWS X-Ray tracing, detailed CloudWatch metrics, throttling, and caching behave exactly as they did before.

Configuration and integration options

Before you create a delivery, confirm the following requirements:

The API Gateway REST API is deployed to a stage.
loggingLevel is set to INFO or ERROR in MethodSettings.
The account-level CloudWatch Logs IAM role is configured. For setup steps, see Set up CloudWatch logging for REST APIs in API Gateway.
For cross-account delivery, the destination has an appropriate resource policy attached through PutDeliveryDestinationPolicy.

Sending logs to a custom CloudWatch Logs log group

The most common starting point is redirecting execution logs to a log group you own. You get direct control over retention policies, metric filters, and subscription filters. The following steps use the AWS Command Line Interface (AWS CLI) with the fictitious REST API ID abc123, stage prod, Region us-east-1, and account 111122223333.

Create a delivery source referencing your stage ARN. The log type for REST API execution logs is EXECUTION_LOGS:

aws logs put-delivery-source \
    --name my-apigw-execution-logs \
    --resource-arn arn:aws:apigateway:us-east-1:111122223333:/restapis/abc123/stages/prod \
    --log-type EXECUTION_LOGS

Create a delivery destination pointing to your custom (existing) log group, then create the delivery that connects them:

aws logs put-delivery-destination \
    --name my-execution-log-destination \
    --delivery-destination-configuration \
        destinationResourceArn=arn:aws:logs:us-east-1:111122223333:log-group:/my-api/execution-logs

aws logs create-delivery \
    --delivery-source-name my-apigw-execution-logs \
    --delivery-destination-arn arn:aws:logs:us-east-1:111122223333:delivery-destination:my-execution-log-destination

Verify that the delivery is active by listing deliveries for the source:
```
aws logs describe-deliveries
```

The response includes the delivery ID, source, and destination ARN after delivery is established. Execution logs flow to /my-api/execution-logs instead of the auto-managed group.

Note: Log delivery adds structured fields (resource_arn, event_timestamp, api_id, stage, resource_path, http_method, and payload) to each event, so a new delivery emits more than your previous logs. To keep the traditional execution log format with nothing extra, set output format and record fields while creating delivery destination and creating delivery:

aws logs put-delivery-destination \
    --output-format "plain" ...

aws logs create-delivery \
    --record-fields "payload" \
    --field-delimiter "" ...

Routing logs to Amazon S3

S3 works well for long-term retention at lower cost, or for feeding logs into analytics tools such as Amazon Athena. The bucket must be in the same region as your API. Create a delivery destination pointing to your bucket:

aws logs put-delivery-destination \
    --name s3-archive-destination \
    --delivery-destination-configuration \
        destinationResourceArn=arn:aws:s3:::amzn-s3-demo-apigw-logs

Then create a delivery using the same source name. CloudWatch Logs delivers the events to your bucket, where you can query them with Athena or catalog them with AWS Glue.

Streaming to Amazon Data Firehose

For real-time analytics pipelines or third-party SIEM integration, Firehose delivery sends execution log events directly to your stream. The setup is identical: create a delivery destination with your Firehose stream ARN, then create a delivery. With direct Firehose delivery, you no longer need to maintain CloudWatch Logs subscription filters and AWS Lambda forwarders to route execution logs to external analytics systems.

Multi-destination delivery and per-destination shaping

A single delivery source supports multiple destinations. You can route the same execution logs to CloudWatch Logs for real-time alerting, S3 for long-term compliance retention, and Firehose for your SIEM, all from one stage. Create additional deliveries using the same delivery source with different destination ARNs.

Each destination receives identical log events. To shape what reaches each destination, apply a CloudWatch Logs subscription filter on the CloudWatch Logs destination. For example, you can forward only ERROR-level events to a Lambda function that pushes alerts to a SIEM, while the same delivery source writes the full event stream to S3 for compliance.

Management console experience

You can also add a log delivery destination in the management console after you enable logging for the stage.

API Gateway console showing the option to add a log delivery destination after logging is enabled for the stage.

You can specify multiple destinations, both in the current or in a different account:

API Gateway console showing multiple delivery destinations configured, including cross-account options.

Keeping existing monitoring intact

If you have dashboards or alarms on the auto-managed log group, use that same log group as one of your delivery destinations. Your existing monitoring keeps working, and you gain the ability to send logs to additional destinations such as S3 or Firehose in parallel.

Best practices

Update dashboards and alarms before enabling log delivery. When you activate log delivery, the auto-managed log group stops receiving logs. Any CloudWatch alarms, dashboards, or Contributor Insights rules pointing to API-Gateway-Execution-Logs_{rest-api-id}/{stage_name} stop working. Migrate these references to your new log group before creating the delivery.

Keep loggingLevel at INFO or ERROR. Log delivery controls routing, not content. If loggingLevel is OFF, no execution log events are produced regardless of whether a delivery exists. Verify your method settings before troubleshooting missing logs.

Treat the 1 MB log event capacity as a security decision, not only a debugging convenience. With dataTraceEnabled set to true, execution logs include complete request and response payloads up to 1 MB. Those payloads might contain personally identifiable information (PII) or other sensitive data. Confirm your log destinations have appropriate access controls, encryption, and retention policies. Mask or filter sensitive fields in mapping templates upstream of logging and enable data tracing selectively per method or only in non-production stages.

Start with a single destination, then expand. Validate that your log group or bucket receives events correctly before adding Firehose or additional destinations.

Log delivery is best-effort. In rare cases, some log events might not be delivered. For audit-critical workloads, build retention and reconciliation that account for occasional missing events rather than treating execution logs as the system of record.

Cleaning up

To avoid ongoing charges from the resources you created while following this post, delete the delivery and then remove the destinations and any example S3 bucket or Data Firehose delivery stream you no longer need. Deleting the delivery returns the stage to standard auto-managed logging.

aws logs delete-delivery --id <delivery-id>

When the delivery is deleted, CloudWatch Logs disables log delivery on the API Gateway stage automatically. The delivery source and delivery destination remain as independent objects. Delete them with delete-delivery-source and delete-delivery-destination if you do not plan to reuse them.

Conclusion

CloudWatch Logs delivery for API Gateway REST API execution logs helps address the 1 KB event truncation and single managed destination constraints. You can now route full execution logs to CloudWatch Logs, Amazon S3, or Amazon Data Firehose, use multiple destinations from a single stage, and pay vended logs pricing.

The feature works alongside existing method settings. No changes to your current logging configuration are required beyond creating the delivery itself.

To get started, refer to Route execution logs with Amazon CloudWatch Logs delivery in the API Gateway documentation. For more about CloudWatch Logs delivery configuration, see Enable logging from AWS services. For pricing details, review the Amazon CloudWatch pricing page. Try it on a test stage and share your experience in the comments.

Bring your own client certificate for backend mTLS in Amazon API Gateway

2026-09-09 Biswanath Mukherjee

Post Syndicated from Biswanath Mukherjee original https://aws.amazon.com/blogs/compute/bring-your-own-client-certificate-for-backend-mtls-in-amazon-api-gateway/

Enterprises that use Amazon API Gateway in front of internal or partner backends often want to bring their own client certificate for backend mutual TLS (mTLS) authentication. During mTLS, the backend presents its own server certificate and also requests the caller to present a client certificate to validate it against a trusted certificate authority (CA). Until now, you could use only an API Gateway-generated, self-signed SSL certificate for the outbound connection, because there was no CA behind it for the backend to trust. Backends that enforce a specific corporate or partner CA reject that self-signed certificate, and the mutual TLS handshake fails. Bringing your own CA-signed certificate is necessary for scenarios such as migrating APIs off legacy gateways or meeting your internal PKI mandates that require certificates from an approved CA.

With API Gateway, you can now bring your own client certificate for backend mutual TLS (mTLS) authentication. You can either use a third-party certificate or a certificate issued by AWS Private Certificate Authority. If you’re using a third-party certificate, you must import the certificate in AWS Certificate Manager (ACM). Then you configure the ACM certificate ARN in your REST API stage. API Gateway presents that certificate during the backend mTLS handshake.

Solution overview

In this post, you build a REST API with an outbound mTLS connection using this newly launched API Gateway feature. This solution demonstrates an outbound mTLS connection between Amazon API Gateway and a backend application running on Amazon Elastic Container Service (Amazon ECS).

The following diagram shows the solution architecture.

Architecture diagram showing API Gateway presenting an ACM client certificate to a Network Load Balancer that forwards traffic to an NGINX sidecar and validator app on Amazon ECS Fargate, with certificates issued by AWS Private CA through ACM

The solution uses:

AWS Private Certificate Authority with a root-subordinate CA hierarchy to issue both the client and server certificates through AWS Certificate Manager (ACM).
Amazon API Gateway REST API stage configured with the ACM client certificate ARN (ClientCertificateId), so that the API Gateway presents the certificate during the outbound TLS handshake.
Amazon ECS on AWS Fargate running an NGINX sidecar that holds the server certificate and validates the incoming client certificate against a CA bundle (root and subordinate chain).

A request goes through the following steps:

Client application invokes the REST API exposed by API Gateway. The API Gateway stage is configured with an ACM client certificate ARN.
API Gateway opens an outbound connection to the Network Load Balancer (NLB) to begin the TLS handshake. The API Gateway presents an ACM client certificate configured at the stage level when the backend requests one.
The NLB listens for the incoming TCP request on port 443 and forwards the call to Amazon ECS Fargate. The NLB acts as a passthrough and does not terminate the TLS connection.
The NGINX sidecar container running on Amazon ECS performs the inbound mTLS handshake:

NGINX presents the backend server certificate and verifies the client certificate against a mounted CA bundle (root and subordinate chain).
After verification, NGINX forwards the request and parsed certificate details to the validator app container over local HTTP.
The validator app re-checks the certificate validity window, matches the common name against an allowlist, and returns a structured JSON response.

Note: The NGINX sidecar is not mandatory for this flow. It demonstrates separation of concerns: NGINX handles the mTLS handshake, and the validator app contains the business logic.

Prerequisites for demo

To follow along, you need the following:

Create an AWS account if you do not already have one.
Access to an AWS account through the AWS Management Console and the AWS Command Line Interface (AWS CLI). The AWS Identity and Access Management (IAM) principal you use must have permissions to make the necessary AWS service calls and manage the resources in this post. Follow the principle of least privilege.
Git installed.
AWS Serverless Application Model (AWS SAM) CLI installed.
Docker installed and running, to build and push the two container images.
Python 3.14 installed.
jq command line tools installed.

Environment setup

Run the following commands to set up the demo environment:

Create a new folder and clone the GitHub repository:

git clone https://github.com/aws-samples/sample-api-backend-mtls
cd sample-api-backend-mtls

Set the environment variables after replacing the placeholders:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION=<Your AWS Region, for example, us-east-1>
STACK_NAME=<Your stack name e.g. outbound-mtls-backend>

Build the container images

Run the following commands to create container images of NGINX sidecar container and the validator app containers:

Create two Amazon Elastic Container Registry (Amazon ECR) repositories, one for NGINX and another for validator app containers respectively:

NGINX_REPO_URI=$(aws ecr create-repository \
  --repository-name $STACK_NAME-nginx-sidecar \
  --image-tag-mutability IMMUTABLE \
  --image-scanning-configuration scanOnPush=true \
  --region "$REGION" \
  --query "repository.repositoryUri" --output text)

VALIDATOR_REPO_URI=$(aws ecr create-repository \
  --repository-name $STACK_NAME-validator-app \
  --image-tag-mutability IMMUTABLE \
  --image-scanning-configuration scanOnPush=true \
  --region "$REGION" \
  --query "repository.repositoryUri" --output text)

aws ecr get-login-password --region "$REGION" \
  | docker login --username AWS --password-stdin "${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com"

Build and push the NGINX and validator app containers:

docker build --platform linux/amd64 -t $STACK_NAME-nginx-sidecar nginx/
docker tag $STACK_NAME-nginx-sidecar:latest "${NGINX_REPO_URI}:latest"
docker push "${NGINX_REPO_URI}:latest"
docker build --platform linux/amd64 -t $STACK_NAME-validator-app validator_app/
docker tag $STACK_NAME-validator-app:latest "${VALIDATOR_REPO_URI}:latest"
docker push "${VALIDATOR_REPO_URI}:latest"

Deploy and test the solution

You first deploy the stack without the client certificate configured in the API Gateway and perform negative testing. The mTLS handshake will fail because of a missing client certificate in the request. Then you update the stack to configure client certificate in API Gateway stage and retest mTLS.

Run the following command to build and deploy the overall stack without client certificate configured at API Gateway stage:

sam build
sam deploy \
  --stack-name $STACK_NAME \
  --resolve-s3 \
  --capabilities CAPABILITY_IAM \
  --region "$REGION" \
  --parameter-overrides \
  NginxRepositoryUri="$NGINX_REPO_URI" \
  ValidatorRepositoryUri="$VALIDATOR_REPO_URI" \
  EnableOutboundMtls=false

Wait for the task to reach RUNNING and pass its target group health check:

EcsClusterName=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region "$REGION" \
  --query "Stacks[0].Outputs[?OutputKey=='EcsClusterName'].OutputValue" \
  --output text)

TargetGroupArn=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region "$REGION" \
  --query "Stacks[0].Outputs[?OutputKey=='TargetGroupArn'].OutputValue" \
  --output text)

aws ecs list-tasks --cluster "$EcsClusterName" --region "$REGION"

aws elbv2 describe-target-health \
  --target-group-arn "$TargetGroupArn" --region "$REGION"

Capture the front API invoke URL from the stack outputs:

FRONT_API_URL=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region "$REGION" \
  --query "Stacks[0].Outputs[?OutputKey=='FrontApiUrl'].OutputValue" \
  --output text)

NLB_DNS_NAME=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region "$REGION" \
  --query "Stacks[0].Outputs[?OutputKey=='NlbDnsName'].OutputValue" \
  --output text)

FRONT_CLIENT_CERT_ARN=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region "$REGION" \
  --query "Stacks[0].Outputs[?OutputKey=='FrontClientCertArn'].OutputValue" \
  --output text)

FRONT_API_ID=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region "$REGION" \
  --query "Stacks[0].Outputs[?OutputKey=='FrontApiId'].OutputValue" \
  --output text)

Wait a minute or two after the stack finishes, then invoke the front API:
```
curl -v "$FRONT_API_URL"
```
The following is the NGINX configuration for mTLS:
```
...
server {
    listen 443 ssl;
    # Server identity (issued by the private CA).
    ssl_certificate /etc/nginx/certs/server.crt;
    ssl_certificate_key /etc/nginx/certs/server.key;
    # Inbound mutual TLS: require and validate the client certificate
    # against the CA bundle (root + subordinate CA chain).
    ssl_client_certificate /etc/nginx/certs/ca_bundle.pem;
    ssl_verify_client on;
    ssl_verify_depth 2;
    ssl_protocols TLSv1.2;
...}
```
The curl command returns HTTP/2 400, with a response body containing 400 No required SSL certificate was sent. Because the API Gateway is not presenting a client certificate on the outbound handshake, the NGINX sidecar container in Amazon ECS rejects the mTLS connection. The following screenshot shows the response:

Now redeploy the solution with outbound mTLS enabled:

sam deploy \
  --stack-name $STACK_NAME \
  --resolve-s3 \
  --capabilities CAPABILITY_IAM \
  --region "$REGION" \
  --parameter-overrides \
  NginxRepositoryUri="$NGINX_REPO_URI" \
  ValidatorRepositoryUri="$VALIDATOR_REPO_URI" \
  EnableOutboundMtls=true

Wait a minute or two after the stack finishes, then invoke the front API again:
```
curl -v "$FRONT_API_URL"
```
Because the client certificate is now presented during the mTLS handshake, the handshake completes successfully, as shown in the following response snippet:

Automatic certificate renewal

When a certificate changes in ACM, API Gateway detects the update and propagates the new certificate automatically. You do not redeploy the stage, and the API experiences no downtime during rotation. Certificate propagation is eventually consistent. During an update, the backend might briefly receive either the old or the new certificate. ACM also emits certificate expiration notifications through Amazon EventBridge, which you can use to set alarms before a certificate expires.

Clean up

If you followed along only for demonstration purposes, to avoid incurring future charges, run the following commands to delete the resources created in this demo:

Clean up the S3 buckets:

NLB_LOGS_BUCKET=$(aws cloudformation describe-stacks \
  --stack-name $STACK_NAME --region $REGION \
  --query "Stacks[0].Outputs[?OutputKey=='NlbAccessLogsBucketName'].OutputValue" \
  --output text)

aws s3api list-object-versions --bucket "$NLB_LOGS_BUCKET" \
  --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' \
  --output json | \
  jq -c '.Objects[]? // empty' | \
  while read -r obj; do
    aws s3api delete-object --bucket "$NLB_LOGS_BUCKET" \
      --key "$(echo "$obj" | jq -r .Key)" \
      --version-id "$(echo "$obj" | jq -r .VersionId)" \
      --region $REGION
  done

aws s3api list-object-versions --bucket "$NLB_LOGS_BUCKET" \
  --query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' \
  --output json | \
  jq -c '.Objects[]? // empty' | \
  while read -r obj; do
    aws s3api delete-object --bucket "$NLB_LOGS_BUCKET" \
      --key "$(echo "$obj" | jq -r .Key)" \
      --version-id "$(echo "$obj" | jq -r .VersionId)" \
      --region $REGION
  done

Delete the stack:

sam delete --stack-name $STACK_NAME --region $REGION --no-prompts

Delete the ECR repository:

aws ecr delete-repository --repository-name $STACK_NAME-nginx-sidecar --force --region $REGION
aws ecr delete-repository --repository-name $STACK_NAME-validator-app --force --region $REGION

Conclusion

In this post, you configured a REST API with an outbound mTLS connection using Amazon API Gateway and an ECS Fargate backend. With this new feature launch in API Gateway, you can now bring your own client certificate for outbound mTLS handshake for your REST APIs. You can now meet your internal PKI mandates to authenticate backends that pin a specific certificate issuer.

To get started, import a certificate from your own PKI into ACM and configure your API Gateway REST API stage for outbound mTLS authentication. For more information, see Present client certificates to backend services with mutual TLS in API Gateway. If you have feedback about this post, leave it in the comments section. For technical questions, you can start a thread on AWS re:Post.

Deliver real-time data to streaming tables for Apache Iceberg with Amazon Kinesis Data Streams

2026-09-01 Nikit Pednekar

Post Syndicated from Nikit Pednekar original https://aws.amazon.com/blogs/big-data/deliver-real-time-data-to-streaming-tables-for-apache-iceberg-with-amazon-kinesis-data-streams/

Amazon Kinesis Data Streams now supports streaming tables, a fully managed capability that continuously delivers your streaming data as queryable Apache Iceberg tables on Amazon S3 Tables. Amazon S3 Tables is a capability of Amazon Simple Storage Service (Amazon S3). Streaming tables reduce data delivery costs to S3 Tables by up to 50% compared to self-managed alternatives and reduce downstream query costs by up to 30% through intelligent inline compaction that eliminates the small file problem. You need no custom applications, no self-managed compute, and no operational overhead.

Customers increasingly want to unify streaming data with Apache Iceberg for near-real-time analytics, fraud detection, personalization, and artificial intelligence and machine learning (AI/ML) feature pipelines. But integrating the two has meant operating complex custom connectors, managing format conversions, and contending with the performance impact of many small Parquet files that slow queries and increase costs. Streaming tables solve this: configure delivery in a few steps from the console or through APIs, and your data becomes queryable from Amazon Athena, Amazon Redshift, and Apache Spark within minutes. Tables are automatically registered in AWS Glue Data Catalog, making them immediately discoverable for analytics engines and AI agents.

For workloads that don’t require Iceberg table format, you can also deliver streaming data to Amazon S3 general purpose buckets. Delivery is in the source data format, ideal for archival, backup, and ML training data pipelines, with the same serverless, fully managed delivery and no infrastructure to operate.

Challenges with delivering streaming data to Apache Iceberg

Customers today face three challenges when integrating streaming data with Apache Iceberg.

Operational complexity: Connecting Kinesis Data Streams to Iceberg tables today requires deploying and maintaining custom connectors, Apache Flink jobs, or consumer applications. Teams must manage pipeline failures, handle format conversions, scale infrastructure, and monitor delivery reliability. These operational tasks consume significant engineering time and introduce ongoing risk of downtime.

Resiliency and the small file problem: Without proper coordination, simultaneous writes from multiple high-throughput shards can conflict, leading to failed commits, data freshness delays, and degraded performance. Streaming ingestion of high-volume data creates large numbers of small Parquet files in Iceberg tables, forcing a difficult trade-off between data freshness and query efficiency.

Cost: Customers typically spend up to $28/TB operating streaming extract, transform, and load (ETL) pipelines from Kinesis Data Streams using self-managed alternatives based on internal analysis. This creates a high price barrier to getting streaming data into queryable formats and makes cost unpredictable as volume grows.

How delivery to streaming tables solves these challenges

Streaming tables are a native capability built directly into Amazon Kinesis Data Streams. There is no separate service to deploy, no connector to version, and no consumer application to maintain. You enable delivery in a few steps from the console or through APIs.

Zero operational overhead: Streaming tables remove the need to build and operate custom consumer applications for data delivery. No pipeline infrastructure to provision, no scaling logic to write, no failure handling to implement. The capability automatically scales to process gigabytes per second of throughput.

Built-in resiliency: Streaming tables provide write coordination and exactly once delivery semantics across all shards in your stream, resolving concurrent writer conflicts and ensuring data integrity without manual intervention.

Intelligent compaction, no trade-offs: During ingestion, streaming tables perform inline compaction that produces query-optimized Parquet files, eliminating the small file problem while maintaining minute-level data freshness. This reduces downstream query costs by up to 30 percent compared to uncompacted delivery.

Consumption-based pricing: You pay only for data delivered: $14/TB for Iceberg delivery to S3 Tables in US East (N. Virginia) Region (us-east-1) (50% savings compared to self-managed alternatives) and $11/TB for general purpose S3 delivery (60% savings compared to self-managed alternatives). When your stream is idle, you pay nothing for delivery. Combined with Kinesis Data Streams On-Demand Advantage pricing, which eliminates per-shard charges and scales automatically, the entire path from ingestion to queryable Iceberg tables operates on a pure consumption model.

End-to-end managed streaming analytics architecture

With delivery to streaming tables, you now have a fully managed end-to-end real-time data architecture from data ingestion through storage to analytics. Your producers publish events to a Kinesis Data Stream, which continuously delivers data as optimized Iceberg read-only tables in S3 Tables. From there, you can query your streaming data using analytics engines like Amazon Athena, Amazon Redshift, Amazon EMR (Apache Spark), or Apache Flink. You can also let AI agents discover and reason over your data through Glue Data Catalog semantic search. This managed experience removes the intermediate infrastructure that customers previously assembled: separate connector clusters, compaction jobs, and custom consumers. It replaces them with a single, serverless pipeline from stream to insight.

The following diagram illustrates this end-to-end architecture.

End-to-end architecture from Kinesis Data Streams producers to Iceberg tables in S3 Tables, queryable by Athena, Redshift, EMR, and Flink

Figure 1: End-to-end managed streaming analytics architecture from ingestion to query

Getting started

To get started, sign in to the Amazon Kinesis Data Streams console, navigate to your streams, and enable delivery to streaming tables in a few steps. Specify the stream you want to deliver, configure your schema settings using AWS Glue Schema Registry, and choose your destination S3 Tables location. After you enable it, delivery to streaming tables immediately begins materializing your streaming data as queryable Iceberg tables in S3 with no further intervention required. There’s no infrastructure to provision and no minimum commitment. You pay only for data delivered.

Additionally, you can use Amazon Kinesis Data Streams APIs to programmatically set up, update, or delete delivery to streaming tables configurations for your data streams. With these APIs, teams can build agentic workflows and infrastructure-as-code patterns to manage configurations across multiple data streams at scale.

Getting started with the Kinesis Data Streams Agent Skill

The Kinesis Data Streams Agent Skill provides AI-assisted guidance for setting up streaming tables integrations for your existing or new data streams. The skill helps you configure delivery to S3 Tables (Iceberg) or S3, including schema registry setup, AWS Identity and Access Management (IAM) role configuration, and validation.

Installing as an Agent Skill

Agent Skills are discovered automatically by compatible tools through the SKILL.md file. Refer to the Agent Toolkit for AWS Skill Installation Guide to install the managing-amazon-kinesis-data-streams Agent Skill. We also recommend you install the AWS MCP Server in your developer tool of choice, which exposes tools for searching AWS documentation, blogs, and Skills dynamically at runtime. These capabilities make agents more accurate and powerful for AWS related development and operational tasks, and make skill discovery and installation more flexible. Refer to Setting up the AWS MCP Server for guidance on installing the AWS MCP Server in your environment.

For example:

aws configure agent-toolkit
aws agent-toolkit add-skill --skill-name managing-amazon-kinesis-data-streams

To verify the installation, interact with the skill in your preferred tool.

To start delivering data from your data streams to Apache Iceberg tables in real time, prompt “Create me a streaming table on my events data stream” to your agent of choice:

Agent chat showing a prompt to create a streaming table on the events data stream

Figure 2: Prompting the agent to create a streaming table

The agent dynamically loads the managing-amazon-kinesis-data-streams skill and starts by gathering the available resources in your AWS account for the streaming tables integration. After it gathers that data, it confirms the resources to use or create, and creates the integration:

Agent confirming the AWS resources to use or create for the streaming tables integration

Figure 3: The agent confirming resources before creating the integration

After creating the integration, the agent summarizes the status and can then help with any other operational tasks with your data. For example, the agent can help you set up AWS Lake Formation permissions to query the data in S3 Tables with Athena, or configure your table maintenance behavior in S3 Tables:

Agent summarizing integration status and offering Lake Formation permissions or table maintenance setup

Figure 4: The agent offering follow-up operational tasks

Conclusion

Streaming tables are available in all AWS Regions where Amazon Kinesis Data Streams is offered. Pricing is $14/TB for delivery to S3 Tables (Apache Iceberg) and $11/TB for delivery to general purpose S3 buckets. To learn more, visit the documentation and pricing pages.

About the authors

PythonOperator and BashOperator Now Available on Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless

2026-08-25 Pradeep Kumar Nalluri

Post Syndicated from Pradeep Kumar Nalluri original https://aws.amazon.com/blogs/big-data/pythonoperator-and-bashoperator-now-available-on-amazon-managed-workflows-for-apache-airflow-amazon-mwaa-serverless/

If you run Apache Airflow workflows on Amazon MWAA Serverless, you can now use PythonOperator and BashOperator to run custom code directly in the serverless runtime. Previously, Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless only supported orchestration of AWS services through operators for scheduling tasks, managing dependencies, and handling retries. It did not support running your own Python functions or shell scripts natively. If you needed custom Python logic or shell commands, you had to wrap code in AWS Lambda functions, start Amazon Elastic Container Service (Amazon ECS) tasks, or use other AWS compute services. These alternatives add complexity, cost, and latency to your orchestration pipelines.

With this launch, you can run custom Python functions and shell scripts directly within the serverless task runtime, without requiring additional infrastructure. This means you can now use PythonOperator and BashOperator many data engineering teams rely on for ETL pipelines and data quality checks – without provisioning additional compute.

In this post, we walk through how this feature works and demonstrate a practical example: building a serverless pipeline that converts CSV files to JSON format using a PythonOperator, and verifies the output using a BashOperator. By the end, you will know how to:

Package a Python module with dependencies and upload it to an Amazon Simple Storage Service (Amazon S3) bucket as a code bundle
Define a multi-task workflow using the dag-factory compatible YAML
Create and run a workflow with the AWS Command Line Interface (AWS CLI)
Verify that your pipeline produced the expected output

How it works

With MWAA Serverless, you can package your custom code, upload it to an Amazon S3 bucket, and reference it when creating a workflow. The service snapshots your code at workflow creation time and uses that snapshot for all subsequent runs of the same workflow version.

Code bundles

A code bundle is the package that contains your custom logic. You package your Python modules or shell scripts and upload them to an Amazon S3 bucket. A code bundle can be:

A single .py file or .sh bash script (uploaded to an Amazon S3 bucket)
A ZIP archive containing multiple shell scripts, Python modules and dependencies (up to 250 MB)

Execution model

When you create or update a workflow, MWAA Serverless snapshots your code bundle from an Amazon S3 bucket provided and stores it on the service side. At task execution time, the service uses this snapshot – not the object currently residing in your Amazon S3 bucket – to run your code in an isolated runtime environment.

Python and Bash tasks do not have internet access. They can reach only Amazon S3, Amazon Elastic Container Registry (Amazon ECR), and Amazon CloudWatch, which are the services the runtime requires to operate. To have internet access, configure the workflow with Amazon VPC so that it can go through the provided VPC.

Supported operators

The following table describes the two operators now available in MWAA Serverless.

Operator	Description
PythonOperator	Executes a Python callable (function) from your code bundle
BashOperator	Runs shell commands or scripts

Security

AWS Key Management Service (AWS KMS) encrypts your code bundles at rest. IAM policies control who can create, update, and trigger the workflows. The execution role scopes what AWS resources your code can access at runtime.

Prerequisites

Before getting started, verify that you have the following resources and tools configured in your AWS account:

An AWS account with access to Amazon MWAA Serverless
AWS CLI v2 (latest version) installed and configured. To install or update, see Installing or updating to the latest version of the AWS CLI.
An Amazon S3 bucket for storing DAG definitions and code bundles
An IAM role that MWAA Serverless can assume (see the execution role setup below)

Walkthrough: Building a serverless CSV-to-JSON pipeline

In this walkthrough, we build a pipeline that converts CSV files to JSON format – a common data transformation for downstream APIs and analytics systems that consume JSON. The pipeline uses a PythonOperator for the conversion logic and a BashOperator to verify the output. Here is what the pipeline does:

Reads a CSV file from an Amazon S3 bucket
Converts it to JSON format with column type inference
Writes the JSON file back to an Amazon S3 bucket
Validates record counts match between source and output

Step 1: Create the execution role

Create an IAM role that your workflow assumes at runtime. The trust policy must allow the airflow-serverless.amazonaws.com service to assume the role:

cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "airflow-serverless.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the role and attach an inline policy granting least-privilege access to your S3 bucket:

aws iam create-role \
  --role-name MWAAServerlessExecutionRole \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy \
  --role-name MWAAServerlessExecutionRole \
  --policy-name MWAAServerlessAccessPolicy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::amzn-s3-demo-mwaa-data",
        "arn:aws:s3:::amzn-s3-demo-mwaa-data/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/mwaa-serverless/*"
    }
  ]
}'

Step 2: Write the Python module

Create a file called csv_to_json.py with the conversion logic:

# csv_to_json.py
import csv
import json
import boto3
import io

def convert(**kwargs):
    """Read a CSV from S3 and write it back as JSON lines."""
    bucket = "amzn-s3-demo-mwaa-data"
    source_key = "raw/sales_data.csv"
    output_key = "processed/sales_data.json"

    s3 = boto3.client("s3")

    # Read source file
    response = s3.get_object(Bucket=bucket, Key=source_key)
    content = response["Body"].read().decode("utf-8")

    # Parse CSV
    reader = csv.DictReader(io.StringIO(content))
    rows = list(reader)

    # Type inference - convert numeric fields
    for row in rows:
        for key, value in row.items():
            try:
                row[key] = float(value)
            except (ValueError, TypeError):
                pass

    # Write as JSON lines
    output = "\n".join(json.dumps(row) for row in rows) + "\n"
    s3.put_object(Bucket=bucket, Key=output_key, Body=output.encode("utf-8"))

    print(f"Converted {len(rows)} rows to JSON lines")
    print(f"Output: s3://amzn-s3-demo-mwaa-data/{output_key}")
    return {"rows": len(rows), "output_key": output_key}

This function uses boto3 (which comes pre-installed with the MWAA Serverless execution environment) and Python’s built-in csv and json modules. The conversion reads the CSV, infers numeric types, and writes a JSON lines file back to the S3 bucket.

Step 3: Write the verification script

Create a file called verify_output.sh. This script validates the pipeline output by comparing the record count in the source CSV against the output JSON file. If the counts do not match, the task fails with a non-zero exit code, which causes the workflow run to fail.

#!/bin/bash
echo "=== Data Validation ==="

# Count source records (skip CSV header)
SOURCE_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/raw/sales_data.csv - | tail -n +2 | wc -l)
echo "Source CSV records: $SOURCE_COUNT"

# Count output records
OUTPUT_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/processed/sales_data.json - | wc -l)
echo "Output JSON records: $OUTPUT_COUNT"

# Validate counts match
if [ "$SOURCE_COUNT" -ne "$OUTPUT_COUNT" ]; then
    echo "FAILED: Record count mismatch (source=$SOURCE_COUNT, output=$OUTPUT_COUNT)"
    exit 1
fi

echo "PASSED: Record counts match ($OUTPUT_COUNT records)"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

This script runs the AWS CLI, which is bundled as a dependency in the code package. The s3 cp streams the file content to stdout without writing to disk, allowing standard shell tools like wc -l and tail to process it. The execution role credentials are automatically available in the execution environment, so the CLI can access S3 without additional configuration.

Step 4: Package and upload the code to Amazon S3

Since the verification script uses the AWS CLI, bundle it as a dependency in the ZIP archive along with your Python module and shell script:

BUCKET="amzn-s3-demo-mwaa-data"
REGION="us-east-1"

# Install awscli into a package directory
pip install awscli \
  --target my_package/ \
  --platform manylinux2014_x86_64 \
  --python-version 3.12 \
  --only-binary=:all:

# Add your module
cp csv_to_json.py my_package/
cp verify_output.sh my_package/

# Create the ZIP archive
cd my_package && zip -r ../code_bundle.zip . && cd ..
# Upload to S3
aws s3 cp code_bundle.zip s3://$BUCKET/code/code_bundle.zip --region $REGION

Upload a sample CSV file for testing:

cat > sales_data.csv << 'EOF'
date,region,product,units,revenue
2026-07-01,us-east,widget-a,150,4500.00
2026-07-01,eu-west,widget-b,89,2670.00
2026-07-02,us-east,widget-a,203,6090.00
2026-07-02,ap-south,widget-c,67,1340.00
2026-07-03,us-east,widget-b,178,5340.00
EOF

aws s3 cp sales_data.csv s3://$BUCKET/raw/sales_data.csv --region $REGION

Step 5: Define the DAG (YAML)

MWAA Serverless uses a declarative YAML format for DAG definitions. Create a file called conversion_dag.yaml:

csv_to_json_pipeline:
  start_date: "2026-01-01"
  schedule: null
  tasks:
    convert_to_json:
      operator: airflow.operators.python.PythonOperator
      python_callable: csv_to_json.convert
    verify_output:
      operator: airflow.operators.bash.BashOperator
      bash_command: "verify_output.sh"
      dependencies:
        - convert_to_json

This DAG defines two tasks:

convert_to_json – Runs the convert function from the Python module to transform CSV to JSON lines.
verify_output – Runs a shell script that validates the pipeline output by comparing source and output record counts, failing the task if they do not match.

Upload the DAG definition to S3. Note: You can also run inline Bash commands directly without a shell script.

aws s3 cp conversion_dag.yaml s3://$BUCKET/dags/conversion_dag.yaml --region $REGION

Step 6: Create the workflow

Create the MWAA Serverless workflow, referencing the DAG definition and the code bundle:

ROLE_ARN="arn:aws:iam::<your-account-id>:role/MWAAServerlessExecutionRole"

aws mwaa-serverless create-workflow \
  --name csv-to-json-workflow \
  --definition-s3-location Bucket="$BUCKET",ObjectKey="dags/conversion_dag.yaml" \
  --code '{"S3Location": {"Bucket":"'"$BUCKET"'","ObjectKey":"code/code_bundle.zip"}}' \
  --role-arn $ROLE_ARN \
  --region $REGION

The response includes a WorkflowArn that you use to trigger runs:

{
  "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123",
  "CreatedAt": "2026-07-15T10:30:00.000000+00:00",
  "WorkflowVersion": "a1b2c3d4e5f6"
}

Step 7: Run the workflow

Trigger a workflow run:

WORKFLOW_ARN="arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123"

aws mwaa-serverless start-workflow-run \
  --workflow-arn $WORKFLOW_ARN \
  --region $REGION

The response confirms the run has started:

{
  "RunId": "6OZV9ABF9enHKXk",
  "Status": "STARTING"
}

Step 8: Monitor execution

Check the status of your run:

RUN_ID="6OZV9ABF9enHKXk"

aws mwaa-serverless get-workflow-run \
  --workflow-arn $WORKFLOW_ARN \
  --run-id $RUN_ID \
  --region $REGION

A successful run returns:

{
  "RunDetail": {
    "Duration": 45,
    "RunState": "SUCCESS",
    "TaskInstances": ["ex_abc123_convert_to_json_1", "ex_abc123_verify_output_1"]
  },
  "RunId": "6OZV9ABF9enHKXk",
  "RunType": "ON_DEMAND",
  "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123",
  "WorkflowVersion": "a1b2c3d4e5f6"
}

Step 9: Verify the output

Confirm the JSON file was written to the S3 bucket:

# List the output file
aws s3 ls s3://$BUCKET/processed/sales_data.json --region $REGION

You should see the JSON file:

2026-07-15 10:32:45 1847 sales_data.json

You can also verify task-level output in Amazon CloudWatch Logs. Open the log group for your workflow and find the convert_to_json task log stream:

Converted 5 rows to JSON lines
Output: s3://amzn-s3-demo-mwaa-data/processed/sales_data.json

Considerations and limits

When planning your workloads on MWAA Serverless with these operators, keep the following considerations in mind:

Code bundle size – ZIP archives must be under 250 MB per bundle.
Network access – Python and Bash tasks do not have internet access. They can reach a limited set of AWS services required for the runtime to function (Amazon S3, Amazon ECR, and Amazon CloudWatch) but cannot call other AWS services or external endpoints. If your workflow requires calls to external APIs, preprocess that data and store it in an Amazon S3 bucket before invoking the workflow.
Runtime dependencies – boto3 and the Python standard library are pre-installed. For additional packages (such as pandas or requests), bundle them in your ZIP archive following the Amazon MWAA Serverless packaging guidelines.
Execution timeout – Tasks are subject to the workflow’s configured timeout limits.
Python version – Check the Amazon MWAA Serverless documentation for the currently supported Python runtime version.
DAG format – MWAA Serverless uses YAML-based DAG definitions, not traditional Python DAG files. If you are migrating from MWAA Provisioned, you will need to convert your DAGs to the YAML format.
Operators not supported – Some Airflow community operators and custom plugins are not available in the Serverless runtime. Refer to the documentation for the full compatibility list.

Clean up

To avoid ongoing charges, delete the resources you created in this walkthrough. The following commands remove the workflow, S3 objects, and IAM role:

Note: $WORKFLOW_ARN is defined in Step 7.

# Delete the workflow
aws mwaa-serverless delete-workflow \
  --workflow-arn $WORKFLOW_ARN \
  --region $REGION

Note: $BUCKET is exported in Step 4. If appropriate, delete the bucket as well.

# Remove S3 objects
aws s3 rm s3://$BUCKET/code/code_bundle.zip
aws s3 rm s3://$BUCKET/dags/conversion_dag.yaml
aws s3 rm s3://$BUCKET/raw/sales_data.csv
aws s3 rm s3://$BUCKET/processed/sales_data.json

# Delete the IAM role
aws iam delete-role-policy \
  --role-name MWAAServerlessExecutionRole \
  --policy-name MWAAServerlessAccessPolicy

aws iam delete-role --role-name MWAAServerlessExecutionRole

Conclusion

With native support for PythonOperator and BashOperator, you can now run the custom code execution patterns that many data engineering teams rely on daily directly in MWAA Serverless. Run data transformations, format conversions, validations, and shell scripts in the serverless runtime – without provisioning additional compute or managing containers.

If you are running Airflow workloads on MWAA Provisioned or self-managed infrastructure, your existing PythonOperator and BashOperator logic requires minimal changes. Convert your Python DAG files to the YAML format, package your code as a bundle, and you are ready to run on MWAA Serverless.

To get started, visit the Amazon MWAA Serverless documentation and try the walkthrough earlier in this post with your own data. For pricing details, visit the Amazon MWAA pricing page. We look forward to your feedback.

About the authors

GPU-accelerated Apache Spark with Amazon EMR and NVIDIA RTX PRO 4500 on Amazon EC2 G7 instances runs up to 3.7x faster

2026-08-25 McCall Peltier

Post Syndicated from McCall Peltier original https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/

For years, Apache Spark has been the backbone of large-scale data processing. However, as datasets grow and artificial intelligence and machine learning (AI/ML) pipelines become more complex, modern workloads demand more computational power. Feature engineering for machine learning models, large-scale extract, transform, and load (ETL) transformations, and real-time analytics workloads are computationally intensive by nature. GPU-accelerated instances improve performance and transform jobs that once took hours into minutes, so you can iterate on models faster and reduce operational costs. You can process larger datasets in single batches, make decisions in real time, and achieve strong performance without over-provisioning infrastructure.

We’re excited to share the benchmarking results on Amazon EMR with Amazon Elastic Compute Cloud (Amazon EC2) G7 instances, powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. For data engineers and data scientists running Apache Spark workloads, this means faster pipelines, shorter iteration cycles, and more time spent on insights.

Amazon EMR on EKS natively supports the NVIDIA cuDF plugin for Apache Spark. This support is the result of joint engineering between AWS and NVIDIA to qualify the cuDF plugin for Amazon EMR, co-optimize Spark execution paths for RTX PRO 4500, and validate performance at scale through shared TPC-DS benchmarking on Amazon EC2 G7 instances. Now, Apache Spark workloads on Amazon EMR on EKS run up to 3.7x faster with Amazon EC2 G7 GPU instances than with comparable CPU instances, and require no changes to existing Spark code.

In the TPC-DS 3 TB benchmark, at the 64 GB memory tier, EC2 G7 instances with RTX PRO finished in 4.7 minutes. If you run large-scale data processing pipelines, you can cut job run times by more than two-thirds while maintaining full compatibility with the applications you already have in production.

The use cases that benefit most are those where speed directly unlocks business value. In AI/ML feature engineering, faster Spark jobs mean data science teams can iterate on features more quickly, reducing the time from raw data to trained model. In complex ETL pipelines, like financial transactions, clickstream aggregation, or supply chain data consolidation, GPU acceleration compresses multi-hour batch windows into near-real-time processing. For real-time analytics, teams running fraud detection, personalization engines, or operational dashboards can process larger volumes of data within tighter latency windows, without redesigning their architecture.

Beyond data analytics, the G7 instances will support a broad range of AI and graphics workloads, including conversational AI, content generation, recommender systems, and video streaming and rendering. Built on the AWS Nitro System, they deliver the security and resource efficiency that production AI, analytics, and graphics workloads demand.

The following sections walk through the cluster configuration, benchmark methodology, and performance results.

Cluster configuration

We benchmarked four instance types to measure the real-world performance of G7 GPU instances against comparable CPU instances for Spark SQL performance. The g7.4xlarge also provides 80 Gbps network bandwidth (compared to 15–17 Gbps on the CPU baselines) and uses RapidsShuffleManager. However, CPU runs showed no evidence of being network- or shuffle-bound at this cluster scale. All tests used Amazon EMR on EKS 7.12.0 with Apache Spark 3.5.6 and cuDF plugin 26.04.2, running the full TPC-DS benchmark at 3 TB scale across 103 queries. Each experiment ran 5 iterations. We report the median. Data was stored as Parquet on Amazon Simple Storage Service (Amazon S3) (same-region gateway endpoint). All instances were launched in a single Availability Zone.

Instance specifications

All four instance types share the same compute footprint of 16 vCPUs and 64 GB system RAM. The g7.4xlarge additionally includes an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB of dedicated video memory (VRAM), which the cuDF plugin uses to accelerate Spark SQL operations. The baseline for all speedup and cost comparisons is m9gd.4xlarge (Graviton), the lowest-cost CPU instance in the group.

.	g7.4xlarge	m9gd.4xlarge	m8id.4xlarge	m8a.4xlarge
Architecture	x86_64	arm64 (Graviton)	x86_64	x86_64
vCPU	16	16	16	16
RAM	64 GB	64 GB	64 GB	64 GB
GPU	1× RTX PRO 4500 Blackwell (32 GB VRAM)	–	–	–
NVMe	875 GB	950 GB	950 GB	EBS only (GP3 16k IOPS and 2000 MB/s throughput to match NVMe
Network	80 Gbps	Up to 17 Gbps	Up to 15 Gbps	Up to 15 Gbps

The g7.4xlarge uses the RTX PRO 4500 Blackwell Server Edition GPU. The CPU baselines cover all three major architectures: m8id.4xlarge (Intel x86), m8a.4xlarge (AMD x86), and m9gd.4xlarge (Graviton arm64).

Spark configuration

All instances used eight executor nodes with the following configuration:

Configuration	GPU instances	CPU instances
Amazon EMR release	emr-7.12.0-spark-rapids-latest	emr-7.12.0-latest
executor.cores	14	14
executor.instances	8	8
executor.memory	20G	20G
executor.memoryOverhead	30G	30G
spark.plugins	com.nvidia.spark.SQLPlugin	–
rapids.memory.pinnedPool.size	8G	–
rapids.sql.concurrentGpuTasks	3	–
shuffle.manager	RapidsShuffleManager	default (sort)
sql.adaptive.enabled	true	true
io.compression.codec	zstd	zstd

CPU instances use the same 30 GB memoryOverhead as GPU to make sure that the memory comparison is apples-to-apples. This setting reserves off-heap memory for shuffle and caching on both sides.

For GPU instances, the cuDF plugin offloads Spark SQL operations to the GPU automatically. No code changes are required. The executor.memoryOverhead value is set higher on GPU instances to accommodate GPU memory management and the RAPIDS shuffle manager.

The cuDF plugin automatically falls back to CPU execution for unsupported operators and user-defined functions (UDFs). Your job still completes, but those stages run without GPU acceleration. To identify which operations run on GPU compared to CPU, set spark.rapids.sql.explain=NOT_ON_GPU in your Spark configuration. For a pre-migration assessment of your workloads, use the NVIDIA cuDF tool to estimate GPU acceleration potential before moving to G7 instances.

To tune settings like concurrentGpuTasks and pinnedPool.size, use the Spark History Server on Amazon EMR on EKS, which provides per-stage execution details to identify CPU fallback and shuffle bottlenecks.

Getting started

Reference the Using cuDF Accelerator for Apache Spark with Amazon EMR on EKS for detailed setup instructions.

Prerequisites

Before running GPU-accelerated Spark on Amazon EMR on EKS, make sure the following are in place:

Amazon EMR on EKS release version 6.9.0 or later (this post uses emr-7.12.0-spark-rapids-latest).

The -spark-rapids release variant ships the NVIDIA cuDF plugin pre-installed.

Amazon Elastic Kubernetes Service (Amazon EKS) cluster with a GPU-enabled node group using G7 instances.
Node AMI: AL2023_x86_64_NVIDIA (Amazon EKS optimized accelerated AMI).

NVIDIA device plugin installed in the cluster to expose GPUs to Kubernetes pods:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml

Amazon EMR on EKS virtual cluster registered to the EKS namespace.

To validate GPU availability on your nodes:

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Note: Getting started with GPU-accelerated Spark on Amazon EMR is straightforward. To use the latest cuDF plugin, overlay the latest version (for example, 26.04.2 as of May 2026) onto the Amazon EMR RAPIDS image using an initContainer technique. This replaces the bundled cuDF JAR with a newer version while preserving all other Amazon EMR dependencies. We recommend using the latest Amazon EMR release to get the most up-to-date cuDF plugin for better performance. In our benchmarks, upgrading from cuDF plugin 25.08.0 to 26.04.2 reduced runtime by 36–38 percent. Download the latest cuDF plugin JAR from the NVIDIA repository. AWS Support covers Amazon EMR. For issues specific to a cuDF JAR, file a GitHub issue or contact NVIDIA at [email protected].

Performance benchmarks and cost efficiency

We ran the full TPC-DS benchmark suite (103 queries) at 3 TB scale on 8-node clusters in us-east-1. The following table summarizes the results:

.	GPU instances	CPU instances
Cost per run	$2.06	$2.93–$3.18
Total time (103 queries)	281s (4.7 min)	1,010–1,043s (16.8–17.4 min)
Speedup compared to CPU instances	3.7×	baseline

Cost per run is the total cluster cost for the benchmark’s duration: Cluster $/hr × (median runtime ÷ 3,600). The hourly rate combines the EC2 On-Demand cost for all 8 nodes and the Amazon EMR on EKS charge for the vCPU and memory the Spark pods consume. Both are billed per second (one-minute minimum), so you pay only for what a job uses while it runs. All runs used Amazon EMR on EKS 7.12.0 in us-east-1, with 8 × 4xlarge nodes (128 vCPU) on both the GPU and CPU sides. The g7.4xlarge cluster runs at $26.35/hr (8 × $3.042 EC2 = $24.34, plus $2.01 for Amazon EMR on EKS) and finishes in 281 seconds, at $2.06 per run. The CPU clusters run at a lower hourly rate ($10.43–$10.99) but take 1,010–1,043 seconds, landing at $2.93–$3.18 per run. All prices reflect On-Demand pricing in us-east-1 as of May 2026. G7 instances are also eligible for EC2 Spot and Compute Savings Plans, which can further reduce costs for recurring batch workloads.

Cost-per-run calculations include EC2 and Amazon EMR charges only. They exclude the EKS control-plane fee, EBS volumes, S3 request and storage costs, and the driver pod.

Bar chart of total TPC-DS runtime by instance type, showing g7.4xlarge finishing far faster than the CPU instances

Figure 1: Total runtime by instance type for all 103 TPC-DS queries at 3 TB scale. The g7.4xlarge with GPU acceleration completed the benchmark in 4.7 minutes, 3.7× faster than CPU instances (16.8-17.4 minutes)

Bar chart of total cost per benchmark run by instance type, showing the g7.4xlarge GPU instance costing less than the CPU instances

Figure 2: Total cost per benchmark run, including both Amazon EC2 instance and Amazon EMR on EKS cost across all 8 nodes. Despite a ~2.5× higher hourly rate, the g7.4xlarge GPU instance costs up to 31% less per run than Graviton because it finishes the workload 3.7× faster

Where GPU acceleration excels

GPU acceleration completed the 103-query power run in 281s compared to 1,032s on CPU, an overall 3.7× speedup that saves 750 seconds per run. GPU was faster on 102 of 103 query executions.

GPU acceleration delivers the largest gains on the long-running, compute- and shuffle-heavy queries where kernel throughput outweighs launch overhead. The biggest absolute time savings:

Query	CPU time	GPU time	Speedup	Time saved
q24 (part 1+2)	81.6s	15.9s	~5.1×	65.7s
q23 (part 1+2)	79.4s	16.4s	~4.9×	63.1s
q93	63.7s	5.6s	11.4×	58.1s
q76	30.4s	3.4s	9.0×	27.0s
q64	35.4s	8.5s	4.2×	26.9s
q50	27.6s	3.5s	7.9×	24.0s

Speedup distribution across all 103 executions:

Speedup band	Queries
≥5×	15
4–5×	13
3–4×	21
2–3×	26
1–2×	27
<1× (CPU faster)	1

Median per-query speedup 2.94× (geomean 2.84×). The heaviest wins (q50, q76, q93) are aggregation- and shuffle-join-intensive queries that convert cleanly to GpuHashAggregate and GpuBroadcastHashJoin.

Where CPU wins

With RAPIDS 26.04.2, the following query showcases a workload pattern where CPU was faster:

Query	CPU time	GPU time	Ratio	Root cause
q16	0.96s	1.44s	CPU 1.5x faster	Trivial/near-empty scan. Sub-second runtime where GPU kernel-launch overhead is not amortized

Choosing the right instance

Instance	Best for	Summary
g7.4xlarge (RTX PRO GPU)	Fastest and most cost-effective	Up to 3.7× faster than comparable CPU instances and up to 31% cheaper per run. Completes in 4.7 min compared to 17.2 min. Best choice for both speed and cost efficiency.
CPU instances (m8a / m8id / m9gd)	Flexibility, availability, and always-on workloads	Multiple architecture options deliver similar Spark SQL performance. Choose CPU when GPUs are unavailable, when clusters need to remain running continuously (for example, overnight jobs ready for next-day analysis), or when workloads cannot use GPU acceleration. CPU instances offer broad availability and predictable capacity without startup delays.

G7 instances require a G-instance vCPU service quota in your account (default is often 0 for GPU types). Request a quota increase through the Service Quotas console, or use On-Demand Capacity Reservations (ODCRs) to guarantee availability for recurring batch jobs.

Based on these benchmark results, consider evaluating GPU acceleration for your own Apache Spark workloads. Start by identifying compute-intensive operations in your current pipelines, particularly those involving large-scale aggregations, joins, or machine learning feature engineering that could benefit from the performance improvements demonstrated here.

Conclusion

Amazon EMR on EKS with NVIDIA RTX PRO 4500 together provide a meaningful step forward for teams running data-intensive Spark workloads at scale. Whether you’re building ML pipelines that demand rapid feature iteration, running complex ETL transformations across massive datasets, or powering real-time analytics that can’t afford to wait on slow batch jobs, GPU-accelerated Spark on G7 delivers the performance and speed to do more. As data and AI workloads continue to evolve, GPU-accelerated analytics on Amazon EMR is becoming the foundation for data teams. Get started with GPU-accelerated Spark on Amazon EMR on EKS today by visiting Amazon EMR documentation to launch your first G7-powered cluster and see the performance gains for yourself.

About the authors

Long-term system tables retention in Amazon Redshift with Amazon S3 Tables

2026-08-20 Nidhi Nayak

Post Syndicated from Nidhi Nayak original https://aws.amazon.com/blogs/big-data/long-term-system-tables-retention-in-amazon-redshift-with-amazon-s3-tables/

Amazon Redshift system tables capture a continuous stream of operational signals: every query that runs, every connection that is made. This data powers observability, performance analysis, and compliance auditing across your data warehouses. Until now, the system tables retained this critical data for only 7 days, making long-term compliance and auditing difficult without custom workarounds.

Amazon Redshift system table integration with Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), automatically delivers your system table logs data to Amazon S3 Tables and stores them in Apache Iceberg format. You can configure retention periods for Amazon Redshift system table beyond the current 7-day limit, giving you extended compliance, auditing, and cross-warehouse observability without custom ETL pipelines or cluster resource consumption. Your data is open, durable, and queryable from Amazon Redshift, Amazon Athena, AWS Glue, Amazon EMR, or other Apache Iceberg-compatible engines.

In this post, we walk through how the Amazon Redshift system table integration delivers log data to Amazon S3 Tables. This feature is supported on RA3 and RG provisioned clusters and Amazon Redshift Serverless workgroups.

The challenge

If you run Amazon Redshift, you often face operational challenges driven by the 7-day system table retention limit:

Limited query trend visibility: You want to compare how the same query performed 30 days ago compared to today. When performance shifts gradually, extended baselines enable data-driven root cause analysis rather than reactive troubleshooting.
Enable before-and-after comparisons: When you add a new workload, change instance type, or adjust Workload Management (WLM) queues, you want to measure the impact precisely. Extended retention preserves the baseline data you need.
Unlock seasonal capacity planning: Month-end spikes, quarter-close surges, and annual peaks require months of historical data to identify and plan. Extended retention reveals seasonal patterns across months and years.
Custom ETL pipeline overhead: To work around the retention limit, teams build custom pipelines that copy system table data hourly/daily into persistent tables within Amazon Redshift Managed Storage. These pipelines consume cluster resources, compete with production workloads, and require ongoing engineering maintenance. When Amazon Redshift updates system table schemas and data sharing configurations, these pipelines require manual intervention and create gaps in records.
Compliance requirements: Regulated industries are required to maintain audit trails spanning months or years. The 7-day limit requires custom infrastructure to meet these requirements. Amazon S3 Tables integration for Amazon Redshift system tables now addresses this.

How it works

Amazon Redshift system tables integration with Amazon S3 Tables is a fully managed capability that automatically writes Amazon Redshift system table data to Amazon S3 tables in Apache Iceberg format. AWS handles partitioning, compression, and retention management automatically. The log writing process runs in an isolated background process that alleviates resource contention with production workloads. AWS manages the pipelines for you.

The feature supports over 25 system views at launch – see the supported system views documentation.

Setting up

Follow these steps to enable system table integration with Amazon S3 Tables from the Amazon Redshift console:

Open the Amazon Redshift console and navigate to the System table integrations page. You can also access this from the detail page of your provisioned cluster or Serverless workgroup.
Choose Create System table integration. This launches the configuration wizard.
Select the Amazon Redshift Provisioned cluster or Amazon Redshift Serverless workgroup that you want to enable the feature on.

Figure 1: Selecting the Amazon Redshift data warehouse in the System table integration wizard
Choose the system views to publish from the Available system tables list. Select individual SYS_* views, or choose Select all supported system tables to publish all current and future supported views. If you select all, new views added in the future are automatically included without requiring a configuration change.

Figure 2: Choosing the system views to publish from the Available system tables list
Select the deployment model. Choose how data is organized in Amazon S3 Tables:

Individual S3 table per system table per data warehouse to keep this warehouse’s data in its own set of tables.
Shared S3 table per system table across data warehouses to consolidate data from multiple warehouses in the account into a shared set of tables.

Optionally configure encryption with an AWS Key Management Service (AWS KMS) customer managed key. By default, data is encrypted with Amazon S3-managed key (SSE-S3) encryption.
Save your changes. Amazon Redshift begins publishing the selected views to Amazon S3 Tables and continues adding new records on a fixed frequency.

To verify the integration is active:

Navigate to your cluster or workgroup detail page.
Check the integration status and the last ingestion time for each view.
You can also view the published data from the Amazon S3 Tables console.

After it’s enabled, Amazon Redshift writes log data to Amazon S3 tables periodically through an isolated background process, separate from production workloads. To start querying the retained logs, you will need to perform a one-time setup that connects your Amazon Redshift environment to Amazon S3 Tables data through AWS Glue Catalog. Complete the following steps:

Set up an AWS Identity and Access Management (IAM) role with the necessary permissions for AWS Glue Data Catalog and Amazon S3 Tables access, then associate it with your Amazon Redshift cluster or Amazon Redshift serverless namespace.
In AWS Glue Data Catalog, create a resource link that points to the Amazon S3 Tables database where your logs reside.

In Amazon Redshift, create an external schema that references the resource link:

CREATE EXTERNAL SCHEMA <schema_name>
FROM DATA CATALOG
DATABASE '<resource_link_database>'
IAM_ROLE '<iam_role_arn>';

With this in place, you can query your historical system table data using familiar 2-part notation:
```
SELECT * FROM <schema_name>.<table_name>;
```

Because access to Amazon S3 Tables is read-only, the integrity of your audit trails is inherently preserved.

For detailed setup instructions including IAM policy examples, see Registering the S3 Tables bucket with AWS Glue Data Catalog.

Your data is now in Apache Iceberg

Your system table data is stored in Apache Iceberg, an open table format, so you have the freedom to choose a compatible query engine. Your observability and auditing data works with the tool you already use.

You can analyze your operational data using:

Amazon Redshift: After the S3 table bucket is integrated with AWS Glue Data Catalog, create an external schema in Amazon Redshift pointing at the resource link to query the retained tables.
Amazon Athena: Run serverless SQL queries against historical logs with zero infrastructure provisioning.
AWS Glue: Build automated data processing and transformation jobs on top of your operational data.
Amazon EMR: Run Spark-based analytics at scale for complex cross-warehouse analysis.

Because the data is stored in open Apache Iceberg format in Amazon S3 Tables, you can query it with Amazon Redshift, Amazon Athena, AI agent skills for natural-language queries, Amazon SageMaker Unified Studio, an Iceberg-compatible engine, business intelligence (BI) tools, and observability systems.

Cost efficiency

Log delivery from Amazon Redshift to Amazon S3 Tables incurs no additional cost. You only pay for Amazon S3 Tables storage, maintenance, and querying the data with the engine of your choice.

Solution overview

The following scenarios illustrate how Amazon Redshift system tables integration with Amazon S3 Tables addresses common operational, compliance, and observability challenges across your Amazon Redshift environment. We also built a dedicated skill, querying-aws-redshift, for this feature and embedded it into the AWS MCP Server so you can query Amazon Redshift system tables from Amazon S3 Tables.

Scenario 1: Track query trends over time

With months or years of SYS_QUERY_HISTORY data retained, you can trace how individual queries perform over extended periods. You can compare execution time, queue time, and resource consumption for a query across days, weeks, or months.

You can pinpoint exactly when performance started degrading and correlate it with what changed: a new schema, a spike in data volume, or an additional concurrent workload. Extended retention turns troubleshooting into proactive, data-driven root cause analysis.

Scenario 2: Assess workload impact before and after changes

Every workload change affects your system: a new ETL pipeline, an instance type change, a Workload Management (WLM) queue adjustment, or a new team of analysts running ad hoc queries. The question is always: how did this change affect performance?

With Amazon S3 Tables integration for Amazon Redshift system table, you can make data-driven decisions with confidence. Query SYS_QUERY_HISTORY to compare execution times, queue wait durations, and concurrency scaling events from the weeks before a change versus the weeks after. If you onboarded a new reporting workload two weeks ago and want to understand its effect on existing queries, the data to confirm that is already there, with zero custom pipeline required.

Scenario 3: Build observability dashboards

Your system table data is stored in Apache Iceberg and cataloged in AWS Glue, which means an observability or business intelligence (BI) tool that reads Apache Iceberg can connect directly to it. Visualize workload distribution trends in Amazon Quick Sight for executive reporting. Use Amazon SageMaker Unified Studio for deeper analytical exploration or to power AI-driven insights from your operational data. Beyond AWS services, connect your preferred third-party observability systems and BI tools to track query volumes, monitor connection patterns, set up alerts for anomalies, or correlate Amazon Redshift operational data alongside application-level logs.

Your observability and auditing data works with tools that you already use. Direct access to durable, structured operational data, with a tool you prefer.

Scenario 4: Plan capacity with seasonal context

Workload demand varies throughout the year. Month-end close, quarter-end reporting, annual planning cycles, and promotional events all create predictable usage spikes, but only if you have enough historical data to see the pattern.

With extended retention, you can analyze utilization trends across multiple business cycles. Identify when you consistently approach capacity limits, measure how demand shifts quarter over quarter, and validate whether your provisioned resources align with actual usage.

Scenario 5: Maintain compliance audit trails

For regulated industries, extended retention delivers a fully managed audit trail with built-in integrity.

SYS_CONNECTION_LOG records every authentication attempt. SYS_USERLOG captures user account changes. SYS_QUERY_HISTORY documents every query executed against your warehouse.

Configure retention to match your organization’s data retention policies: whether that is 90 days, one year, or multiple years. The read-only access policy helps prevent records from being altered after they are written, including by administrators.

Scenario 6: Centralize fleet observability across your warehouse

If you run multiple Amazon Redshift warehouses, you benefit from a unified view of operational data. The feature supports two deployment patterns to match your organizational structure:

Individual tables per warehouse: Each warehouse writes to its own dedicated Amazon S3 tables, providing complete data isolation for compliance-sensitive environments. To query multiple warehouses, a UNION operation is required.
Shared tables: Warehouses across the same account and same AWS Region write to a single shared set of Amazon S3 tables, with data distinguished by the warehouse_name column. Filter by warehouse for instant cross-cluster analysis.

Best practices

Identify warehouses with logs requiring isolation for privacy reasons and select the individual table per warehouse option for those. For the remaining warehouses, use the Shared tables (consolidated) option for ease of management.
Align retention duration with your compliance requirements. Configure the minimum retention period that satisfies your compliance requirements to reduce storage costs.
When querying retained system tables, filter on metadata columns such as warehouse_account_id, warehouse_region_name, warehouse_namespace_arn, warehouse_name, and s3_tables_ingestion_time to reduce scan scope and improve performance. This is particularly important when querying large volumes of historical data across multiple warehouses.
Rely on the built-in read-only access for audit trail integrity. Use the Amazon S3 Tables configuration APIs to manage retention and encryption settings.
Plan your encryption strategy early. Choose your encryption key carefully at setup, as changes require recreating the integration. If you anticipate consolidating warehouses in the future, choose a shared AWS KMS key from the start.

Conclusion

Amazon Redshift system table integration with Amazon S3 Tables replaces custom ETL pipelines with a fully managed solution to preserve your Amazon Redshift operational data. With automatic Apache Iceberg-based storage, open format queryability, and built-in audit integrity, you get months or years of observability data, fully managed. You can enable it through the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDKs.

To learn more, visit the Amazon Redshift system tables documentation.

About the authors

AWS Network Firewall now supports rule hit count

2026-08-20 Preetkumar Shah

Post Syndicated from Preetkumar Shah original https://aws.amazon.com/blogs/security/aws-network-firewall-now-supports-rule-hit-count/

As firewall rule sets grow in complexity, security teams face a common challenge: manual log analysis is used to determine which rules are actively matching traffic and which are consuming capacity without being triggered. This lack of visibility creates operational and compliance gaps. Organizations with governance policies that require removal of dormant rules after a defined period have no mechanism to identify them. Teams responsible for compliance frameworks such as Payment Card Industry (PCI) 4.0 and Digital Operational Resilience Act (DORA) can’t provide evidence that specific controls are actively functioning. Central teams managing firewalls on behalf of multiple business units have no way to determine which rules are unused or need updating.

In this post, you learn how a new AWS Network Firewall capability—rule hit count—addresses these challenges by providing traffic match data for stateful rules across both custom and managed rule groups. With this data, you can identify and remove unused rules, accelerate incident response, and validate security control effectiveness for compliance.

How it works

Rule hit counts track how often each stateful rule matches network traffic. The hit counter increments only when a rule match results in an alert log being created. This means any rule with an alert, drop, or reject action will increment the hit counter, because these actions generate alert logs. However, rules configured with a pass action don’t generate alert logs by default, meaning they won’t appear in the rule hit count metric.

To gain visibility into traffic matching pass rules, you can include the alert keyword within the pass rule. This generates an alert log while still permitting the traffic to its intended destination. The following Suricata rule demonstrates this approach:

pass tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"Pass and Log HTTPS traffic"; alert; sid:1000001; rev:1;)

This rule passes HTTPS traffic to its destination while also generating an alert log, making sure the rule appears in the hit count metric.

The rule hit count feature adds the following metadata to each alert log. Metadata is included by default and doesn’t require additional configuration:

“aws_metadata": { “resource_arn": “arn:aws:network-firewall:us-east-1:123456789012:stateful-rulegroup/StatefulRuleGroup” }

The following example shows a complete alert log with this metadata included:

{ 

    "firewall_name": "egress-and-east-west-firewall", 

    "availability_zone": "us-east-1a", 

    "event_timestamp": "1786112515", 

    "event": { 

        "tx_guessed": true, 

        "aws_category": "", 

        "tx_id": 0, 

        "app_proto": "http", 

        "ip_v": 4, 

        "src_ip": "10.2.1.205", 

        "src_port": 46240, 

        "event_type": "alert", 

        "alert": { 

            "severity": 3, 

            "signature_id": 10000003, 

            "rev": 0, 

            "signature": "Egress HTTP but not port TCP/80", 

            "action": "blocked", 

            "category": "" 

        }, 

        "ts_progress": "request_complete", 

        "flow_id": 927132830538451, 

        "dest_ip": "3.226.253.175", 

        "proto": "TCP", 

        "verdict": { 

            "action": "drop" 

        }, 

        "http": { 

            "hostname": "3.226.253.175", 

            "http_port": 4444, 

            "url": "/", 

            "http_user_agent": "curl/8.17.0", 

            "http_method": "GET", 

            "protocol": "HTTP/1.1", 

            "length": 0 

        }, 

        "tc_progress": "response_started", 

        "dest_port": 4444, 

        "pkt_src": "geneve encapsulation", 

        "aws_metadata": { 

            "resource_arn": "arn:aws:network-firewall:us-east-1:123456789012:stateful-rulegroup/StatefulRuleGroup" 

        }, 

        "timestamp": "2026-08-07T14:21:55.611810+0000", 

        "direction": "to_server" 

    } 

}

The alert log data in the preceding example is the source for rule hit count metrics. Network Firewall pushes these alert logs to your Amazon CloudWatch Logs or Amazon Simple Storage Service (Amazon S3). To identify the specific rule that generated an alert, you can search using the combination of the sid (signature ID) and resource_arn fields. The firewall monitoring dashboard uses these fields to generate hit counts for each rule, so you can review rule activity directly on the dashboard without querying logs.

You can also access and analyze this data by querying those logs directly using CloudWatch Logs Insights for logs stored in CloudWatch, or Amazon Athena for logs stored in Amazon S3.

Getting started

Network Firewall rule hit count is enabled by default, so you don’t need to perform any additional configuration to start tracking rule hits on your firewall policies. This means that as soon as you deploy your firewall rules, you can begin to monitor which rules are being triggered, helping you gain visibility into your traffic patterns and identify potential security concerns.

Prerequisites

This walkthrough requires an existing network firewall configured to inspect traffic from your Amazon Virtual Private Cloud (Amazon VPC). If you don’t have one set up yet, follow the Getting started with AWS Network Firewall guide.

Additionally, ensure the following:

Alert log delivery must be configured: The firewall must have alert logging enabled. Note that rule hit count metadata is captured regardless of log destination; however, the native dashboard feature requires logs to be sent to CloudWatch Logs or Amazon S3.
Firewall monitoring must be enabled: To see the dashboard widget shown in figure 1, you must enable detailed monitoring through the firewall’s logging configuration or the Monitoring tab in the AWS Management Console for AWS Network Firewall. However, if you have a custom dashboard solution, the metadata required for rule hit count analysis is automatically included in the firewall logs regardless of whether detailed monitoring is enabled—so you can build your own visualizations using the log data directly.
Pass rules must include the alert keyword to appear in hit count metrics: Rules configured with a pass action don’t generate alert logs by default. To track pass rule activity in the hit count metric, include the alert keyword in your pass rules, as demonstrated in the How it works section of this post.

The Top Rule Hits dashboard shows aggregated hit counts per firewall across all Availability Zones within the AWS Region where that firewall is deployed. To view rule hit count metrics, open the Network Firewall console and select your firewall. Navigate to the Monitoring and observability section. Under Top analysis, you will see the Top Rule Hits metric. Select a lookback period to view rule activity within that timeframe.

Figure 1: Rule hit count from the dashboard

Figure 1 shows the Top Rule Hits panel from the AWS Network Firewall console, displaying the most frequently triggered stateful rules. It includes columns for Hit Count (with bar chart and fraction), percentage of total hits, Resource ARN, Signature ID, Description (the msg field from the Suricata rule), and Last Occurrence (UTC).

Signature IDs 2, 4, 6, and 8 are system-generated signatures corresponding to the firewall policy’s strict order default actions. Because these signatures originate from the policy rather than a rule group, the resource_arn field displays the firewall policy Amazon Resource Name (ARN) instead of a rule group ARN. They appear in the Top Rule Hits when the policy has default actions such as Drop established, Alert established, or their application-layer variants configured. These signatures fire on established connection packets that don’t match any explicit rule, enforcing the policy’s default deny posture.

The following examples demonstrate how rule hit counts help you address common operational challenges.

Identifying unused rules: Figure 1 shows all rule signature IDs and their descriptions (the msg field from the Suricata rule) that actively matched traffic during the selected lookback period. Any rule in your firewall policy whose signature ID doesn’t appear in this metric hasn’t matched any traffic during the specified timeframe. These rules are either stale or not ordered correctly within your rule group.
Accelerating incident response: Figure 1 shows signature ID 2525124575 (traffic_to_oast [oast[.]fun]) with six hits and a last occurrence of August 7, 2026, at 6:28:44 PM UTC. This rule is detecting traffic to an out-of-band application security testing (OAST) domain, which could indicate an attacker attempting to exfiltrate data or validate a vulnerability in your environment. By filtering the top rule hits metric to the timeframe of a suspected incident, your team can quickly identify this type of suspicious activity and scope the impact without manually parsing thousands of log entries.
Validating a newly added rule: Figure 1 shows signature ID 100000010 (Domain Category is AI/ML) with five hits and a last occurrence of August 7, 2026, at 6:28:21 PM UTC. After adding this rule to monitor or restrict traffic to AI/ML related domains, the hit count confirms the rule is actively matching traffic as intended. Similarly, signature ID 100000009 (Drop traffic to countries other than US) shows four hits, validating that the geofencing rule is functioning and blocking outbound connections to destinations outside the United States. These hit counts provide security teams with concrete evidence that newly deployed controls are working.

Pricing

Rule hit counts are included with Network Firewall at no additional cost. However, standard charges apply for storing and querying log data. If you configure log delivery to CloudWatch Logs, CloudWatch pricing applies. If you store logs in Amazon S3 and query them with Athena, standard Amazon S3 storage and Athena query charges apply. For complete pricing details, see AWS Network Firewall pricing.

Considerations

Keep the following in mind when you use rule hit counts:

To manage costs, review your log utilization and configure log filtering or retention policies.
Rule hit counts apply to stateful rules. Stateless rules don’t support hit count tracking at this time.
Rule hit counts are available in all AWS Regions where AWS Network Firewall is supported, except Middle East (UAE) and Middle East (Bahrain).

Conclusion

In this post, you learned how rule hit counts in AWS Network Firewall give you visibility into your firewall rule utilization and effectiveness. By tracking how frequently each rule matches traffic, you can identify unused or redundant rules, optimize rule ordering, validate security controls for compliance, and respond faster during security investigations. For more information, see AWS Network Firewall.

If you have feedback about this post, submit comments in the Comments section below.

Track SageMaker Unified Studio project costs with custom tags and AWS CUR

2026-08-20 Nisha Gambhir

Post Syndicated from Nisha Gambhir original https://aws.amazon.com/blogs/big-data/track-sagemaker-unified-studio-project-costs-with-custom-tags-and-aws-cur/

Organizations running machine learning (ML), analytics, and generative AI workloads on Amazon SageMaker Unified Studio domains and projects face a common cost governance challenge. System tags (AmazonDataZoneDomainId and AmazonDataZoneProject) are automatically propagated to all underlying project resources. However, custom tags such as CostCenter, Team, or Environment are not propagated to dynamic resources created through the Studio UI. This creates a gap when you need to report project costs grouped by custom tags.

In this post, we walk through a serverless solution that bridges this gap by enriching AWS Cost and Usage Report (CUR) data with custom project tags. By the end of this post, you can build an Amazon Quick Sight dashboard to filter and analyze Amazon SageMaker Unified Studio project costs by any custom tag dimension that you define. This gives your team the visibility to make informed spending decisions.

Solution overview

The solution consists of three automated subsystems:

Event-driven tag lookup management – An Amazon EventBridge rule captures Amazon DataZone project lifecycle events (Create, Update, Delete) and triggers an AWS Lambda function. The function maintains an Amazon DynamoDB lookup table that maps each project’s DomainId and ProjectId to its custom tags.
CUR enrichment pipeline – An AWS Glue extract, transform, and load (ETL) job reads CUR 2.0 Parquet data from Amazon Simple Storage Service (Amazon S3). The job joins each billing line item with the DynamoDB lookup table using the system tags (DomainId, ProjectId), appends the custom tag values as new columns, and writes the enriched data back to Amazon S3.
Cost visualization – An Amazon Quick Sight dashboard backed by a custom SQL dataset over Amazon Athena provides interactive cost and consumption analytics filtered by custom tags.

Architecture

The following diagram shows the end-to-end architecture:

Figure 1: SageMaker Unified Studio project custom tag cost reporting

The workflow is as follows:

An Amazon SageMaker Unified Studio administrator creates or updates a project with custom tags.
AWS CloudTrail captures the API call.
Amazon EventBridge matches the event.
The Lambda orchestrator writes the tag mapping to DynamoDB.
Separately, AWS Data Exports delivers CUR data to Amazon S3.
The AWS Glue ETL job enriches CUR line items with custom tags from DynamoDB.
The AWS Glue Crawler catalogs the enriched data.
Amazon Quick Sight visualizes costs by custom tags.

Prerequisites

Before deploying this solution, you need:

An Amazon SageMaker Unified Studio domain (you create projects after deployment).
AWS Cloud Development Kit (AWS CDK) CLI installed.
Python 3.12+.
Amazon Quick Sight Enterprise edition enabled in your account.
An AWS Identity and Access Management (IAM) user or role with permissions to deploy AWS CloudFormation stacks.

Step 1: Configure custom tags on your project profile

You configure custom tags on project profiles through the Amazon DataZone API. First, enable custom tags on your project profile:

aws datazone update-project-profile \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_PROFILE_ID \
  --region $REGION \
  --allow-custom-project-resource-tags \
  --project-resource-tags '[
  {"key": "CostCenter", "value": "default", "isValueEditable": true},
  {"key": "Team", "value": "default", "isValueEditable": true},
  {"key": "Environment", "value": "default", "isValueEditable": true}
]'

When creating or updating a project, set the tag values:

aws datazone update-project \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_ID \
  --project-profile-version latest \
  --region $REGION \
  --resource-tags '{"CostCenter": "CC-100", "Team": "ML-Platform", "Environment": "Production"}'

Important: The AmazonSageMakerProvisioning-<domainAccountId> role needs an inline policy that permits your custom tag keys. Without this, project environment deployment fails.

The following is the inline policy that’s used for the custom tags shared in this post:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCustomTagKeys",
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:DeleteTags",
        "cloudformation:TagResource",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack"
      ],
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:TagKeys": [
            "AmazonDataZone*",
            "CostCenter",
            "Team",
            "Environment"
          ]
        }
      }
    }
  ]
}

Step 2: Activate cost allocation tags

Activate the SageMaker Unified Studio system tags as cost allocation tags so they appear in CUR data:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Active"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Active"}
]'

These tags take up to 24 hours to start appearing in CUR reports after activation.

Step 3: Deploy the infrastructure

The solution is packaged as a CDK application. Clone the GitHub repository and deploy:

# Install dependencies
pip install -r requirements.txt

# Bootstrap CDK (first time only)
cdk bootstrap aws://$ACCOUNT_ID/$REGION

# Deploy
cdk deploy

This creates the following resources:

DynamoDB table (smus-project-tag-lookup) – stores project-to-tag mappings.
Lambda function (smus-orchestrator) – processes project lifecycle events.
Amazon EventBridge rule – matches Amazon DataZone CreateProject/UpdateProject/DeleteProject events.
S3 buckets – for raw CUR and enriched CUR data.
AWS Glue ETL job (smus-cur-enrichment) – enriches CUR with custom tags.
AWS Glue Crawler – catalogs enriched data.
Amazon Simple Notification Service (Amazon SNS) topic – pipeline failure alerts.

Note: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight), so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Step 4: Configure CUR delivery

Create a CUR 2.0 export through AWS Data Exports that delivers Parquet files to the CUR S3 bucket created by the stack. The following screenshots show the complete configuration process in the AWS Billing and Cost Management console.

To create the export, follow these steps:

Go to AWS Billing and Cost Management and then choose Data Exports.
Choose Create in the upper right corner of the Exports and dashboards page. The Data Exports console shows any existing exports, their status, export type, data table, and last refresh date.
On the Create export page, under Export details, select Standard data export and enter an export name. Under Data table content settings, select CUR 2.0.
Under Data table configurations, set Time granularity to Hourly. The configuration page also lets you choose additional export content options such as including resource IDs, split cost allocation data, caller identity allocation data, and capacity reservation columns.
Under Data export delivery options, set Compression type and file format to Parquet. Under Data export storage settings, configure the S3 bucket to: smus-cur-report-{account-id}-{region} and set the S3 path prefix as needed. Choose Create to finish.

Data Exports console listing existing exports with status, type, and last refresh date

Figure 2: Data Exports page listing existing exports

Create export page with Standard data export selected and CUR 2.0 chosen

Figure 3: Create export page with Standard data export and CUR 2.0 selected

Figure 4: Data table configurations with time granularity set to Hourly

Data export delivery options with Parquet format and the S3 storage destination configured

Figure 5: Data export delivery options with Parquet format and S3 storage settings

Step 5: How the event-driven tag capture works

When a project is created or updated in Amazon SageMaker Unified Studio (through the Studio UI or API), the following happens automatically:

CloudTrail logs the Amazon DataZone API call.
Amazon EventBridge matches the event.
Amazon EventBridge invokes the Lambda function.
The Lambda extracts custom tags from the CloudTrail event payload.
The Lambda writes a record to DynamoDB with the DomainId, ProjectId, and all custom tag key-value pairs.

The Lambda function reads tags directly from the responseElements.resourceTags field of the CloudTrail event rather than making a separate GetProject API call. This avoids a race condition where GetProject might return empty tags while the project is in the UPDATING state.

def _extract_tags_from_event(detail):
    tags = {}
    response_elements = detail.get("responseElements") or {}
    for tag_entry in response_elements.get("resourceTags", []):
        if isinstance(tag_entry, dict) and "key" in tag_entry:
            tags[tag_entry["key"]] = tag_entry["value"]
    request_params = detail.get("requestParameters") or {}
    req_tags = request_params.get("resourceTags", {})
    if isinstance(req_tags, dict):
        tags.update(req_tags)
    return tags

Step 6: How the CUR enrichment works

The AWS Glue ETL job runs on a schedule (after each CUR delivery):

Reads CUR Parquet files from the CUR S3 bucket.
Reads all records from the DynamoDB lookup table.
Performs a left outer join on DomainId and ProjectId.
Appends custom tag columns (CostCenter, Team, Environment, and so on) to each CUR line item.
Writes enriched Parquet to the enriched S3 bucket.

Line items without a matching project in the lookup table retain all original columns with NULL custom tag values. No data is dropped.

joined_df = cur_df.join(
    lookup_df,
    on=(
        (cur_df[DOMAIN_COL] == lookup_df["domainId"])
        & (cur_df[PROJECT_COL] == lookup_df["projectId"])
    ),
    how="left_outer",
)

Step 7: Set up the Amazon Quick Sight dashboard

After the first ETL run and crawler execution, set up the Amazon Quick Sight dashboard:

python scripts/setup_quicksight.py \
  --account-id $ACCOUNT_ID \
  --region $REGION \
  --quicksight-user $QUICKSIGHT_USER_ARN

This creates a dashboard with five visuals:

Cost by Custom Tag (CostCenter) – horizontal bar chart.
Cost by Project – horizontal bar chart.
Daily Cost Trend – line chart.
Cost by Service per Project – stacked bar chart.
Usage by Project & Service – summary table.

And six interactive list filters: Domain, Project, CostCenter, Team, Environment, Service.

The custom SQL includes a CASE statement for service categorization:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_usage_amount,
  line_item_unblended_cost,
  resource_tags_user_amazondatazone_domain_id AS domain_id,
  resource_tags_user_amazondatazone_project AS project_id,
  costcenter, team, environment,
  CASE
    WHEN line_item_product_code = 'AmazonSageMaker' THEN 'SageMaker'
    WHEN line_item_product_code = 'AmazonS3' THEN 'S3'
    WHEN line_item_product_code = 'AWSGlue' THEN 'Glue'
    ELSE line_item_product_code
  END AS service_category
FROM "smus_cost_reporting"."enriched_cur"
WHERE line_item_unblended_cost > 0

Step 8: Verifying the solution

After deploying the infrastructure and setting up the dashboard, verify that each component of the pipeline is functioning correctly.

8.1 Verify Amazon EventBridge is capturing project events

Open the Amazon EventBridge console.
In the navigation pane, choose Rules.
Select the rule created by the CDK stack (for example, SmusCostReporting-ProjectTagRule).
Choose the Monitoring tab.
Confirm that the invocations are being recorded in the metrics.

Create or update an Amazon SageMaker Unified Studio project with custom tags using the following command:

aws datazone update-project \
  --domain-identifier <domain-id> \
  --identifier <project-id> \
  --custom-tags CostCenter=Engineering Team=DataPlatform Environment=Production

Within a few seconds, the Amazon EventBridge rule should show a new invocation in its metrics.

8.2 Verify DynamoDB schema and tag mappings

The DynamoDB lookup table uses a simple key schema:

Attribute	Type	Role
domainId	String	Partition Key
projectId	String	Sort Key
CostCenter	String	Custom tag
Team	String	Custom tag
Environment	String	Custom tag

Custom tags are stored as dynamic attributes. Any tag key set on a project becomes a column in the table.

8.2.1 Verify DynamoDB table contains tag mappings

Open the DynamoDB console.
Navigate to the table created by the stack (for example, SmusCostReporting-ProjectTagsTable).
Choose Explore table items.

Scan for your project with the following keys:

Partition key (domainId): <your-domain-id>
Sort key (projectId): <your-project-id>

Confirm the item contains the expected custom tag attributes (CostCenter, Team, Environment) with the values you assigned.

Alternatively, use the AWS CLI:

aws dynamodb get-item \
  --table-name SmusCostReporting-ProjectTagsTable \
  --key '{"domainId": {"S": "<domain-id>"}, "projectId": {"S": "<project-id>"}}'

8.3 Verify the AWS Glue ETL job enriches CUR data

Wait for the next CUR delivery (hourly if configured as described in Step 4).
Wait for the subsequent AWS Glue job execution.
Open the AWS Glue console.
In the navigation pane, choose ETL Jobs.
Confirm the job completed successfully (status: Succeeded).

Query the enriched data in Amazon Athena to confirm custom tag columns are populated:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_unblended_cost,
  costcenter,
  team,
  environment
FROM "smus_cost_reporting"."enriched_cur"
WHERE costcenter IS NOT NULL
LIMIT 10;

You should see rows with your custom tag values populated in the costcenter, team, and environment columns.

8.4 Verify the Amazon Quick Sight dashboard displays enriched data

Open the Amazon Quick Sight console and navigate to the dashboard created by the setup script.
Confirm that:
- The Cost by Custom Tag (CostCenter) bar chart displays cost data grouped by your CostCenter values.
- The list filters for CostCenter, Team, and Environment contain selectable values.
- Selecting a filter value correctly narrows the displayed data.
If the dashboard shows no data, verify that:
- The AWS Glue Crawler has run after the ETL job (check the crawler’s last run status in the AWS Glue console).
- The SPICE dataset has been refreshed. In the Amazon Quick Sight console, navigate to Datasets, select the dataset, and then choose Refresh now.

Figure 6 shows the Amazon Quick Sight dashboard with two side-by-side horizontal bar charts: Cost by Cost Center and Cost by Project. Domain Name and Project Name list filters appear at the top.

Amazon Quick Sight dashboard with Cost by Cost Center and Cost by Project bar charts and Domain and Project filters

Figure 6: Amazon Quick Sight dashboard showing cost data by custom tags, including Cost by Cost Center and Cost by Project bar charts with Domain Name and Project Name filters

Note: The first end-to-end cycle can take up to 48 hours depending on CUR delivery timing. After the initial cycle completes, subsequent updates will flow automatically on the configured schedule.

Operational considerations

Monitoring: The Amazon SNS topic smus-cost-reporting-alerts receives notifications when the AWS Glue ETL job fails or the Lambda orchestrator encounters repeated errors. Subscribe an email address or Slack webhook to stay informed. For instructions on how to create a subscription, see Subscribing to an Amazon SNS topic.

Cost: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight, SPICE) so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Scaling: The DynamoDB table uses on-demand capacity and can scale to accommodate your projects. You can scale the AWS Glue ETL job by increasing the number of workers for larger CUR datasets. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

New tag keys: When you add new custom tag keys to projects, the ETL automatically picks them up as new columns. The AWS Glue Crawler’s UPDATE_IN_DATABASE policy adds new columns to the catalog table without manual intervention.

Cleanup

Warning: The following cleanup steps will permanently delete all CUR data, project tag mappings, and Amazon Quick Sight dashboards.

To remove all resources:

# Delete Amazon Quick Sight resources
python scripts/setup_quicksight.py --account-id $ACCOUNT_ID --region $REGION --quicksight-user $QS_USER --clean

# Delete CDK stack
cdk destroy

Go to AWS Billing and Cost Management, and then choose Data Exports and delete the CUR 2.0 export created in Step 4.

Deactivate the cost allocation tags that were activated in Step 2:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Inactive"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Inactive"}
]'

Conclusion

In this post, we showed how to build an end-to-end cost reporting solution for Amazon SageMaker Unified Studio projects using custom tags. This solution combines tag capture driven by Amazon EventBridge, CUR enrichment through AWS Glue ETL, and visualization in Amazon Quick Sight. With it, organizations can track and attribute costs by CostCenter, Team, Environment, or any custom dimension. This works even for resources created through the Studio UI that don’t receive custom tag propagation.

This solution serves as an extension to the custom tag propagation feature and reports cost for all project resources. The architecture is fully serverless, automated, and can be deployed to any AWS account using the provided CDK application.

To start building your custom tag cost reporting pipeline, visit the GitHub repository. To learn more about the underlying services, visit the Amazon SageMaker Unified Studio service page. For a related approach to custom tag governance, see Use Amazon SageMaker custom tags for project resource governance and cost tracking

References

About the authors

Set up your AI coding agent to build with AWS Step Functions

2026-08-19 D Surya Sai

Post Syndicated from D Surya Sai original https://aws.amazon.com/blogs/compute/set-up-your-ai-coding-agent-to-build-with-aws-step-functions/

You want to build an AWS Step Functions workflow, and you have an AI coding agent open in your terminal or IDE. But the agent doesn’t know about Amazon States Language (ASL), service integrations, or how to deploy state machines. Before you can start, you need to find the right Model Context Protocol (MCP) server package, figure out the configuration format for your specific agent, and set up credentials.

AWS Step Functions has added a “Copy agent prompt” button to the AWS Step Functions console that removes this setup entirely. You choose the button, paste the prompt into your agent, and the agent configures itself with Serverless skills and an MCP server. You can start building workflows with natural language immediately. The feature works with Claude Code, Kiro CLI, Cursor, GitHub Copilot, Codex, Devin Desktop, OpenCode, and any other MCP-compatible agent.

How it works

The button appears in three places in the Step Functions console:

The home page, under “How it works”.
The Create State Machine modal (at the top, before you begin building).
The Local Development section on the home page.

Here’s an example from the Create State Machine flow:

Open the Step Functions console and choose Create state machine.
At the top of the modal, you see the banner: “Set up your agent to build with Step Functions. Copy and paste this prompt into your AI agent to set up Step Functions skills and MCP server.”

Step Functions console modal showing the Copy agent prompt banner and button

Figure 1: Step Functions console modal showing the Copy agent prompt

Choose Copy agent prompt. The console copies a fetch instruction to your clipboard.
Paste the prompt into your AI agent’s chat or terminal.
The agent reads the setup guide and self-configures.

The copied prompt is a fetch instruction that points to a setup guide hosted on AWS documentation. You paste it into your agent, and the agent installs two things:

AWS Serverless skill (from the Agent Toolkit for AWS) provides your agent with deep context on Step Functions. It includes how to write ASL, structure workflows with retries and error handling, choose between Standard and Express workflow types, implement patterns like saga orchestration and parallel fan-out, and deploy using AWS Serverless Application Model (AWS SAM) or AWS Cloud Development Kit (AWS CDK).

AWS Serverless MCP Server gives your agent direct access to AWS. Through the Model Context Protocol, your agent can create and update state machines, start and describe executions, inspect workflow history, and manage resources in your account.

Supported agents

The setup guide auto-detects your agent and provides the correct configuration format:

Claude Code: Installs through the plugin marketplace and registers the MCP server with claude mcp add.
Kiro CLI: Writes to ~/.kiro/settings/mcp.json.
Codex: Registers with codex mcp add.
Cursor: Writes to .cursor/mcp.json.
GitHub Copilot: Writes to .vscode/mcp.json.
Devin Desktop: Writes to .devin/mcp_config.json.
OpenCode: Writes to ~/.config/opencode/opencode.jsonc.

If you use a different MCP-compatible agent, the guide provides a generic JSON configuration block you can add to your agent’s config file.

What you can build

Once your agent is configured, you can describe workflows in natural language, and the agent produces valid, deployable state machines. Here are a few examples:

Order processing with compensation: “Build a workflow that validates a payment, reserves inventory and sends a confirmation email. If payment fails, release the inventory reservation.”

Parallel fan-out: “Create an Express workflow that calls three AWS Lambda functions in parallel, waits for all to complete, and merges the results into a single response.”

Human approval gate: “Add a step that pauses the workflow and waits for a manager to approve before proceeding with the deployment.”

Error handling: “Add retry with exponential backoff and a maximum of three attempts to the payment processing step. If all retries fail, route to a fallback notification step.”

Because the agent has the MCP server connected, it can also deploy the workflow directly to your account, start test executions, and inspect the results without leaving the agent interface.

Advantages

Always current: The Agent Toolkit for AWS content stays up to date as Step Functions adds new features, integrations, and patterns. When you run the prompt, your agent gets the latest skills and configurations automatically.

No context switching: You stay in your agent’s interface for the entire workflow: design, build, deploy, test, and iterate. No switching between the console, documentation, and your editor.

Works with your existing credentials: The MCP server uses your local AWS profile. No new AWS Identity and Access Management (IAM) roles or permissions are required beyond what you already use for Step Functions development.

Agent-agnostic: Whether you use Claude Code, Kiro, Cursor, Copilot, or another tool, the same button and prompt works. You don’t need to find agent-specific setup instructions.

Get started

Open the AWS Step Functions console.
Choose Copy agent prompt from the banner (on the home page under “How it works,” in the Local Development section, or in the Create State Machine modal).
Paste the prompt into your AI coding agent.
Start describing the workflow you want to build.

This feature is available in all commercial AWS Regions at no additional cost. To learn more about the setup process, see the agent setup guide. For more on the Agent Toolkit for AWS, see the GitHub repository. For AWS MCP Servers, see the documentation.

We’d like to hear how you use this feature. Tell us about it in the comments.

Powering agentic AI with real-time streaming data on AWS

2026-08-18 Mazrim Mehrtens

Post Syndicated from Mazrim Mehrtens original https://aws.amazon.com/blogs/big-data/powering-agentic-ai-with-real-time-streaming-data-on-aws/

Two years ago, the conversation about streaming data and generative AI centered on a straightforward question: how do you feed real-time context into a large language model (LLM) so it can answer questions using fresh data? We explored that question in our 2024 blog post, “Exploring real-time streaming for generative AI applications,” which introduced patterns for connecting streaming pipelines to foundation models.

The landscape has shifted. Today’s generative AI systems don’t only answer questions. They observe, reason, and act. Agentic AI applications have moved from research prototype to production reality. Agentic AI-powered data pipelines now monitor streaming telemetry, detect anomalies, decide on remediation strategies, and execute actions without human intervention. They maintain memory across sessions, query live data sources on demand, and coordinate with other agents to solve complex problems.

This shift demands a fundamentally different relationship between streaming infrastructure and AI. It’s no longer enough to inject context into a prompt. You need architectures where streaming data continuously powers autonomous agent action and keeps a real-time lakehouse fresh for training and retrieval. That data also flows into multiple consumption patterns, such as generative business intelligence (BI) for humans, standardized protocols for agent queries, and proactive memory hydration for low-latency agent context.

This post introduces three architectural patterns that together form a unified streaming backbone for the agentic AI era:

Streaming feature engineering → real-time inference → action: Continuous data flows build features, invoke AI models, and act in a single pipeline.
Event-driven agent invocation: Streaming pipelines detect patterns across millions of events and trigger agentic workflows with full context already assembled.
Real-time context synchronization: Change data capture (CDC) and streaming pipelines keep agents’ memory current, so agents can respond instantly rather than making expensive external calls.

The following sections explore each pattern in depth.

Pattern 1: Streaming feature engineering → real-time inference → action

You’re watching a live football match. As a striker receives the ball in the box, AI-generated commentary appears on screen: “This is Smith’s third touch in the penalty area in the last 3 minutes. His conversion rate from this zone is 34% this season.” That insight was computed from streaming event data, passed through a feature pipeline, and fed to a generative AI model. All of this happened within the time it takes the striker to turn and shoot.

This pattern combines two capabilities that are often treated separately: using real-time data to continuously improve AI models, and using real-time data to invoke those models for immediate action. The streaming pipeline does both: it builds the features that train the model and the features that drive inference.

Streaming events (user interactions, sensor readings, game events, and transaction records) flow into Amazon Managed Streaming for Apache Kafka (Amazon MSK) or Amazon Kinesis Data Streams. Amazon Managed Service for Apache Flink processes these events through windowed aggregations (tumbling windows, sliding windows, or session windows) to produce features: rolling averages, counts, ratios, behavioral sequences, or other derived signals relevant to your use case.

These features serve two paths simultaneously:

The inference path: At the end of each window (or on each event, depending on your latency requirements), features are passed to a generative AI or machine learning (ML) inference endpoint: Amazon Bedrock for generative output, or Amazon SageMaker for custom models. The model produces a result (commentary, a recommendation, a personalization decision, or a risk score) and the pipeline acts: posting content to a user, updating a recommendation feed, sending a notification, or writing to a downstream system.

The training path: The same streaming features are continuously written to a real-time data warehouse or lakehouse such as Apache Iceberg tables on Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), that keeps training datasets fresh. Amazon SageMaker lakehouse architecture provides unified access for training jobs and fine-tuning pipelines. As new data streams in, your models can be retrained or fine-tuned on data that’s minutes old rather than days old. This matters for domains where patterns shift quickly, such as fraud detection, personalization, and industry dynamics.

Amazon S3 Tables handles the Iceberg table management automatically, including compaction, snapshot management, and metadata optimization. Your team focuses on feature logic rather than storage operations. The AWS Glue Data Catalog makes these tables discoverable across training jobs, inference pipelines, and analytics consumers. Glue Data Catalog supports business context and semantic search. This context helps models discover and select the right data asset for any given task.

Scenarios

Real-time sports commentary: Streaming game events (passes, shots, player positions) flow through Apache Flink on Managed Service for Apache Flink, which computes rolling features (possession percentage, shot frequency by zone, player heat maps). These features feed a generative AI model through Amazon Bedrock that produces natural-language commentary and statistical insights in real time. Simultaneously, the features are written to S3 Tables to improve the model’s understanding of game patterns over time.

Streaming personalization: User clickstream data flows through Managed Service for Apache Flink, which computes behavioral features (session duration, category affinity scores, recency-weighted purchase history). These features invoke a personalization model that updates the user’s experience in real time by reranking product recommendations, adjusting content feeds, or triggering targeted offers. The same features feed the lakehouse to retrain the personalization model nightly.

Streaming data flows through Managed Service for Apache Flink, then forks into a real-time inference path and a training path

Figure 1: Streaming feature engineering feeding a real-time inference path and a continuous training path

Pattern 2: Event-driven agent invocation

At 2:47 AM, a pressure sensor on a manufacturing line begins drifting. Within seconds, a streaming pipeline detects the anomaly, assembles full context (device history, maintenance schedule, correlated sensor readings), and invokes an agent that opens a maintenance work order, adjusts the device’s sampling rate, and notifies the on-call engineer. All of this happens before a human sees an alert.

Pattern 1 invokes inference on every window or event. It runs continuously. Pattern 2 adds to this approach: the streaming pipeline continuously analyzes data and invokes an agentic workflow when specific conditions are met or a pattern is detected. The pipeline is the sensor. The agent is the responder. Dynamic rules are the bridge between them.

The key distinction is that the events and triggers are dynamic. They’re defined by rules programmed into the streaming pipeline or traditional ML models for prediction or detection. The pipeline determines when and how the agent is triggered, making the system fluid and adaptive. You can update detection logic without redeploying the agent. You can add new anomaly patterns without changing the response logic.

Streaming telemetry flows into Amazon MSK or Amazon Kinesis Data Streams. Managed Service for Apache Flink runs continuous anomaly-detection logic, such as statistical models, windowed aggregations, threshold-based rules, or ML-based scoring. Critically, when Flink detects an anomaly, it doesn’t only publish a raw alert. It assembles a context package: the anomaly details, relevant historical data, correlated signals from other streams, and metadata the agent needs to act immediately.

This context package is published to a downstream topic and consumed by an Amazon Bedrock AgentCore agent. Because the pipeline has already assembled full context, the agent doesn’t waste time gathering information. It can reason and act immediately. AgentCore Runtime hosts the agent, AgentCore Observability provides tracing and logging, and AgentCore Memory maintains state across invocations (so the agent knows, for example, that this is the third anomaly from this device this week).

The benefit of this pattern over a polling-based or scheduled approach is twofold:

Latency: The agent is invoked within seconds of the anomaly, not at the next polling interval.
Context richness: The pipeline has already done the work of correlating signals and assembling context. A polling-based agent would need to make multiple queries to reconstruct what the pipeline already knows.

The rules that trigger invocation are a powerful abstraction. They can be simple thresholds (“temperature exceeds 95°C”), statistical (“value deviates more than 3σ from the rolling mean”), or ML-based (“anomaly score from an embedded model exceeds 0.85”). You can update these rules dynamically by adding new detection patterns, adjusting sensitivity, or routing different anomaly types to different agents.

Managed Service for Apache Flink detects anomalies and sends a context package to an Amazon Bedrock AgentCore agent that acts on them

Figure 2: Event-driven agent invocation triggered by anomaly detection in the streaming pipeline

Pattern 3: Real-time agent context

A customer messages their bank: “Was that $847 charge at the airport legitimate?” The agent responds in under two seconds with full context (the customer’s recent travel pattern, the merchant’s fraud-risk score, and the transaction details) because all of this was already loaded into the agent’s context layer through streaming CDC. A reactive agent without this synchronization would need to make five separate API calls across three systems, taking 8–12 seconds and risking timeout failures.

This pattern addresses a fundamental question: how proactive should your agent be about gathering context?

A proactive agent has the full context, continuously synchronized with the state of the world. When a user asks a question, the agent already has the relevant knowledge from context. It responds from memory rather than making expensive external calls. A reactive agent starts cold. It knows nothing until it queries for information, making multiple calls across security boundaries, handling authentication, and stitching together data from disparate sources. For latency-sensitive use cases, where a user sends a prompt and expects a fast response, this difference is critical.

Real-time context synchronization uses CDC and streaming pipelines to keep agent memory current. The agent’s knowledge graph becomes a synchronized replica of the distributed systems it needs to reason about.

No agent is purely proactive or purely reactive. The design decision is: what data should be pre-loaded, and what should be fetched on demand? This is a spectrum, and where you land depends on three factors:

Latency sensitivity: If users expect fast, contextually relevant responses, pre-load the data the agent needs most frequently.
Data volume: Synchronizing everything is impractical. An efficient, fast search that still produces accurate results matters more than exhaustive pre-loading. Be selective about what you push.
Data freshness requirements: Some data changes every second (stock prices, session state). Other data changes rarely (customer preferences, account configuration). Load what changes frequently and matters immediately.

Streaming pipelines (Managed Flink reading from Amazon MSK, Kinesis Data Streams, or CDC streams from operational databases) continuously process events and write aggregated results to the agent’s knowledge graph, or the context layer. These stores can take multiple forms depending on your access patterns:

AWS Context automatically maps relationships across your existing data into a knowledge graph and supports agentic search so AI agents can access governed data relationships, business rules, and domain knowledge at runtime. Data stewards manage the graph through an intuitive console, reviewing inferred relationships, promoting them to production, and attaching domain-specific knowledge like business definitions and usage rules.
Amazon Bedrock AgentCore Memory for structured agent context that persists across sessions.
Amazon DynamoDB for low-latency key-value lookups (customer profiles, account state).
Amazon OpenSearch Serverless for semantic search over unstructured context (past conversations, documents).
Amazon Neptune for relationship-rich data (knowledge graph).
Amazon S3 Tables fully managed Apache Iceberg tables in Amazon S3, for interoperability between multiple query engines.

For data that isn’t pre-loaded, the agent falls back to on-demand retrieval. This applies when the data is too large, changes too rarely to justify streaming, or is needed only in edge cases. The Model Context Protocol (MCP) provides a standardized interface for this. MCP servers expose heterogeneous data sources through a uniform protocol. The agent queries MCP when it needs context that isn’t in its synchronized memory.

This same real-time context synchronization pattern serves different consumers:

AI agents access fresh context through a real-time knowledge graph or a context layer, and MCP servers (pull tier), as in the preceding sections.

Human analysts and executives access the same context layer, which can directly query Apache Iceberg tables on S3 Tables through its direct query mode. Amazon Quick chat provides natural-language access to real-time lakehouse data. No intermediate warehouse is required. This is the generative BI expression of the same underlying pattern: streaming data keeps the lakehouse current, and Amazon Quick gives humans conversational access to it.

Training and fine-tuning pipelines access the synchronized lakehouse through Amazon SageMaker Lakehouse, keeping models fresh (as described in Pattern 1).

The underlying principle is the same across consumers: streaming pipelines synchronize distributed data into accessible stores, and each consumer accesses those stores through the interface that fits their needs.

A streaming synchronization layer feeds multiple stores that serve AI agents, human analysts, and training pipelines

Figure 3: Real-time context synchronization serving agents, analysts, and training pipelines from shared stores

Bringing it together

The three patterns in this post form a unified architecture built on a single streaming backbone:

Pattern 1 uses streaming pipelines to build features that simultaneously drive real-time inference and keep training data fresh. Your models improve continuously while serving predictions in real time.

Pattern 2 uses streaming pipelines as intelligent sensors that detect anomalies and invoke agents with full context already assembled. This separates detection logic from response logic for maximum flexibility.

Pattern 3 uses streaming pipelines to synchronize distributed system state into the agent’s context layer, making agents more proactive and serving multiple consumers (agents, humans, and training jobs) from the same pre-loaded data.

The streaming infrastructure you build (Amazon MSK, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon S3 Tables) serves all three patterns simultaneously. A Flink application can compute features for inference (Pattern 1), detect anomalies that trigger agents (Pattern 2), and synchronize state into agent memory (Pattern 3).

To get hands on with the patterns described in this post, refer to Agentic AI-Powered anomaly detection: Spotting anomalies in real-time.

You don’t need to implement all three patterns at once. Start with the one that addresses your most pressing need. But design your streaming infrastructure knowing it will serve multiple patterns. In the agentic AI era, every stream is a potential input to an agent, a model, and a human decision-maker.

About the authors

Updates to your AWS Sign-In experience

2026-08-17 Vaibhav Chowla

Post Syndicated from Vaibhav Chowla original https://aws.amazon.com/blogs/security/updates-to-your-aws-sign-in-experience/

Amazon Web Services (AWS) is gradually introducing updates to the AWS Sign-In and sign-up experience to a limited number of customers. We’re sharing these changes so you will know what to expect as we gradually make the updated experience available to more customers. These updates include new options for creating and accessing AWS accounts. To support these options and provide a more consistent experience, we’ve redesigned the AWS sign-in page and refreshed the session selection page. While some screens and interactions have changed, existing customers will continue using the same sign-in methods and credentials they use today. If you see a sign-in page that looks different from what you’re used to, this is an expected change.

In this post, we walk through what’s new with screenshots so you’ll know what to expect. If your organization relies on the current sign-in interface for browser automation or scripted workflows, review these updates to understand how they might affect your configuration.

Redesigned Sign-In page

The AWS Sign-In page is getting a new look. Figure 1 shows the current sign-in page, where you choose between Root user and IAM user before entering your sign-in information.

Figure 1: Current AWS sign-in page

The redesigned sign-in page, shown in Figure 2, introduces a unified email entry point for signing in to AWS. Root users and customers using the new email-based sign-in method for AWS accounts created with the updated sign-up experience enter their email address and choose Continue. AWS automatically determines the appropriate sign-in flow based on the email address provided.

If you’re signing in as an IAM user, choose IAM User to continue to the IAM User Sign-In page. Enter your account ID or alias, AWS Identity and Access Management (IAM) username, and password to sign in.

Figure 2: Redesigned AWS sign-in page

The redesigned page also includes sign-in options for customers whose AWS account was created using a supported identity provider, such as Google, GitHub, Apple, or an Amazon.com account. If you’re an existing AWS customer, continue using the same credentials you use today, and AWS will guide you through the appropriate sign-in experience. Sign-in with a supported identity provider is available only for AWS accounts that were created using that identity provider.

If your organization uses AWS IAM Identity Center or IAM federation to access AWS, continue signing in through your organization’s access portal or federation URL. Your existing sign-in process doesn’t change.

Note: Although you don’t need to take specific actions to benefit from these updates, if your setup depends on the current UI for automated tasks, you might notice changes. For the most reliable and stable experience, use the AWS supported options to grant programmatic access to your users. For more information, see the programmatic access options.

Try the redesigned sign-in experience

Before the redesigned sign-in experience becomes the default, AWS will display a banner on the existing sign-in page inviting you to try it, as shown in Figure 3. Selecting Change to new experience takes you to the updated sign-in flow. The existing experience remains available until the redesigned experience becomes the default.

Note: After you select Change to new experience, you’ll continue to see the redesigned sign-in experience in that browser. To return to the existing experience while it’s still available, clear your browser cookies.

Figure 3: Current AWS sign-in page with the banner to try the redesigned experience

Redesigned session selection experience

AWS supports multiple active account and role sessions, so you can stay signed in to more than one account at a time. We’ve redesigned the AWS session selection page with a refreshed look that simplifies viewing and managing your active sessions. When you return to AWS while you have active account or role sessions, the session selection page displays those sessions in one place, including the account, role, and recent sign-in information to help you identify the session you want to use, as shown in Figure 4.

Figure 4: New session selection page

From this page, you can select an existing session, sign out of one or all sessions, or add another AWS session. Choose Add session to sign in to another AWS account while remaining signed in to your existing sessions.

Conclusion

The redesigned AWS Sign-In and session selection pages provide an updated experience while continuing to support the sign-in methods you use today. If you’re an existing AWS customer, there’s no change to how you sign in to your account. We encourage users who rely on browser automation or other workflows that interact with the sign-in experience to review these updates and ensure their systems are compatible with the redesigned experience.

To learn more, see the AWS Sign-In User Guide. If you have questions or feedback, start a new thread in IAM re:Post or reach out to AWS Support.

Serverless vehicle tracking at scale: Bosch L.OS on AWS

2026-08-14 Yogish Kutkunje Pai

Post Syndicated from Yogish Kutkunje Pai original https://aws.amazon.com/blogs/architecture/serverless-vehicle-tracking-at-scale-bosch-l-os-on-aws/

When Bosch Mobility Platform Solutions set out to unify vehicle tracking across India’s fragmented spot logistics market, they faced a daunting reality: dozens of telematics providers, incompatible data formats, and thousands of concurrent tracking requests — all needing real-time resolution. The result was L.OS, a serverless platform on AWS that standardizes this chaos into a single visibility layer.

In this post, we’ll show you how Bosch Mobility Platform Solutions (MPS) uses AWS services to solve these challenges through their L.OS solution. You’ll learn how Bosch built a scalable, serverless architecture that standardizes and integrates multiple tracking data sources, so you can achieve real-time visibility and data-driven decision-making across complex logistics networks.

Key challenges in logistics visibility

If you manage a modern supply chain, you face several critical challenges:

Data fragmentation and integration complexity.
- Multiple tracking systems with incompatible data formats.
- Different communication protocols across providers.
- Lack of standardization in data exchange.
- Complex and costly point-to-point integrations.
Operational inefficiencies.
- Manual coordination between stakeholders.
- Time-consuming reconciliation of conflicting information.
- Difficulty in providing accurate ETAs.
- Limited real-time visibility into shipment status.
Scale and performance issues.
- High volume of concurrent tracking requests.
- Variable data quality from different sources.
- Performance bottlenecks during peak operations.
- Cost implications of real-time tracking.
Regional complexities.
- Fragmented spot logistics networks.
- Multiple intermediaries in the supply chain.
- Varying levels of technological adoption.
- Regional compliance requirements (such as AIS140 and FASTag in India).

Introducing L.OS on AWS: A unified visibility solution

To address these challenges, Bosch’s Logistics Operating System (L.OS) on AWS provides a horizontal integration layer that connects previously siloed logistics solutions. The solution features a service catalog where solution providers and consumers can collaborate to solve complex use cases, fostering innovation in the logistics sector. Let’s explore how L.OS enhances vehicle visibility through its core workflows: discovery, tracking, and termination.

Discovery

When a client needs to track a vehicle, the service app makes a discovery call to the L.OS gateway. This call includes essential details such as the vehicle number plate or vehicle identification number (VIN). Upon receiving the request, the L.OS solution performs necessary authentication and authorization. L.OS then broadcasts the request and waits for acknowledgment from one or more connected participants. The responses contain information such as the mode, frequency, and reliability of tracking, which can be used for shortlisting and decision-making.

The following diagram illustrates the discovery workflow, showing how a client’s tracking request flows through L.OS to connected participants and back.

Figure 1 – The discovery flow: the service app sends a discovery call to the L.OS gateway with vehicle identifiers. L.OS broadcasts the request to connected participants, collects acknowledgments containing tracking mode, frequency, and reliability details, and returns them to the consumer for shortlisting.

Tracking

Once the consumer has selected a vehicle and a service provider (if there are multiple options), a request is sent to the L.OS to initiate tracking. This request is relayed to the specific service provider. The tracking mode determines who must grant consent. For SIM tracking, a consent request goes to the driver. For GPS tracking, it goes to the fleet owner. The solution waits for the tracking provider to create the trip. Upon receiving confirmation, L.OS registers the tracking request and provides a unique tracking ID to indicate that tracking has been initiated. From here, the consumer is asynchronously notified of the vehicle’s location at the specified frequency, or the maximum frequency supported by the service provider, whichever is faster. Consumers can also request the live location of the vehicle at any time between the regular reporting intervals.

The following diagram shows the tracking workflow, from initiation through consent, trip creation, and ongoing location updates.

Tracking workflow diagram showing initiation, consent, trip creation, and location updates

Figure 2 – The tracking flow: the consumer sends a tracking request to L.OS, which relays it to the selected service provider. A consent request is issued (to the driver for SIM tracking, or the fleet owner for GPS tracking). Once the provider confirms trip creation, L.OS returns a unique tracking ID and begins delivering asynchronous location updates at the agreed frequency.

Termination

The tracking is automatically terminated when the vehicle enters the destination geo-fence. Alternatively, tracking can be terminated manually by sending an explicit request to L.OS, which is then relayed to the service provider.

Architecture overview

The L.OS solution built on AWS uses various services to create a scalable, secure, and maintainable system. The architecture implements serverless components (AWS Lambda adapters) where appropriate while using containers (Amazon Elastic Container Service (Amazon ECS) with AWS Fargate) for the core connector service. Let’s explore how these AWS managed services work together to create a flexible and scalable integration solution. The following diagram shows the end-to-end architecture, illustrating how requests flow from client applications through the API layer, into the core connector service, and out to individual tracking providers.

L.OS end-to-end architecture on AWS showing client applications, API Gateway, ECS Fargate connector, Lambda adapters, and Amazon MSK

Figure 3 – L.OS architecture on AWS: Client applications connect through Amazon API Gateway to the Tracking Connector running on Amazon ECS Fargate, which handles protocol standardization, routing, and session management. Provider-specific Lambda adapters translate between the standardized connector API and each tracking provider’s API. Amazon MSK serves as the event bus for asynchronous location updates. Amazon ElastiCache provides low-latency caching for frequently accessed data, Amazon DynamoDB stores business rules and security policies, and the Marketplace Subscription Management service (also on Fargate) handles authentication, customer relationships, and provider configurations. Amazon QuickSight delivers real-time monitoring and usage analytics.

Key components

The architecture comprises five core components that work together to deliver reliable, real-time vehicle tracking at scale. Each component handles a distinct responsibility — from protocol translation to event streaming — allowing the system to scale and evolve independently.

Centralized orchestration with Amazon ECS Fargate

The Tracking Connector, running on Amazon ECS Fargate, serves as the central orchestration layer. It handles critical functions including:

Protocol standardization across multiple providers.
Intelligent request routing.
Response aggregation.
Session management.
Comprehensive error handling.
Performance optimization using Amazon ElastiCache.

Serverless provider integration

We use AWS Lambda to implement Tracking Adapters that handle provider-specific transformations. These adapters efficiently translate between our standardized connector API and various provider APIs, allowing for easy onboarding of new providers.

Event-driven communication

Amazon MSK (Managed Streaming for Apache Kafka) powers our message bus, enabling:

Standardized topic patterns.
Support for multiple domain connectors.
Real-time data streaming for tracking, parking, vehicle health, charging, and fleet management.

Subscription and access management

The Marketplace Subscription Management service, deployed on Amazon ECS Fargate, manages:

Customer relationships.
Service consumer configurations.
Provider integrations.
Authentication and authorization token claims.

Policy and security enforcement

We use Amazon DynamoDB to store and manage:

Business rules.
Security policies.
Authorization configurations.
Routing rules.

Monitoring and analytics

Amazon QuickSight provides:

Real-time system performance metrics.
Usage analytics.
Health monitoring.
Anomaly detection.

Benefits

By implementing this serverless architecture on AWS, Bosch L.OS achieved significant improvements in vehicle tracking capabilities:

Operational efficiency

The combination of standardized Lambda adapters and the centralized Tracking Connector on ECS Fargate eliminates the manual coordination that previously slowed provider onboarding. Where ISVs once spent 2–4 weeks on bespoke integration work for each new customer request, the standardized connector API and adapter pattern reduces this to within 3 days. Real-time data validation at the connector layer — before events reach downstream consumers — also improves data accuracy by catching format inconsistencies at ingestion rather than during reconciliation.

Scalability and performance

Because the core connector runs on Fargate with auto-scaling task definitions, and each provider adapter is an independent Lambda function, the system scales horizontally without manual intervention. Bosch’s deployment currently handles 35,000 trips per day — each generating multiple location events — with sub-second response times for 99.9% of tracking queries. As new ISVs are onboarded, additional Lambda adapters are deployed independently, so scaling the provider network does not add load to existing integrations.

Cost optimization

Integrations in fragmented logistics markets often stall because multiple vendors must coordinate through manual processes — handoffs, SIM card provisioning, consent management, and troubleshooting. By automating these workflows within the L.OS connector layer and MSK event bus, Bosch estimates integration costs are reduced by 15–20%. The architecture also removes per-vendor overhead (SIM management, consent flows, provider-specific troubleshooting) that was previously passed on to small transporters. This potentially lowers their total tracking costs by 25–30%.

Enhanced customer experience

The unified API Gateway endpoint and MSK-powered event streaming mean consumers receive location updates from any connected provider through a single interface — regardless of the underlying tracking technology. What previously required hours of manual coordination across providers now surfaces as a consolidated event within approximately 1 minute, according to Bosch. Improved ETA accuracy is a direct result: with standardized, high-frequency location data flowing through ElastiCache, downstream planning systems can compute more reliable arrival predictions.

Compliance and security

DynamoDB-backed policy enforcement ensures that business rules, authorization configurations, and regional compliance requirements (such as India’s AIS140 and FASTag mandates) are evaluated consistently on every request. The built-in security features of AWS — IAM roles, virtual private cloud (VPC) isolation, and encryption at rest and in transit — provide the baseline. Automated audit trails captured through the event bus give organizations a verifiable record of all tracking operations.

L.OS growth

L.OS is currently operational in India with 10 integrated ISVs. The serverless adapter pattern makes geographic expansion straightforward: new region-specific adapters can be deployed as independent Lambda functions without modifying the core connector. Bosch plans to use this approach to expand into Europe for trailer monitoring use cases.

Conclusion

In this post, we showed how Bosch built L.OS, a serverless vehicle tracking platform on AWS that unifies fragmented logistics visibility into a single integration layer. By using AWS services such as Amazon ECS with Fargate for centralized orchestration and AWS Lambda for provider-specific adapters, the architecture standardizes multiple tracking providers into a unified API.

This standardization eliminates the need for maintaining multiple point-to-point integrations, freeing you to focus on core operations instead of managing repetitive integration tasks. Through strategic collaboration with key stakeholders in the visibility solutions space, L.OS is helping businesses achieve measurable outcomes: enhanced customer experience, increased operational agility, reduced operational expenses, and improved profit margins.

What started as a vehicle tracking solution is now evolving into a broader mobility services portfolio, powered by the scalable infrastructure that AWS provides. This evolution positions L.OS to address not only today’s tracking needs, but a broader range of logistics use cases as they emerge.

If you have questions or feedback about this post, leave a comment in the comments section.

To learn more about the AWS services used in this solution, explore these resources:
- Explore the sample code for this solution on GitHub — includes Infrastructure as Code templates, Lambda function source code, and step-by-step deployment instructions.
- Building real-time fleet tracking applications on AWS
- Best practices for container-based applications using Amazon ECS
- Implementing event streaming with Amazon MSK

For more information about the Bosch L.OS solution and its capabilities, visit Bosch L.OS website.

Contact your AWS account team to learn how we can help you build similar solutions for your logistics operations.

About the authors

Designing for failure: Building resilient systems on AWS

2026-08-13 Dhvani Vora

Post Syndicated from Dhvani Vora original https://aws.amazon.com/blogs/compute/designing-for-failure-building-resilient-systems-on-aws/

In cloud computing, failure in distributed systems isn’t a matter of if, but when. Modern applications span servers, Availability Zones, and Regions. Each component represents a potential point of failure. Resilient applications engineer fault tolerance into their architecture, building systems that self-recover and maintain availability. This post is written for engineers and architects who run distributed data systems such as Apache Cassandra, Apache Kafka, or HDFS on Amazon Elastic Compute Cloud (Amazon EC2) and want to build resilience against hardware failure.

We were working with a customer during one such incident and wanted to share the example. The customer runs a web application that uses Cassandra as its data store, handling both read-heavy and write-heavy workloads at a scale of millions of queries per day.

The 2 AM wake-up call nobody wants

Consider a platform that monitors millions of enterprise network devices across hospitals, universities, and airports worldwide. It detects problems before IT teams even notice them. For that platform, a 2 AM page is more than inconvenient. When your value proposition is catching failures before anyone else does, being caught off-guard by your own infrastructure failure is existential.

The engineering team was deep in quarterly planning when their monitoring dashboard lit up. Three Cassandra nodes had gone dark simultaneously. This was not a graceful shutdown or a rolling restart. It was a hard failure with no warning.

Their architecture is typical of high-scale telemetry platforms. Kafka-powered microservices ingest device telemetry, Apache Flink handles real-time anomaly detection, and Apache Airflow orchestrates batch analytics and firmware updates. All of these rely on Apache Cassandra as the distributed database backbone. The database stores billions of daily writes and handles millions of queries per day.

What actually happened

Three i4i.4xlarge instances running Cassandra nodes failed simultaneously in the SFO region. Investigation revealed that all three instances were colocated on the same physical host. That host suffered a hardware failure, taking all three instances offline at once.

Engineers spent ninety minutes digging through system logs trying to determine the root cause. The root cause was architectural. The deployment lacked Partition Placement Groups, creating a single point of failure where logical replication was undermined by physical collocation.

The good news: Cassandra maintained service availability with no data loss thanks to its replication factor. The bad news: for over an hour, the system ran on a thin safety margin. One more node failure in the same replication group would have caused data unavailability for a subset of queries. That is real customer impact for a platform that promises always-on monitoring.

This is the insidious nature of correlated failures. Individual node failures are expected and designed for. That is the whole point of replication. But when your replicas share physical infrastructure, replication becomes a paper guarantee. You have three copies of the data, but they all live on the same machine.

Making matters worse, their monitoring tools completely missed the initial failure. System status checks correctly flagged the host-level problem. But without Amazon CloudWatch alarms configured to act on those checks, detection was entirely reactive. The team found out because other things started behaving oddly, not because an alarm told them three nodes were down.

Hardware fails. You can’t fix it with a patch or configuration change. The real questions are how fast you detect it, how well your system handles it, and whether failures are correlated.

How the team responded and what they changed

The operations team manually replaced two failed instances with new ones on healthy hardware and restarted the third for log collection. Once replacement instances came online, new Cassandra nodes automatically rejoined their clusters and streamed data from surviving replicas. This process took several hours depending on data volume. Only after full synchronization did the clusters return to full redundancy.

The team recognized that this ninety-minute manual scramble wouldn’t scale. Similar problems had happened before, and each time they followed the same reactive pattern: page, investigate, manually replace, wait for streaming, breathe. Here’s what they implemented to break that cycle, and what you should implement too.

Two-track incident timeline. The top track, labeled Before automation: about 90 plus minutes of manual response, shows five milestones: at 0 minutes three nodes fail simultaneously. At about 5 minutes cascading errors are noticed with no alarm. At 90 minutes the root cause is found in system logs. At 90-plus minutes instances are manually replaced. And after several hours data streaming completes and full redundancy is restored. The bottom track, labeled After automation: under 5 minutes to recovery, shows four milestones: at 0 seconds the system status check fails. At about 60 seconds a composite alarm fires. At about 2 minutes Auto Scaling replaces the node. And in under 5 minutes a lifecycle hook rejoins the node to the cluster.

Figure 1: The same failure handled two ways. Manual response took over 90 minutes plus hours of streaming. The automated path completes recovery in under 5 minutes.

1. Use Partition Placement Groups to isolate failure domains

The three crashed servers shared a physical machine because no one told AWS otherwise. Without placement group constraints, instances are placed based on available capacity. That can mean multiple instances land on the same host. For stateless web servers, this rarely matters. For distributed databases whose entire resilience model depends on replicas being independent, it’s a silent architecture bug waiting to become a 2 AM incident.

Partition Placement Groups fix this by distributing instances across separate hardware racks. Each partition maps to a distinct set of physical infrastructure, with separate power and separate network switches. When one rack fails, it affects only the instances in that partition.

Diagram comparing two Cassandra deployments. On the left, labeled Before, all three Cassandra nodes run on a single physical host, so a host failure takes down all three replicas. On the right, labeled After, the three nodes are distributed across three Partition Placement Group partitions on separate racks (Rack A, Rack B, Rack C). When Rack B fails, only Node 2 is lost and the cluster survives.

Figure 2: Distributing Cassandra replicas across Partition Placement Group partitions so a single rack failure affects only one node.

You can create up to seven partitions per Availability Zone, with as many instances as needed in each. By mapping Cassandra replicas to separate partitions, a single hardware failure takes down one node instead of three. This applies to any distributed system that maintains replicas, such as Kafka, HDFS, or Cassandra.

Key insight: Align your Partition Placement Group partitions with your application’s replication topology. If Cassandra uses a replication factor of 3, place each replica in a different partition. This means the physical isolation boundary matches the logical replication boundary.

CLI example:

aws ec2 create-placement-group \
  --group-name cassandra-partitioned \
  --strategy partition \
  --partition-count 3

aws ec2 run-instances \
  --placement "GroupName=cassandra-partitioned,PartitionNumber=1" \
  --instance-type i4i.4xlarge \
  --image-id ami-xxxxxxxx

Partition Placement Groups (up to 7 partitions per AZ, unlimited instances per partition) are designed for large distributed workloads. Spread Placement Groups (max 7 instances per AZ, each on a separate rack) suit small critical clusters. For a Cassandra deployment at scale, Partition is the right choice. Learn more in the Amazon EC2 placement groups documentation.

2. Monitor system status checks and use composite alarms

The Cassandra team’s monitoring blind spot came down to a distinction many teams overlook. AWS runs two health checks on every instance: instance status checks (your guest OS and software) and system status checks (the physical hardware underneath). When a system status check fails, the problem is below your control. This includes a host crash, a power failure, or network loss at the rack level. No amount of SSH-ing will help, because the box is unreachable.

The Cassandra team had no Amazon CloudWatch alarms configured on either check type. That meant the only signal was cascading application errors noticed by engineers who happened to be awake. Set these up on day one, before your first production deployment.

To avoid false alarms during normal reboots, where metrics may briefly go missing, combine system status checks with application-level health monitoring using composite alarms. When both fail together, you know there’s a real problem. See the CloudWatch composite alarms documentation for setup details.

3. Automate instance recovery and replacement

The Cassandra team’s ninety-minute recovery wasn’t slow because the engineers were incompetent. It was slow because humans were in the loop. Waking up, assessing, deciding, acting, and verifying: each step adds minutes that compound under pressure. Auto Scaling groups remove the human from the critical path.

Place your Cassandra nodes in an Auto Scaling group. Auto Scaling continuously runs health checks on every instance, and when it marks an instance unhealthy, it terminates it and launches a replacement on different physical hardware, automatically placed within your Partition Placement Group. Under normal conditions, an instance whose system status checks fail is replaced within a few minutes.

The gap to close is detection, not replacement. Rather than waiting for Auto Scaling to reach its own conclusion, have the composite alarm from the previous section explicitly tell Auto Scaling the instance is unhealthy by calling the SetInstanceHealth API. As soon as your combined signal (system status check plus application-level check) confirms a real failure, mark the instance unhealthy and let Auto Scaling replace it immediately. This sidesteps any ambiguity in detection and starts recovery in seconds rather than minutes.

For stateless services, this is enough. For stateful systems like Cassandra, you need an additional step. Lifecycle hooks pause new instances before they join the cluster. A raw Amazon EC2 instance isn’t a functioning Cassandra node. It needs to join the ring, stream data from peers, and verify consistency before serving traffic. Read more in the Amazon EC2 Auto Scaling lifecycle hooks documentation.

In this customer’s case, automating these steps cut recovery time from ninety minutes of manual intervention to under five minutes of automated recovery.

A note on stateful recovery: automated replacement only handles the infrastructure layer. For Cassandra specifically, the new node still needs to stream data from peers before it’s fully operational. The key improvement isn’t eliminating that streaming time. It’s eliminating the human response time before streaming even begins.

4. Build automated incident response with AWS Systems Manager

When servers fail, you face competing priorities. You need to replace them fast to restore capacity, and you need to preserve logs for root cause analysis. These goals conflict when done manually. The Cassandra team restarted one failed node solely to collect diagnostic data before replacing it, adding time to an already long recovery.

AWS Systems Manager runbooks automate this tradeoff away. Build a workflow that runs these steps in sequence:

Isolate the failed instance by detaching it from the load balancer target group.
Create an Amazon EBS snapshot and capture available logs to Amazon S3.
Terminate the instance so that Auto Scaling can replace it.
Notify the on-call channel with the instance ID, failure type, and Amazon S3 log location.

A subtle but important detail: when the instance’s lifecycle is managed by an Auto Scaling group, let the group replace it. Terminating the instance directly only delays recovery, because the group first has to notice the instance is gone before it launches a replacement. Instead, call the TerminateInstanceInAutoScalingGroup API. This tells EC2 Auto Scaling to terminate the unhealthy instance and immediately launch a replacement in one coordinated action. Trigger this runbook automatically with Amazon EventBridge rules that match Amazon EC2 state-change events. The result is that forensic data is preserved, replacement happens in parallel, and the on-call engineer gets a notification after the system has already healed, rather than a page asking them to start fixing it.

Five-step automated recovery workflow shown left to right. Step 1: the Amazon EC2 system status check fails on the host. Step 2: an Amazon CloudWatch composite alarm triggers. Step 3: Auto Scaling terminates the unhealthy node and launches a replacement. Step 4: an AWS Systems Manager runbook takes a snapshot and sends logs to Amazon S3. Step 5: a lifecycle hook streams data, verifies, and rejoins the node to the cluster. The whole flow is triggered by Amazon EventBridge and reduces recovery from about 90 minutes of manual work to under 5 minutes.

Figure 3: The automated recovery workflow, from hardware failure detection through node rejoin, orchestrated by Amazon EventBridge, Auto Scaling, and AWS Systems Manager.

5. Invest in observability before you need it

After resolving the Cassandra incident, the team asked a harder question: what else is silently failing? They ran a broader health assessment, and the answer was sobering. Unstable Redis connections were dropping under load. Amazon EBS volumes were running with elevated latency. Application Load Balancer health check intervals were misconfigured. Secondary databases were approaching connection pool exhaustion. Any of these could cause the next outage, and none of them had triggered a single alert.

This is the pattern. Teams invest in monitoring for the system that recently broke while the next failure quietly builds elsewhere. The better approach is treating observability as infrastructure. Deploy it everywhere from day one, not bolted on after the post-mortem.

Deploy the CloudWatch agent for system-level and application-level metrics. Use Amazon CloudWatch Synthetics canaries to continuously test critical user paths such as login, data ingestion, and dashboard rendering. Set up distributed tracing with AWS X-Ray to identify latency bottlenecks across your microservice mesh. The goal isn’t only knowing that services are running. It’s continuously confirming they’re working correctly from the customer’s perspective.

The Cassandra team built what they call their “resilience dashboard.” It’s a single view surfacing Partition Placement Group distribution, replica lag, system status check state, and Auto Scaling group health. When the next incident happens, they won’t be scrambling to figure out what’s broken. They’ll open one dashboard and know immediately whether their defenses are holding.

Placement groups: Quick reference

The team’s outage involved Partition Placement Groups, but Amazon EC2 offers three placement group types. Choosing the wrong one is a common mistake, so here’s how they compare:

Type	Max instances	Isolation level	Best for
Partition	Unlimited (up to 7 partitions per AZ)	Separate racks per partition	Large distributed databases (Cassandra, Kafka, HDFS)
Spread	7 per AZ	Each instance on a separate rack	Small critical clusters needing maximum isolation
Cluster	Unlimited	Same rack (co-located)	HPC, ML training, low-latency workloads

If the Cassandra team had used Spread Placement Groups instead, they would have hit the 7-instance-per-AZ ceiling almost immediately at their scale. Partition Placement Groups gave them isolation and room to grow. For the highest-criticality deployments, combine placement groups with multiple Availability Zones. You get separate racks and separate data centers, protecting against both rack-level failures and zone-wide events like power grid outages.

The bigger picture: Resilience is a practice

Building resilient systems isn’t a one-time project. It’s a practice that evolves with your architecture. Start by assessing your workloads with the AWS Well-Architected Tool to identify single points of failure you might not see day-to-day. Define Service Level Objectives, so your team agrees on what “good enough” looks like. Not every service needs 99.99% availability, but you need to know which ones do.

Then layer your defenses. Placement groups prevent correlated hardware failures, composite alarms detect problems within minutes, and automated recovery fixes common issues without waking anyone up.

Test regularly. Run disaster recovery drills quarterly. Don’t rely only on tabletop exercises. Run actual failovers in pre-production environments. Use AWS Fault Injection Service to simulate hardware failures and zone outages in a controlled way. Hold blameless post-mortems after every incident to understand what broke, why it wasn’t caught earlier, and what you’ll change.

After this incident, the team deployed Partition Placement Groups, configured composite alarms, and automated their response process. The next time hardware fails, and it will, it won’t cause the same damage.

Consider adopting Chaos Engineering as a discipline. The principles of Chaos Engineering encourage teams to proactively inject failures into production-like environments to uncover weaknesses before they cause real outages. AWS Fault Injection Service makes it straightforward to run these experiments safely, with guardrails that automatically stop experiments if impact exceeds defined thresholds.

For related guidance, see the AWS Well-Architected Framework Reliability Pillar and the Amazon EC2 Auto Scaling User Guide. A sample Systems Manager runbook and AWS CloudFormation template for the automated recovery workflow described in this post is available in the AWS Samples GitHub repository.

If you’ve implemented similar resilience patterns or have questions about placement groups and automated recovery, share your experience in the comments.

Key takeaways

Challenge	Solution
Multiple instances on same physical host	Partition Placement Groups
No health notification for sudden failures	Amazon CloudWatch alarms on system status checks
Missing metrics during host reboots	Composite alarms with application-level health checks
Manual, slow incident response	Automated recovery with Auto Scaling and lifecycle hooks
Delayed root cause identification	Systematic triage starting at the infrastructure layer
Reduced redundancy after failure	Auto Scaling groups for automatic replacement
Recurring confidence erosion	Proactive architectural reviews and observability investment

Amazon EC2 provides tools like placement groups, managed services with built-in high availability, and automation frameworks like AWS Systems Manager. Select the right ones for your workload and test them relentlessly. Failure is inevitable. Your readiness determines the outcome.

Track generative AI costs with Amazon Bedrock inference profiles

2026-08-13 Erik Mack

Post Syndicated from Erik Mack original https://aws.amazon.com/blogs/architecture/track-generative-ai-costs-with-amazon-bedrock-inference-profiles/

Tracking generative AI costs is a common challenge when multiple teams share a single foundation model through Amazon Bedrock. Your HR team answers policy questions, Accounting analyzes financial documents with it, and IT troubleshoots infrastructure issues. All three use the same foundation model. But usage shows up as one line item on the bill. As a result, finance can’t charge back each department, set per-team budgets, or identify who’s driving the most spend.

With Amazon Bedrock application inference profiles, you can solve this. An inference profile is a tagged wrapper around a foundation model. You can use it to attribute costs to specific teams or departments. By combining these profiles with AWS cost allocation tags, you can view per-department Amazon Bedrock costs as separate line items in AWS Cost Explorer.

In this post, we show you how to create application inference profiles for three departments and tag them for cost allocation. You also update your application to route invocations through department-specific profiles and view the per-department cost breakdown in Cost Explorer.

Solution overview

The following diagram shows the solution architecture. Users authenticate at the application layer, and the application identifies each user’s department. It then routes the request to that department’s tagged inference profile in Amazon Bedrock. All profiles use the same foundation model. The application calls Amazon Bedrock using a single IAM role, and individual user identities are not passed to AWS. Cost attribution comes from the inference profiles rather than the calling identity. Amazon Bedrock records usage against each profile’s Team tag, and AWS Cost Explorer displays the costs grouped by department.

Architecture diagram showing application routing to three tagged inference profiles pointing to one foundation model, with cost allocation flowing to AWS Cost Explorer

Figure 1 — Solution architecture for per-department cost tracking with application inference profiles

Amazon Bedrock can also attribute inference costs to the IAM principal that makes each call. This works well when each team calls Amazon Bedrock under a distinct IAM identity. In this architecture, a single application serves all departments under one role. Per-caller attribution can’t separate team costs without adding per-user session management. With application inference profiles, you can attribute costs per team by routing each team to a tagged profile.

To track costs per department:

Create an application inference profile for each department, associating each one to the same foundation model.
Tag each profile with a cost allocation tag (for example, Team=HR).
Activate the tag in the AWS Billing and Cost Management console.
Update your application to route invocations through each department’s inference profile Amazon Resource Name (ARN).
View the per-department cost breakdown in Cost Explorer.

You pay the same per-token rate whether you invoke the model directly or through an inference profile – no additional charges for cost attribution.

Create and configure inference profiles for cost tracking

The following sections walk you through creating inference profiles, activating cost allocation tags, updating your application, and viewing costs in Cost Explorer.

Prerequisites

To configure this solution, you need the following:

An AWS account.
Model access enabled for your chosen foundation model in Amazon Bedrock (for instructions, refer to the Amazon Bedrock User Guide).
AWS Identity and Access Management (IAM) permissions including bedrock:CreateInferenceProfile, bedrock:TagResource, bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream, ce:GetCostAndUsage, and ce:UpdateCostAllocationTagsStatus.
Access to the AWS Billing and Cost Management console to activate cost allocation tags and view Cost Explorer. For more information, refer to Managing access permissions for AWS Billing.
Python 3.12 with boto3 1.35.7 or later (for testing invocations).

Estimated time: 30 minutes (plus 24–48 hours for cost data to appear in Cost Explorer).

Estimated cost: Based on invocations at standard model pricing. For more information, refer to Amazon Bedrock Pricing.

Create application inference profiles

Create an application inference profile for each department. Each profile points to the same foundation model but has a unique tag for cost tracking.

To create an application inference profile:

On the Amazon Bedrock console, in the navigation pane, choose Inference profiles.
Choose the Application tab.
Choose Create inference profile.
For Profile name, enter HR.
For Model, select your foundation model (for example, Anthropic Claude).

Note: Model availability varies by Region. Check the Amazon Bedrock model availability documentation for the current list.

To tag the inference profile:

In the Tags section, choose Add tag.
For Key, enter Team.
For Value, enter HR.
Choose Create. The inference profile status changes to Active.
Repeat for Accounting (Tag: Team=Accounting) and IT (Tag: Team=IT).

The following figure shows the create inference profile page with the profile name and tag configured.

Amazon Bedrock console showing the create inference profile page with profile name HR and tag Team=HR configured

Figure 2 — Creating an application inference profile with a department tag

After you create all three profiles, the Application inference profiles list shows the HR, Accounting, and IT profiles, each with a status of Active and its corresponding Team tag. The following figure shows the three inference profiles after creation.

Amazon Bedrock console showing three application inference profiles: HR, Accounting, and IT

Figure 3 — Three application inference profiles, one per department

To provision inference profiles at scale (for example, one per team across dozens of teams), use the AWS::Bedrock::ApplicationInferenceProfile AWS CloudFormation resource instead of creating each profile manually.

Activate the cost allocation tag

After creating the inference profiles, you activate the cost allocation tag so that tagged costs appear in Cost Explorer. In multi-account environments using AWS Organizations, activate the Team cost allocation tag in the management (payer) account. Tagged usage from member accounts then consolidates in Cost Explorer. For more information about cost allocation tags, refer to Using AWS cost allocation tags.

To activate the cost allocation tag:

Open the AWS Billing and Cost Management console.
In the navigation pane, choose Cost allocation tags.
In the search box, enter Team.
Select the Team tag.
Choose Activate.

The tag status changes to Active.

Note: Cost allocation tags are case-sensitive. Team and team are different tags. Tagged costs can take 24–48 hours to appear in Cost Explorer after activation.

Update the application to use inference profiles

To attribute costs to a department, pass the inference profile ARN as the modelId parameter instead of the foundation model ID. The API call remains the same. You only change the ID you pass.

To find an inference profile ARN:

On the Amazon Bedrock console, choose Inference profiles.
Select the profile.
Copy the ARN from the details panel.

The ARN appears in the format arn:aws:bedrock:region:account-id:application-inference-profile/profile-id.

The following example shows how to route invocations based on the user’s department:

import boto3
from botocore.exceptions import ClientError

client = boto3.client('bedrock-runtime', region_name='us-east-1')

# Replace with your actual inference profile ARNs from the Amazon Bedrock console
DEPARTMENT_PROFILES = {
    'HR': 'arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/abc123',
    'Accounting': 'arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/def456',
    'IT': 'arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/ghi789',
}

# Determine the user's department from your application's authentication layer.
# Examples:
# - An OIDC/SAML claim from your app login: token['custom:department']
# - A lookup in your user database: db.get_user_department(user_id)
# - A value stored in the user's session: session['department']
# The application then calls Amazon Bedrock using its own IAM role;
# individual user identities are not passed to AWS.
department = get_department_from_user_session()

if department not in DEPARTMENT_PROFILES:
    raise ValueError(f"Unknown department: {department}")

try:
    response = client.converse(
        modelId=DEPARTMENT_PROFILES[department],
        messages=[{'role': 'user', 'content': [{'text': 'Your prompt here'}]}],
        inferenceConfig={'maxTokens': 300}
    )
except ClientError as e:
    print(f"Error invoking model: {e}")
    raise

The full code is available on the GitHub repo.

When using inference profiles in production, validate user inputs and consider using Amazon Bedrock Guardrails to filter unintended content. API communications with Amazon Bedrock are encrypted in transit using Transport Layer Security (TLS). For more information about data protection, refer to Data protection in Amazon Bedrock.

Configure the IAM policy for Amazon Bedrock access

Because a single application calls Amazon Bedrock on behalf of all departments, it uses one IAM role. The following policy grants that role permission to invoke the department inference profiles and the underlying foundation model:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeDepartmentInferenceProfiles",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": [
        "arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/*",
        "arn:aws:bedrock:us-east-1::foundation-model/<your-in-region-model-id>"
      ]
    }
  ]
}

The wildcard (*) in the application inference profile ARN lets the single application role invoke the department profiles. The foundation model ARN is required because invoking through an inference profile needs permissions on both the profile and the underlying model. The application determines which department each request belongs to and routes it to the matching profile, and cost attribution comes from each profile’s Team tag. To further restrict access, replace the wildcard with the specific ARNs of your profiles.

Replace 111122223333 with your AWS account ID in the preceding policy.

To create the policy:

On the IAM console, choose Policies.
Choose Create policy.
Choose the JSON tab.
Paste the preceding policy.
Choose Next.
For Name, enter BedrockDepartmentAccessPolicy.
Choose Create policy.

The BedrockDepartmentAccessPolicy appears in the policies list.

To attach the policy to a role:

In the navigation pane, choose Roles.
Select the role used by your application.
Choose Add permissions.
Choose Attach policies.
Search for BedrockDepartmentAccessPolicy.
Select BedrockDepartmentAccessPolicy.
Choose Add permissions.

The BedrockDepartmentAccessPolicy appears in the role’s permission list. To add a department later, create another tagged inference profile and map it in your application. With the wildcard policy, no IAM change is needed. If you scoped the policy to specific ARNs, add the new profile’s ARN.

View per-department costs in Cost Explorer

To view the per-department breakdown in Cost Explorer:

Open the Billing and Cost Management console.
In the navigation pane, choose Cost Explorer.
Set the date range to cover the period after you ran invocations.
For Granularity, select Daily or Monthly.
Choose Group by.
Select Tag.
Select Team.

To view exact amounts, scroll down to view the cost breakdown table.

The following figure shows the per-department cost breakdown in Cost Explorer. The bar chart displays a separately-colored segment for each department – HR, Accounting, and IT – with the cost amount for each. The table below the chart lists the exact dollar amount per department for the selected time period.

AWS Cost Explorer showing per-department Bedrock costs grouped by the Team tag

Figure 4 — Per-department Amazon Bedrock costs in Cost Explorer, grouped by the Team tag

After running invocations through each inference profile, verify the following:

Each inference profile shows the correct Team tag in the Amazon Bedrock console.
The Team cost allocation tag is active in the Billing and Cost Management console.
Per-department costs appear in Cost Explorer when you group by the Team tag.

If costs don’t appear after 48 hours, verify that the cost allocation tag is active and that invocations were made through the inference profile ARNs. If invocations fail, confirm that the inference profile status is Active and the IAM role has the required permissions.

Clean up

Inference profiles don’t incur charges on their own. You only pay for model invocations made through them. As a cleanup step, delete the inference profiles you created for this walkthrough to prevent accidental invocations.

Note: Deleting an inference profile immediately affects applications using that profile ARN. Verify that applications are not actively using these profiles before deletion. To recover, recreate the profile — note that it receives a new ARN, so update your application references.

Delete the following resources:

Application inference profiles – For instructions, refer to Delete an application inference profile.
IAM policy – For instructions, refer to Deleting IAM Policies.
Team cost allocation tag – Deactivate it in the Billing and Cost Management console. For instructions, refer to Deactivating cost allocation tags.

Conclusion

In this post, we showed you how to split generative AI costs by team using Amazon Bedrock application inference profiles and cost allocation tags. With this approach, you can see each department’s costs as a separate line item in Cost Explorer.

To add a new department, create another tagged profile. Costs show up as a separate line item.

You can also:

Set per-department spending alerts and control with AWS Budgets.
Detect unusual spending patterns with AWS Cost Anomaly Detection.
Monitor token usage per department with Amazon CloudWatch.
Attribute costs for higher-level Amazon Bedrock features – reference the same tagged profile ARN in the Knowledge Bases (RAG) to extend per-team attribution beyond direct model invocation.

For more background on application inference profiles, refer to Track, allocate, and manage your generative AI cost and usage with Amazon Bedrock.

For more information about inference profiles, refer to the Amazon Bedrock User Guide.

For help implementing this solution, contact your AWS representative.

About the author

Reducing Text2SQL latency with parameterized query templates

2026-08-13 Yury Brukau

Post Syndicated from Yury Brukau original https://aws.amazon.com/blogs/architecture/reducing-text2sql-latency-with-parameterized-query-templates/

If your Text2SQL system takes 25-30 seconds to respond, user engagement drops significantly. For teams scaling beyond pilot projects, this latency gap between a working demo and a production-ready tool is the biggest barrier to adoption. Without caching, every question triggers a Large Language Model (LLM) call to generate SQL, and those calls introduce challenges: unpredictable response times, throttling limits, and token costs that grow linearly with traffic. Parameterized query templates provide an intelligent caching layer that in our production deployment, reduced end-to-end latency by 80% and cut token consumption by over 50%, turning a slow prototype into a responsive production system. In this post, we walk through the architecture behind this approach, covering the implementation details, performance results, and lessons learned from running a Text2SQL system in production.

When you move AI applications from pilot to production, you need solutions that scale under real traffic and perform consistently. Traditional caching strategies, storing expensive computations once and serving them many times, don’t translate directly to generative AI. End users rarely phrase the same question the same way, context varies between sessions, and outputs depend on small input variations. Yet the underlying principle (caching) still holds value. Rather than abandoning caching entirely, the key is finding the right abstraction layer where similar requests can share cached results.

Solution overview

We applied the solution described in the following section to a system where business users query operational databases using natural language. You ask questions like “What were total sales in Q3?” or “Show me top performing products this month?” and the system generates SQL queries, executes them against the database, and returns results in conversational format. The system translates natural language to SQL using Amazon Bedrock foundation models, while AWS Lambda orchestrates the workflow. You can see a basic overview of used architectural components in Diagram 1.

Architecture diagram showing the Text2SQL system with Amazon Bedrock for SQL generation and AWS Lambda for workflow orchestration

Figure 1 — Solution overview architecture

During the initial implementation phase, the approach with generating and executing SQL queries for user questions on the fly worked well. Response quality was high, and users found the interface intuitive. After these positive results, we started looking into scaling the solution for production traffic. Preserving accuracy was the main priority. Experiments with smaller, faster models didn’t provide a good trade-off between query quality and latency reduction. The accuracy degradation wasn’t acceptable for our system.

This led us to explore alternative approaches, and caching naturally came to mind. Caching user question and answer pairs is the most straightforward option, but it has a fundamental limitation: underlying data changes constantly. An answer about Q3 sales cached today becomes incorrect as soon as new transactions are recorded. The cache would need constant invalidation, undermining its purpose.

Caching the SQL query instead solves this problem. A query like:

SELECT SUM(revenue) FROM sales WHERE quarter = 'Q3'

always fetches fresh data when executed, regardless of when it was cached. Structured Query Language (SQL) captures the user’s intent in a structured, deterministic form that remains valid even as data evolves. It also happens to target the most time and token consuming step in the pipeline, since generating SQL queries requires sending full schema context and examples to a frontier model.

Analyzing the generated queries revealed an opportunity to go further. Many queries follow the same structure, different only in their filter values. A question about Q3 sales produces:

SELECT SUM(revenue) FROM sales WHERE quarter = 'Q3'

while Q2 sales produce:

SELECT SUM(revenue) FROM sales WHERE quarter = 'Q2'

The same pattern appeared across product lookups, date ranges, and category filters. This led to the templating approach: instead of caching complete queries, we generalize them into templates with placeholders. A single template now covers an entire family of questions:

SELECT SUM(revenue) FROM sales WHERE quarter='{quarter}'

Flow diagram showing a cache hit path where a user question matches a stored template, fills placeholders with extracted entities, and executes the SQL query directly

Figure 2 — Templated SQL query cache hit

Templating solves the limited reusability of plain user question, but still leaves a challenge: how do you match an incoming question to the right template when users phrase things differently? “Show me Q3 sales” and “What were sales in Q3?” ask for the same data but share few words. Traditional string matching or keyword lookup would miss these connections. We address this by storing each template alongside a vector embedding of its original question. When a new question arrives, we compute its embedding and perform semantic similarity search against the cache. Because embeddings capture meaning rather than surface wording, both phrasings map to the same template with high confidence. If a match is found above a confidence threshold, we extract entities from the question using lightweight named entity recognition, fill the template placeholders, and execute the query directly, bypassing the LLM entirely. In Diagram 2, you can see the flow of a cache hit.

For questions without matching templates, the system falls back to full LLM generation. It then generalizes the newly generated query into a template, pairs it with the question’s embedding, and adds it to the cache. This creates a self-improving system where cache coverage grows organically as more query patterns are encountered.

Walkthrough – Text2SQL pipeline

The following sections describe each step of the template caching pipeline. Each user’s question flows through entity extraction, template retrieval, and SQL query execution. Cache misses trigger full LLM generation, with new queries feeding back into the cache. The following diagram shows the complete flow of a user question through the newly introduced caching layer.

Complete pipeline flow showing entity extraction, template retrieval, template filling, response generation, and the reinforcement loop for cache growth

Figure 3 — Text2SQL pipeline with template caching layer

1. Entity extraction

After a user submits a question, the system performs entity recognition to extract named entities and values. This step considers not only the current question but also conversation history, current date, and user preferences. This context helps resolve ambiguous references like “last month” or “my region”. Using a lightweight model like Amazon Nova 2 Lite or a custom-trained named entity recognition (NER) model, we identify entities such as dates (“Q3 2024”), names (“Product X”), categories (“electronics”), and numeric values (“top 10”). The system stores these extracted entities separately and uses them later to fill out template placeholders.

2. Template retrieval through semantic search

The system converts the user’s question into an embedding vector using the same embedding model used during cache population. This vector queries the template cache through semantic similarity search, returning the closest matching templates above a confidence threshold. The search matches based on the question’s intent and structure rather than exact wording, so “What were Q3 sales?” and “Show me revenue for third quarter” both match the same template despite different phrasing.

It’s important to note that the confidence threshold governs the cache retrieval layer’s precision-recall trade-off. Set it too high and the system rejects valid, differently worded questions, forcing it to build SQL from scratch. Set it too low and loosely related templates slip through, risking confident answers built on the wrong query. The right value is domain-dependent: narrow, well-templated domains tolerate stricter thresholds, while broad or sparsely covered ones need looser ones.

Rather than relying on a single threshold, we suggest monitoring retrievals in production, logging matched templates and their similarity scores, so we can see when valid questions are being rejected or unrelated templates are slipping through. When embedding similarity alone doesn’t give enough precision, we added a lightweight reranking step: first we retrieve a broader set of candidate templates with a looser threshold, then re-score them with a small LLM or a specialized reranker model to select the best match. This improves precision without sacrificing recall and still costs far less than generating SQL from scratch.

3. Template filling and query execution

When a matching template is found, the system maps extracted entities to template placeholders. If the template contains `{quarter}` and entity recognition extracted “Q3”, the system replaces the placeholder with the actual value. The system validates the filled SQL query for syntax correctness, then executes it directly against the database. This path bypasses the time and token intensive LLM call that generates the SQL query.

This design helps the system to protect against SQL injection on two levels. First, it validates each extracted entity against the expected format for its placeholder: a `{quarter}` must match a known set of values, a `{date}` must parse as a valid date, a numeric threshold must be a number. The system rejects values that do not pass validation before they ever reach the query. Second, the system fills the placeholders using parameterized database queries (prepared statements) rather than string interpolation, so the parameterized query mechanism treats entity values as data rather than executable SQL. This approach also catches entity-extraction errors, improving answer reliability beyond the security benefit.

For richer responses, the system can retrieve multiple top-K similar templates and execute them in parallel. This provides additional context and related information beyond the primary query, for example returning both: quarterly sales totals and a breakdown by product category. The parallel execution adds minimal latency while delivering more comprehensive answers.

4. Response generation and validation

After executing the query, the system sends results to a response generation model. This model has two jobs, both handled in a single call: judge whether the results answer the question, and, if they do, summarize them into a conversational response.

The sufficiency check is driven by instructions in the prompt. The system instructs the model to confirm that the results are non-empty, that they contain the fields the question asked about, and that they cover every part of the question rather than only some of it. For example, if a user asks for “Q3 sales by region” but the matched template returns only a Q3 total, the results are incomplete, and the model is instructed to flag them as insufficient instead of answering with partial data. The model returns this judgment as a structured signal alongside its response, so the pipeline can branch on it deterministically. This step helps verify that users receive accurate answers rather than partial or misleading information from imperfect template matches.

This task is fundamentally simpler than SQL generation: instead of writing structured code from natural language, the model only needs to read tabular data and either summarize it or declare it insufficient. Because the task is simple, a smaller, faster model like Claude Haiku 4.5 can handle it effectively.

On a cache hit, there is only a single lightweight LLM call, which improves both latency and cost thanks to the smaller model. On a cache miss, the model flags the template results as insufficient and the system falls back to full SQL generation before producing the answer, for three calls in total: the sufficiency check, the SQL generation, and the response. That is one call more than the uncached pipeline, so misses carry extra latency. The trade-off is favorable because the added call is the cheap sufficiency check rather than another expensive generation, and because at a healthy hit rate the savings on hits outweigh the penalty on misses.

5. Fallback to full generation

If no template matches the confidence threshold, or if the validation step determines that cached results are insufficient, the system falls back to the standard Text2SQL pipeline. The question, along with the full context, goes to the foundation model for SQL generation. The generated query executes against the database, and results return to the user. Importantly, this newly generated query doesn’t disappear. It enters the reinforcement loop.

6. Reinforcement loop for cache growth

After a successful fallback generation, the system evaluates whether the new query should join the template cache. If the query executed successfully and returned valid results, it becomes a candidate for templating. The system generalizes the query by replacing specific values with placeholders and computes the original question’s embedding. It then adds this new template-question pair to the vector store, expanding cache coverage. Over time, the cache grows organically to cover query patterns specific to your users’ actual needs.

Results and performance gains

The figures in this section come from our production deployment but treat them as an illustrative model rather than a fixed benchmark. Exact token counts and latencies depend on your schema size, prompt design, model choice, and query mix. What generalizes is the direction of the improvement, not the specific numbers.

The dominant cost and latency in a Text2SQL request come from a single step: generating the SQL query. That call sends the user question, conversation history, the database schema, few-shot examples, and domain guidance to a powerful LLM such as Anthropic Claude Sonnet, which is needed to produce reliable queries. In our deployment this prompt runs on the order of 60K input tokens for a few hundred output tokens, and takes roughly 15-20 seconds. Every other step: embedding, vector search, template filling, and query execution, is minor by comparison. Entity recognition, runs on a dedicated NER model hosted on Amazon SageMaker AI rather than an LLM, adding negligible cost and latency next to SQL generation. Optimizing the pipeline is therefore mostly about avoiding that one expensive call.

On a cache hit, the system skips SQL generation entirely. What remains is response summarization, turning the query results into a conversational answer, which runs on a small model with a small prompt (on the order of a couple thousand input tokens). Because summarization is needed on both, the cached and uncached paths, a cache hit does not remove tokens completely, but it eliminates the 60K-token generation call, cutting token consumption by roughly 90% on that request.

This 90% is the saving on a single cache hit. Overall cost depends on the average across all requests, since cache misses still incur the full generation cost. At the roughly 60% hit rate we observed in production, the blended reduction across all traffic comes out above 50%. Latency follows the same pattern. An uncached request spends 15-20 seconds on the SQL call, retries and error handling included, then a few more seconds on summarization, putting a typical request in the 25-30 second range. On a cache hit, retrieval, template filling, and execution finish well under a second, and the remaining time is almost entirely the summarization call. That brings the end-to-end cache-hit path under 5 seconds, roughly an 80% reduction, or about 6x faster. It also pinpoints where the residual latency comes from: not the cache lookup, but the one LLM call that still has to run.

These per-request gains only matter if cache hits are common. In our production system the hit rate reached about 60% after roughly two weeks of active use, though the achievable rate depends heavily on the domain and how repetitive the queries are. Cache misses run the full pipeline plus the small sufficiency check, so they cost marginally more than a purely uncached request, which means the net gain comes entirely from hits. As the reinforcement loop keeps adding templates, the hit rate climbs and both the cost and latency benefits continue to compound.

Conclusion

Scaling AI applications to production often requires rethinking traditional optimization strategies. In this post, we demonstrated how template-based caching addresses the latency and cost challenges of Text2SQL systems without sacrificing accuracy. By caching SQL query structures rather than complete responses and using semantic similarity to match user questions to templates, the system can bypass expensive LLM inference calls. The reinforcement loop ensures cache coverage grows organically based on actual usage patterns.

In practice, this means: 6x faster response times on cache hits, inference costs decrease proportionally to your cache hit rate, and accuracy remains high because templates are generated by the most capable models. The patterns we covered, such as semantic matching, output generalization, entity extraction, and continuous improvement loops, extend beyond Text2SQL to any AI system where similar requests should produce structurally similar outputs.

About the authors

How AWS IAM role manager rethinks the starting point for IAM roles

2026-08-13 Zach Jiang

Post Syndicated from Zach Jiang original https://aws.amazon.com/blogs/security/how-aws-iam-role-manager-rethinks-the-starting-point-for-iam-roles/

When you build a new application or capability on Amazon Web Services (AWS), you want to focus on what you’re building. Getting a service running almost always begins with AWS Identity and Access Management (IAM). Many AWS services that act on your behalf need an IAM role, an identity the service assumes to access your resources with a defined set of permissions. You then author a trust policy so the service can assume the role, choose the permissions the workload needs, and attach it. Configuring roles and policies for common patterns is repeatable work that doesn’t need to be manual.

IAM role manager does that work for you. When role manager is enabled, AWS creates and configures the IAM roles as you build in supported service consoles, so you can start using a service and let AWS handle the role behind it. You create the resource you want, and role manager provisions and attaches the role you need as part of the same flow, so you can build now and refine permissions as your workload matures.

With that step automated, getting started takes minutes. You can create an AWS Lambda function and start running your code, with its execution role already created and attached, without switching context to set one up. Role creation becomes an automated part of building your application rather than a separate step.

Role manager is especially useful when you’re getting started: the moments when you want to stand up a service or get a proof of concept running and want to defer role configuration until later in your development process. You don’t need prior IAM experience to get started. You keep full control of what it creates, because the roles are ordinary IAM roles that you can view, edit, or delete like any role you author yourself. When you want to tighten a role, AWS IAM Access Analyzer reviews how it has been used and recommends a policy scoped to only the permissions it needs.

How to enable role manager

Role manager has two states, enabled and disabled. Enabling it for an account authorizes AWS to create roles in that account. In an organization, administrators can use a service control policy (SCP) to control whether member accounts can enable or use role manager. To enable it:

Open the IAM console and choose Account settings.
In the role manager section, choose Enable.

Figure 1: Enable Role Manager

Some AWS services already create a role for you when you create a resource that needs one. Role manager doesn’t change that: those services keep creating roles automatically, and roles you already created keep working. What role manager adds is a single account-level control, and coverage for a case that built-in flows can’t handle: tasks whose permissions AWS can’t determine in advance, such as running your own code. For those tasks, role manager provisions a role that you can narrow later.

Example: Create an Amazon EventBridge rule

Start with a common task: an Amazon EventBridge rule that invokes a target, such as an Amazon Simple Queue Service (Amazon SQS) queue or an Amazon Simple Notification Service (Amazon SNS) topic. Without role manager, you would pause here to create a role that lets EventBridge invoke the target, write the role’s trust policy, attach the required permissions, and then return to finish the rule. With role manager enabled, you define the rule and its target, choose Create, and role manager provisions the role and attaches it for you. The EventBridge console shows the rule created and ready, and you never open the role-creation flow.

Figure 2: Creating an EventBridge rule with no manual role setup

The role comes from an AWS managed role template: a definition AWS builds and maintains for a specific task, with the trust policy and permissions already worked out. The console calls a new IAM API, AcquireRole, which finds the matching template, provisions the role from it, and returns it to EventBridge. Depending on the service, AcquireRole either creates a new role or reuses one that already fits, so an account does not fill up with duplicate roles for the same task.

Role manager creates the role using your own IAM permissions, not a separate role-manager permission. To provision a new role, you need permission for the actions the template performs: at minimum, you need permissions to create and attach roles. When AcquireRole reuses an existing role instead of creating one, it needs only iam:GetRole and iam:GetRoleTemplateVersion. If you’re missing either of these permissions, the console tells you which one is needed rather than creating the role.

Run code that calls other AWS services

Not every task has a set of permissions AWS can define in advance. When a role runs your own code, such as a Lambda function, AWS has no way of knowing which services that code will call. Role manager covers this case too: create a Lambda function with role manager enabled, and it attaches an execution role that your code can use right away and that you can narrow once you know what the function calls.

Because the permissions your code needs aren’t known up front, role manager attaches the AWS managed policy PowerUserAccess to the role. PowerUserAccess grants access to AWS services so your function can call what it needs. By design, it doesn’t grant permission to manage IAM, AWS Organizations, or account settings. The template also configures the role to trust only the Lambda service.

Figure 3: Create an AWS Lambda function with no manual role setup

Role manager attaches an execution role, and your function is ready to run. Figure 4 shows the Execution role panel on the function’s Configuration tab, with the role that role manager attached.

Figure 4: Role manager provides a role automatically to an AWS Lambda function

You can open the role in the IAM console to review its permissions. Figure 5 shows the role’s Permissions tab with the PowerUserAccess policy attached.

Figure 5: Permissions of the role provided by role manager for an AWS Lambda function

You keep full visibility into what role manager creates. Every role it creates records the role template it came from, and both GetRole and ListRoles return that template reference. You can inspect any role in your account and tell which were created by role manager. You read a role’s trust policy and permissions the same way you would for a role you authored, and AWS CloudTrail records each role’s creation.

Refining roles as workloads mature

As your workloads mature, refine the roles that role manager created to follow least privilege. When you’re ready, you can disable role manager and get IAM Access Analyzer unused access analysis at no additional cost for 90 days. Access Analyzer looks at how each role has been used and recommends a policy you can apply that keeps only the permissions the role needs. Start with the roles attached to your most critical workloads and work outward.

Disabling role manager doesn’t disrupt anything already running: your resources keep the roles they have, those roles stay in your account until you change them, and from that point you author new roles yourself, the same as before. If you would rather narrow a single role than the whole account, editing that role removes it from role manager’s control and it becomes a standard customer-managed role, with your changes preserved. In sandbox or development accounts, keeping role manager enabled saves time. For production workloads, disable role manager and refine the roles it created to least privilege before going live.

Conclusion

Role manager automates IAM role setup so you can focus on building from the start. When you enable it, AWS creates and attaches the IAM roles your resources need as you build, so you can start in minutes without prior IAM experience. Because these are IAM roles that you fully control, you keep the same visibility and the same tools you already use. Keep role manager enabled while you build, and refine the roles it created as your workloads mature.

To get started, enable role manager in the IAM console and create a resource in a supported service. To learn more, see IAM role creation and the list of supported services in the IAM User Guide.

If you have feedback about this post, submit comments in the Comments section below.

The challenge of running dbt at scale

Why Amazon Athena as the default dbt engine

Serverless processing

Why Athena fit our solution

Optimized incremental processing

Architecture overview

Architecture diagram

Cross-account processing model

Centralizing dbt runs with the dbt launcher

Run lifecycle

Configuration-driven runs with Amazon DynamoDB

CI/CD pipeline diagram

Publishing run results with Amazon EventBridge

Validation and observability

Cost control and resource cleanup

Results

Conclusion

Resources

About the authors

How Amazon SES tracking works

Prerequisites

Option A: Use per-request tracking overrides (recommended)

Option B: Use separate configuration sets

Disabling tracking on individual links

Manage consent in your application

Monitoring and governance

Clean up

Conclusion

Related posts

About the authors

Solution overview

Prerequisites

Method 1: DSN-based connection (Athena Power BI connector)

Add your SSO user as a member of your SageMaker Unified Studio project

Gather configuration values to configure your Amazon Athena ODBC DSN

Configure the ODBC DSN

Connect Power BI Desktop to your data

Create your dashboard and publish it

Method 2: DSN-less connection (Power BI ODBC connector)

Administrator setup

Add the IAM identity as a member of SageMaker Unified Studio project

Gather configuration values

Connect Power BI Desktop to your data and publish

Configure the on-premises data gateway and view your report on Power BI Service

Create and attach an IAM role to the Power BI Gateway EC2 instance

Add the Power BI Gateway IAM role as a member of SageMaker Unified Studio project

Configure the data source on Power BI Gateway

Configure the data source and view your report on Power BI Service

Method 1 (DSN-based)

Method 2 (DSN-less)

Clean up

Conclusion

About the authors

Solution overview

Prerequisites

Administrator setup

Create a permission set in IAM Identity Center

Assign users to the permission set

Configure AWS IAM Identity Center profile

Verify authentication and daily use

Add your IAM identity as a member of your Amazon SageMaker Unified Studio project

Gather the information to authenticate

Method 1: DSN-based connection (Athena Power BI connector)

Gather configuration values to configure your Amazon Athena ODBC DSN

Configure the ODBC DSN

Connect Power BI Desktop to your data

Create your dashboard and publish it

Method 2: DSN-less connection (Power BI ODBC connector)

Gather configuration values to configure your DSN-less connection

Connect Power BI Desktop to your data and publish

Configure the gateway and view your report on Power BI Service

Create and attach an IAM role to the Power BI Gateway EC2 instance

Add the Power BI Gateway IAM role as a member of SageMaker Unified Studio project

IAM-based domain

Configure the data source on Power BI Gateway

Configure the data source and view your report on Power BI Service

Method 1 (DSN-based)

Method 2 (DSN-less)

Clean up

Conclusion