Tag Archives: Amazon EventBridge

Scheduling email campaigns at scale with Amazon EventBridge Scheduler

Post Syndicated from Oluwaseun Ademuwagun original https://aws.amazon.com/blogs/compute/scheduling-email-campaigns-at-scale-with-amazon-eventbridge-scheduler/

Scheduling email campaigns becomes more complex when you need to send email to millions of recipients at the unique time best suited for each customer. Consider these examples:

  • A flash sale might need to hit inboxes at 9 AM local time across every time zone.
  • A follow-up email (often known as a drip sequence) might need to send a second message exactly 3 days after the first message per subscriber.
  • A re-engagement campaign might target users who haven’t logged in for 30 days.

The scheduling requirements involve multiple considerations. You’re sending hundreds of millions of messages, each at its own optimal moment personalized to the recipient’s time zone and behavior.

In this post, we walk through how to use Amazon EventBridge Scheduler to personalize email notifications to each recipient. We create one schedule per recipient to deliver each email at its individually optimal moment, with zero idle compute cost. We also show how Amazon EventBridge Scheduler handles higher volumes. Amazon EventBridge Scheduler supports billions of schedules. By default, you have a quota of 10 million schedules.

Solution overview

When every recipient has their own ideal delivery time, you need a scheduling layer that can hold billions of individual send intents and fire each one at the right moment. Most teams reach for one of three familiar patterns, each with tradeoffs that become painful at scale.

  1. Batch cron jobs: A job runs every hour, queries for all messages due in the next window, and sends them out. Recipients get email in imprecise hourly batches. At scale, the batch job itself becomes a bottleneck, processing millions of rows per run, competing for database connections, and creating a sudden spike in load on the email provider.
  2. Delay queues: You can use Amazon Simple Queue Service (Amazon SQS) as a delay queue. A delay queue postpones the delivery of new messages to a customer for a set time. A limitation of this approach is that Amazon SQS caps delays at 15 minutes.
  3. Third-party campaign tools: Offload to a SaaS email platform. This works until you need tight integration with your application data, custom send-time optimization, or control over delivery infrastructure. You’re also paying per-recipient fees that compound at scale.

All three approaches either sacrifice precision (batching), hit architectural limits (delay queues), or surrender control (third-party tools).

The building block approach

Amazon EventBridge Scheduler treats each email send as a discrete scheduled action. Instead of “process all messages due this hour,” you express the intent directly: “send this email to this person at this time.” Amazon EventBridge Scheduler holds that intent with zero compute cost until the moment arrives, then triggers the scheduled action. See the Amazon EventBridge Scheduler User Guide for the full API reference and current service quotas.

For email campaigns, Amazon EventBridge Scheduler becomes the send-time dispatcher, the component that schedules every email in a campaign for its individually optimal moment, whether that’s timezone-adjusted, behavior-triggered, or sequence-driven.

Architecture diagram

The architecture follows an event-driven, per-recipient scheduling pattern for an email campaign. To start the campaign, you first define the target audience and the content they receive. Next, you need a way to create the per-recipient schedule. To do that for a campaign that can contain millions of recipients, you need a scalable mechanism to create the schedules. You can achieve this with an AWS Step Functions state machine, a serverless workflow service that coordinates multiple AWS services into structured, visual workflows called state machines. In this solution, we orchestrate the creation of the schedules by using a Distributed Map state within the state machine, which lets us fan out and accelerate schedule creation. It does this by splitting a large dataset into chunks and processing them across thousands of parallel child executions. It reads the recipient list from Amazon Simple Storage Service (Amazon S3), applies time zone logic per recipient, and creates an individual Amazon EventBridge Scheduler resource for each recipient in parallel. After the workflow creates all schedules, the execution completes.

The actual email delivery happens later, entirely decoupled from the campaign creation step. At the scheduled time, Amazon EventBridge Scheduler invokes Amazon Simple Email Service (Amazon SES) directly, passing the template name and personalization data as template variables. For campaigns requiring complex personalization logic (conditional content, real-time suppression checks, or data enrichment), you can optionally route through an AWS Lambda function before SES. If you need to adjust timing or content for specific recipients, you can update their individual schedules directly without reprocessing the entire campaign.

Figure 1: Per-recipient email scheduling architecture with Amazon EventBridge Scheduler

Walkthrough

The solution uses four core components that work together: a campaign manager to define send-time rules, Step Functions Distributed Map to fan out and accelerate schedule creation, Amazon EventBridge Scheduler to hold each per-recipient intent and deliver through Amazon SES directly, and automatic cleanup through schedule self-deletion.

How it works

  1. Create the campaign: A marketer defines the campaign: audience segment, email template, and send-time rules (for example, “9 AM in each recipient’s local time zone” or “24 hours before a Black Friday sale”).
  2. Campaign manager fans out: An AWS Step Functions workflow uses Distributed Map to iterate over the recipient list and create one Amazon EventBridge Scheduler schedule per recipient per campaign step directly through SDK integration. Each schedule encodes the exact send time for that individual.
  3. Amazon EventBridge Scheduler fires at the right moment: At each recipient’s scheduled time, Amazon EventBridge Scheduler invokes Amazon SES directly through a universal target, passing the template name and personalization data (recipient name and attributes) as template variables.
  4. SES personalizes and sends: Amazon SES renders the email template with the provided data and delivers the message.
  5. Schedule self-deletes: ActionAfterCompletion='DELETE' prevents the accumulation of spent schedules.

Prerequisites

To follow along with this walkthrough, you need the following:

  • AWS account and permissions: An active AWS account with permissions to create Amazon EventBridge Scheduler schedules, AWS Step Functions state machines, and Amazon SES identities, along with an AWS Identity and Access Management (IAM) role for Amazon EventBridge Scheduler to invoke Amazon SES.
  • Development environment: Python 3.13 or later, AWS SDK for Python (Boto3) version 1.26 or later, and AWS Command Line Interface v2 (AWS CLI v2).
  • Amazon SES configuration: Move your Amazon SES account out of sandbox mode to allow sending to arbitrary recipients.

Scaling the fan-out with Step Functions

For campaigns with millions of recipients, use AWS Step Functions Distributed Map to parallelize schedule creation. When you want to activate a campaign, you trigger a Step Functions workflow. This workflow fans out and creates schedules across the recipient list by using a Distributed Map with direct SDK integration. The direct SDK integration between Step Functions and Amazon EventBridge Scheduler lets each child execution call CreateSchedule directly. The following state machine definition reads recipients from an Amazon S3 CSV file and creates schedules in parallel:

{
  "Comment": "Fan out campaign schedule creation via direct SDK integration",
  "StartAt": "EnsureScheduleGroup",
  "States": {
    "EnsureScheduleGroup": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:scheduler:createScheduleGroup",
      "Parameters": {
        "Name.$": "States.Format('campaign-{}', $.campaign_id)"
      },
      "ResultPath": null,
      "Catch": [
        {
          "ErrorEquals": [
            "Scheduler.ConflictException"
          ],
          "ResultPath": null,
          "Next": "FanOutRecipients"
        }
      ],
      "Next": "FanOutRecipients"
    },
    "FanOutRecipients": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "BuildScheduleInput",
        "States": {
          "BuildScheduleInput": {
            "Type": "Pass",
            "Parameters": {
              "schedule_name.$": "States.Format('campaign-{}-{}', $.campaign_id, $.recipient.id)",
              "group_name.$": "States.Format('campaign-{}', $.campaign_id)",
              "schedule_expression.$": "States.Format('at({}T{}:00:00)', $.send_date_date, $.send_hour)",
              "timezone.$": "$.recipient.timezone",
              "target_input": {
                "FromEmailAddress": "[email protected]",
                "Destination": {
                  "ToAddresses.$": "States.Array($.recipient.email)"
                },
                "Content": {
                  "Template": {
                    "TemplateName.$": "$.template_id",
                    "TemplateData.$": "States.JsonToString($.recipient.attributes)"
                  }
                }
              }
            },
            "Next": "CreateSchedule"
          },
          "CreateSchedule": {
            "Type": "Task",
            "Resource": "arn:aws:states:::aws-sdk:scheduler:createSchedule",
            "Retry": [
              {
                "ErrorEquals": [
                  "Scheduler.SdkClientException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 3,
                "BackoffRate": 2
              }
            ],
            "Parameters": {
              "Name.$": "$.schedule_name",
              "GroupName.$": "$.group_name",
              "ScheduleExpression.$": "$.schedule_expression",
              "ScheduleExpressionTimezone.$": "$.timezone",
              "FlexibleTimeWindow": {
                "Mode": "FLEXIBLE",
                "MaximumWindowInMinutes": 5
              },
              "Target": {
                "Arn": "arn:aws:scheduler:::aws-sdk:sesv2:sendEmail",
                "RoleArn": "arn:aws:iam::976764934189:role/CampaignFanOutRole-dev",
                "Input.$": "States.JsonToString($.target_input)",
                "RetryPolicy": {
                  "MaximumEventAgeInSeconds": 7200,
                  "MaximumRetryAttempts": 5
                }
              },
              "ActionAfterCompletion": "DELETE"
            },
            "ResultPath": null,
            "End": true
          }
        }
      },
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:getObject",
        "ReaderConfig": {
          "InputType": "CSV",
          "CSVHeaderLocation": "FIRST_ROW"
        },
        "Parameters": {
          "Bucket.$": "$$.Execution.Input.recipient_bucket",
          "Key.$": "$$.Execution.Input.recipient_key"
        }
      },
      "ItemSelector": {
        "campaign_id.$": "$$.Execution.Input.campaign_id",
        "template_id.$": "$$.Execution.Input.template_id",
        "send_date_date.$": "$$.Execution.Input.send_date_date",
        "send_hour.$": "$$.Execution.Input.send_hour",
        "recipient": {
          "id.$": "$$.Map.Item.Value.id",
          "email.$": "$$.Map.Item.Value.email",
          "timezone.$": "$$.Map.Item.Value.timezone",
          "attributes": {
            "first_name.$": "$$.Map.Item.Value.first_name",
            "signup_date.$": "$$.Map.Item.Value.signup_date"
          }
        }
      },
      "MaxConcurrency": 1000,
      "ResultPath": null,
      "End": true
    }
  }
}

Concurrency alignment with Amazon EventBridge Scheduler API limits

Step Functions Distributed Map supports up to 10,000 concurrent child workflows. Each child calls the CreateSchedule API directly, which has a default rate limit of 5,000 TPS. This limit is sufficient for most campaigns. If your campaign volumes require higher throughput, check your current quotas in the Service Quotas console and request an increase.

To avoid throttling, set MaxConcurrency below the CreateSchedule TPS quota. A value of 2,500 provides a comfortable buffer to account for bursts and retries without requiring a quota change. For larger campaigns, request an increase through AWS Service Quotas (adjustable to tens of thousands) and raise MaxConcurrency to match.

Canceling a campaign

A schedule group is an Amazon EventBridge Scheduler resource used to organize schedules. For this use case, we have a schedule group per campaign. If you need to pull a campaign (error in content, legal issue, or strategy change), you can cancel all scheduled sends for that campaign by deleting the entire schedule group. The following code shows how to cancel all pending sends for a campaign:

def cancel_campaign(campaign_id):
    """Cancel all pending sends for a campaign by deleting its schedule group."""
    scheduler.delete_schedule_group(
        Name=f'campaign-{campaign_id}'
    )

Operational considerations

Moving to production introduces a few scaling and reliability concerns to plan for.

Handling invocation spikes at delivery time

When a mass campaign schedules millions of messages for the same time, this creates cascading pressure across two limits:

  • Amazon EventBridge Scheduler invocations throttle limit: The default is 1,000 TPS per AWS Region, and it is adjustable to tens of thousands of TPS through AWS Service Quotas. Amazon EventBridge Scheduler queues invocations internally and retries with exponential backoff when the downstream target throttles.
  • Amazon SES sending quotas: Your SES account has a per-second sending rate. If the effective invocation rate exceeds this, messages fail with throttling errors. Align Amazon SES sending quotas with your campaign volume. Check your current SES quota in the Service Quotas console and request an increase before launching large campaigns. See Amazon SES best practices for deliverability at scale.

To handle an invocation spike, we recommend using the FlexibleTimeWindow feature of Amazon EventBridge Scheduler. Setting MaximumWindowInMinutes lets Amazon EventBridge Scheduler spread invocations across a time window rather than firing them all at the exact second. Size the window based on your campaign: divide the total schedules by your effective TPS to determine the minimum spread needed. For example, 500,000 schedules at 5,000 TPS need at least a 2-minute window.

Cost model

You pay for Amazon EventBridge Scheduler on a per-invocation basis.

Cleanup

To avoid ongoing charges, delete the resources created during this walkthrough:

  1. Delete any runtime-created schedule groups. 
    aws scheduler delete-schedule-group --name campaign-<campaign-id>

  2. Delete the Step Functions state machine. 
    aws stepfunctions delete-state-machine \
    --state-machine-arn arn:aws:states:us-east-1:<account-id>:stateMachine:CampaignFanOut

Note: If you have active schedules still waiting to fire, deleting the schedule group will cancel all pending sends.

IAM role for Amazon EventBridge Scheduler and Step Functions

The Step Functions state machine needs an execution role with permissions to create schedules, send email, and pass the role to the Amazon EventBridge Scheduler service. Amazon EventBridge Scheduler needs permissions to call SES. The following policy shows the combined permissions for both scenarios:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPassRoleToScheduler",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/CampaignFanOutRole",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "scheduler.amazonaws.com"
        }
      }
    },
    {
      "Sid": "AllowSESSend",
      "Effect": "Allow",
      "Action": [
        "ses:SendEmail",
        "ses:SendTemplatedEmail"
      ],
      "Resource": "arn:aws:ses:<REGION>:<ACCOUNT_ID>:identity/[email protected]"
    },
    {
      "Sid": "DistributedMapExecution",
      "Effect": "Allow",
      "Action": [
        "states:StartExecution",
        "states:DescribeExecution",
        "states:StopExecution"
      ],
      "Resource": [
        "arn:aws:states:<REGION>:<ACCOUNT_ID>:stateMachine:CampaignFanOut",
        "arn:aws:states:<REGION>:<ACCOUNT_ID>:execution:CampaignFanOut:*"
      ]
    },
    {
      "Sid": "ReadRecipientsBucket",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::campaign-recipients-<ACCOUNT_ID>",
        "arn:aws:s3:::campaign-recipients-<ACCOUNT_ID>/*"
      ]
    },
    {
      "Sid": "CreateSchedules",
      "Effect": "Allow",
      "Action": "scheduler:CreateSchedule",
      "Resource": "arn:aws:scheduler:<REGION>:<ACCOUNT_ID>:schedule/campaign-*"
    },
    {
      "Sid": "CreateScheduleGroups",
      "Effect": "Allow",
      "Action": "scheduler:CreateScheduleGroup",
      "Resource": "arn:aws:scheduler:<REGION>:<ACCOUNT_ID>:schedule-group/campaign-*"
    },
    {
      "Sid": "PassRoleToScheduler",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/SchedulerCampaignRole",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "scheduler.amazonaws.com"
        }
      }
    }
  ]
}

This policy scopes the scheduler:CreateSchedule and scheduler:CreateScheduleGroup actions to resources prefixed with campaign-*, following least-privilege principles.

A condition restricts the iam:PassRole permission so that it can only pass the role to the Amazon EventBridge Scheduler service.

Conclusion

In this post, we walked through how to use Amazon EventBridge Scheduler to personalize email campaign delivery for each recipient. An email campaign system has two core problems: deciding what to send and deciding when to send it. Most teams over-engineer the “when” with polling infrastructure, batch jobs, and queue chains. Amazon EventBridge Scheduler collapses that into a single CreateSchedule API call per recipient.

To get started, explore Amazon EventBridge Scheduler on the AWS Management Console. Browse Serverless Land patterns for more than 20 Amazon EventBridge Scheduler patterns and other use cases beyond email campaigns.

Suggested tags: Amazon EventBridge, architecture, events, modernization, serverless.

Serverless ICYMI Q2 2026

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/serverless-icymi-q2-2026/

In this 33rd quarterly recap post, discover the most impactful AWS serverless launches, features, and resources from Q2 2026 that you might have missed. Stay current with the latest serverless innovations that can improve your applications.

In case you missed our last ICYMI, read about what happened in Q1 2026.

Serverless ICYMI Q2 2026 banner

AWS Lambda MicroVMs

AWS Lambda MicroVMs is a new serverless compute primitive for running user or AI-generated code in isolated, stateful execution environments. Built on the same Firecracker virtualization that powers over 15 trillion monthly Lambda invocations, MicroVMs give you VM-level isolation with near-instant launch and resume. Each MicroVM runs in its own Linux environment with no shared kernel or resources between sessions. This isolation makes it a useful solution for AI coding assistant sandboxes, interactive code or multi-tenant development environments, CI/CD build environments, data analytics platforms, vulnerability scanners, and game servers that run user-supplied scripts.

Standard Lambda functions are best for event-driven, request-response workloads which have a 15-minute timeout. MicroVMs are purpose-built for single end user or session workloads and can preserve state for up to 8 hours. You get full lifecycle controls including launch, suspend, resume, and terminate. You can suspend them during the 8 hours if you don’t need them active. MicroVMs retain memory and disk state for the length of the session, even while suspended. They can auto resume when you need to use them again.

Serverless Land contains example applications and a resources page with more details. The Serverless Office Hours live stream has more explanations and live demos.

Amazon S3 Files and Lambda integration

Amazon S3 Files makes your S3 buckets accessible as high-performance file systems. S3 files is a fully featured, POSIX-compatible file system to access to your data with approximately 1ms latency.

For serverless workloads, the Lambda integration with S3 Files lets your functions mount an S3 bucket as a local file system. Your function reads and writes files at a local mount path like /mnt/data, and the file system handles synchronization with S3 automatically. You can avoid downloading objects to /tmp from S3 within your function and work directly with files. Applications that assume a file system can now run on Lambda without rewriting their I/O layer. Use cases include sharing data between functions, ML model loading, document processing, media transcoding, or any pipeline that treats data as files rather than objects.

AWS Lambda durable functions

The Lambda durable functions SDK for Java is now generally available, joining Python and TypeScript. This allows Java developers to build multi-step workflows with automatic checkpointing and recovery without adding external orchestration. Durable functions is also now available in 16 additional AWS Regions. Learn how to build fault-tolerant multi-agent AI workflows to coordinate multiple AI agents that call tools, make decisions, and hand off work. There is automatic recovery if any agent fails mid-task. Voice analytics with Amazon Bedrock shows building a pipeline that processes call recordings through transcription, sentiment analysis, and summarization with durable checkpoints between each stage. For best practices, AI patterns, and futures, view the live stream.

AWS Lambda Managed Instances

Lambda Managed Instances now allows you to build memory-intensive apps with up to 32 GB (3x more than standard Lambda). This allows use cases like in-memory caching, large dataset analytics, and ML inference that previously required considering other services.

Architecture diagram for AWS Lambda Managed Instances memory-intensive apps

Figure 1 — AWS Lambda Managed Instances for memory-intensive apps architecture

Scheduled scaling lets you pre-warm capacity for predictable traffic patterns with Amazon EventBridge Scheduler. This helps reduce cold start latency during known demand spikes. Tag propagation automatically applies your function tags to the underlying Amazon EC2 instances, Amazon Elastic Block Store volumes, and network interfaces. This helps finance teams with cost allocation visibility without manual tag management.

Other Lambda updates

Response streaming is now available in all commercial AWS Regions, bringing full regional parity for progressively streaming data back to clients. This is useful for LLM-powered applications where users expect to see tokens as they generate rather than waiting for a complete response.

The tenant isolation mode now integrates with Event Source Mappings from Amazon SQS, Amazon Kinesis, and Amazon EventBridge. Multi-tenant SaaS applications can process messages in isolated execution environments without building custom routing logic.

If you have a fleet of functions on older runtimes, you can now upgrade runtimes at scale using AWS Transform custom. This uses AI to analyze your function code, identify breaking changes for the target runtime version, and generate the code modifications needed. This can help teams save manual migration effort across many functions. The Serverless Office Hours live stream has more information.

Lambda added the Ruby 4.0 runtime. In addition to providing access to the latest Ruby language features, Lambda adds support for Lambda advanced logging controls.

AWS Serverless Application Model (AWS SAM) CLI now supports BuildKit for building container images from Dockerfiles. This allows faster multi-stage builds with better caching, cross-architecture image builds, and Docker secrets to keep credentials out of final image layers.

Containers with Mama J




Serverless with Mama J

Mama J is back in the second video of a series where Eric Johnson explains what he does all day at work to his mother. Previously, they talked serverless and Lambda. This time it’s containers, what they are, why they exist, and how AWS manages them at scale. Eric goes through the “it works on my machine” problem, how Docker builds images, container orchestration and how containers differ from Lambda.

View the video on the AWS Developers YouTube channel.

AWS Step Functions

AWS Step Functions has an Amazon Bedrock AgentCore-powered agentic reasoning step. You can embed AI agent reasoning directly inside a workflow as a native step type. This bridges structured orchestration with autonomous agent behavior. Your workflow handles the deterministic parts such as branching, retries, timeouts, parallel execution, while the agentic step handles the parts that require flexible reasoning.

Amazon EventBridge

Amazon EventBridge Scheduler added 619 new SDK API actions as targets, including Lambda Managed Instances operations. This means you can schedule calls to a much broader set of AWS APIs without writing a Lambda function.

A new post walks through building a multi-Region event-driven failover architecture with Amazon EventBridge and Amazon Route 53. The pattern uses Amazon EventBridge global endpoints with Route 53 health checks to automatically route events to a healthy Region during failures. This provides active-active or active-passive resilience for event-driven workloads.

Amazon Bedrock AgentCore

The Amazon Bedrock AgentCore harness reached general availability. Two API calls give you a running agent in seconds which runs in its own isolated environment with a filesystem and shell. It can read files, run commands, and write code safely.

AgentCore Payments (preview) allows agents to autonomously access and pay for APIs and MCP servers, opening up agent-to-agent commerce. AgentCore Memory has metadata for long-term memory so agents retain and recall context across sessions. Web Search on AgentCore grounds agents in current, cited web knowledge. The Runtime now supports bring-your-own file systems from S3 Files and Amazon Elastic File System, and Node.js for direct code deployment.

Strands Agents SDK

The open source Strands Agents SDK shipped three capabilities. Context management that cuts token costs in half by intelligently pruning what goes into the model context window, Strands Shell for sandboxed agent code execution, and Strands Evals 1.0 with chaos testing and adversarial red teaming. This can reduce costs to help make production agent workloads cheaper without sacrificing quality. A Serverless Office Hours live stream covered the new features in depth.

The TypeScript SDK reached general availability, giving JavaScript and TypeScript developers the same model-driven agent framework. Erik Hanchett ran this live stream with more details. A new blog post on building research assistants with Strands walks through the full app from prototype to working application in about 200 lines of Python.

Agent Toolkit for AWS and AI coding

The Agent Toolkit for AWS became generally available with three plugins (aws-core, aws-agents, aws-data-analytics), over 30 curated skills, and the AWS MCP Server. View this video for an introduction. This gives AI coding agents such as Kiro, Claude Code, and Cursor expert AWS knowledge which helps to reduce errors and lower token costs. For more information on the serverless tools available when using AI, see this Serverless Land resources page.

Serverless Office Hours ran a live stream series finding out how experts use AI to build serverless applications. Hear from:

Kiro launched Kiro Pro Max and an iOS mobile app for approving and monitoring agentic coding sessions from your phone. Amazon Q Developer IDE plugins are transitioning to Kiro. The Kiro power for AWS DevOps Agent connects your IDE directly to production intelligence. You can investigate incidents and generate fixes without context switching.

Serverless blog posts

April

June

Serverless Office Hours

Join our live stream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. View episodes on-demand at serverlessland.com/office-hours.

April

May

June

Still looking for more?

The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Developer Advocacy team to get the latest news, follow conversations, and interact with the team.

And finally, visit Serverless Land for your serverless needs.

Patch perfect: Automating Amazon Redshift patch testing

Post Syndicated from Eva Donaldson original https://aws.amazon.com/blogs/big-data/patch-perfect-automating-amazon-redshift-patch-testing/

Amazon Redshift continuously innovates to deliver improved performance and advanced features. In some releases, Amazon Redshift patches might introduce behavior changes. Testing patches in a non-production environment confirms that production workloads continue to function and you can maintain your applications’ service level agreements. As a best practice, keep Dev/QA clusters on the Current patch track and Production on the Trailing track. Test on Dev/QA when a patch lands, allowing 1–6 weeks of review before the scheduled production deployment.

In this post, we demonstrate an automated test suite that validates your Amazon Redshift cluster automatically after any patch, reboot, or modification. It uses standard drivers against real workload patterns to provide a verified gate between a patch landing and that patch reaching production.

Architecture

The solution uses native AWS services to create an automated validation pipeline.

Architecture diagram of the patch testing pipeline: Amazon EventBridge triggers AWS Lambda, which runs an AWS Fargate task that tests the cluster and reports to Amazon S3 and Amazon SNS

Figure 1 — High-level architecture diagram

Process overview showing the four stages: event detection, orchestration, test execution, and reporting

Figure 2 — Process overview

  1. Event Detection: When your Amazon Redshift cluster receives a patch, reboot, or modification, the Amazon Redshift cluster event notifications fire. Amazon EventBridge rules match these events automatically.
  2. Orchestration: A lightweight AWS Lambda function receives the event from the Amazon EventBridge rule and launches an AWS Fargate task. The task runs in a subnet within the same Amazon Virtual Private Cloud (VPC) as your Amazon Redshift cluster, giving the test runner direct network connectivity to the cluster endpoint.
  3. Test Execution: A Docker container runs a comprehensive test suite in four phases:
    • JDBC Driver Tests – Validates the official Amazon Redshift JDBC driver, testing DatabaseMetaData API calls, connection handling, and queries that tools like SQL Workbench/J depend on.
    • ODBC Driver Tests – Validates the PostgreSQL ODBC driver with SQLTables, SQLColumns, and other ODBC API calls that RStudio and similar tools use.
    • Catalog SQL Queries – Runs approximately 35 queries against pg_catalog, information_schema, and svv_* views, organized by client (SQL Workbench, DBeaver, RStudio, JDBC metadata API).
    • Performance Benchmarks – Executes your custom workload queries and compares execution time against known baselines, flagging regressions. For convenience, the solution includes sample queries to be replaced with performance validation queries from your workloads.
  4. Reporting: Detailed JSON results land in Amazon Simple Storage Service (Amazon S3) for historical analysis. An Amazon Simple Notification Service (Amazon SNS) notification sends your team an email immediately with a pass/fail summary. Full JSON results are written to Amazon S3 with timing data for every individual query, row counts, error details, and the Amazon EventBridge event that triggered the run. If tests fail, you have specific, actionable evidence (which queries broke, which drivers failed, which benchmarks regressed) to open a support case requesting a rollback and defer maintenance until the case is resolved. When tests succeed, you can move forward with confidence to production.

For real-time feedback while the tests are running, a quick command tells you the current state:

aws lambda invoke --function-name my-redshift-tests-trigger \
--payload '{}' --cli-binary-format raw-in-base64-out /dev/stdout

What gets tested

The test suite covers two critical areas: client tool compatibility and query performance.

Client compatibility queries

The test suite replicates the connection behavior of popular SQL clients by issuing the same metadata API calls and queries they perform when connecting to your cluster.

Client What’s tested
SQL Workbench/J Connection queries, schema browsing, metadata enumeration
DBeaver Database object discovery, catalog traversal
RStudio (DBI/odbc) ODBC-specific catalog queries, column type mapping
JDBC Metadata API getTables(), getColumns(), getPrimaryKeys(), and other DatabaseMetaData method equivalents

The package contains the exact queries these clients execute upon connection.

Performance regression detection

The benchmark phase of the suite automatically detects whether it has been run before. On the first execution, it captures baseline query execution times as the “known good” state for your pre-patch environment. On every subsequent run, it compares current query timings against the stored baseline and flags any regressions. If a query that previously completed in 2 seconds now takes 15, the report calls it out immediately. This phase is designed to test your most performance-sensitive queries.

Prerequisites

Before deploying, make sure your environment meets the following requirements:

Docker installed. Consider building the image with AWS CloudShell, which comes with Docker pre-installed. You can do this either by uploading the customized repo to Amazon S3 and then downloading it to AWS CloudShell, or by cloning and customizing the repo directly within AWS CloudShell.

Getting started

The full solution is available on GitHub. It includes the AWS CloudFormation template, Docker build scripts, test suite, and documentation.

Clone the GitHub repo, customize it for your workload, deploy it against a Dev/QA cluster.

Detailed instructions are included in the package README.md. Reference those for deployment.

Step 1: Clone the repo

Clone the GitHub repo.

Step 2: Customize the scripts for your environment

The test suite ships with comprehensive default queries. After cloning and before deployment, edit the scripts as described in the following sections for each phase.

Add your performance-critical queries

Edit bundle/run_tests.py and replace the example queries with queries where performance is critical:

BENCHMARK_QUERIES = {
    "daily_patient_summary": """
SELECT department, COUNT(DISTINCT patient_id), AVG(los_days)
FROM clinical.encounters
WHERE admit_date >= CURRENT_DATE - 30
GROUP BY 1
""",
    "revenue_rollup": """
SELECT payer_type, SUM(total_charges)
FROM billing.claims
WHERE service_date >= DATE_TRUNC('month', CURRENT_DATE)
GROUP BY 1
""",
}

Add client-specific catalog queries

If your team uses custom views or schemas, add them to bundle/client_catalog_queries.py:

"custom_view_check": {
    "description": "Verify our reporting view works after patching",
    "sql": "SELECT * FROM analytics.monthly_kpis LIMIT 10",
},

Step 3: Build the Docker image

Execute build-image.sh, which creates an Amazon ECR repository, builds the Docker image (with JDBC and ODBC drivers bundled), and pushes it, outputting the image URI for the next step.

# Upload project to S3, then build in CloudShell
./build-image.sh --stack-name my-redshift-tests

Step 4: Deploy the stack

Use the AWS Command Line Interface (AWS CLI) to deploy the AWS CloudFormation stack with your environment-specific parameters. The stack creates the required components: Amazon Elastic Container Service (Amazon ECS) cluster, AWS Fargate task definition, security groups, VPC endpoints (to keep AWS Secrets Manager and Amazon SNS traffic off the NAT gateway), Amazon S3 bucket, Amazon SNS topic, AWS Lambda trigger, and Amazon EventBridge rules.

aws cloudformation deploy \
--template-file template.yaml \
--stack-name my-redshift-tests \
--parameter-overrides \
RedshiftSecretArn=arn:aws:secretsmanager:... \
RedshiftHost=my-cluster.xxxx.us-east-2.redshift.amazonaws.com \
RedshiftClusterIdentifier=my-cluster \
VpcId=vpc-xxxxxxxx \
VpcSubnetIds=subnet-aaa,subnet-bbb \
RedshiftSecurityGroupId=sg-xxxxxxxx \
EcrImageUri=123456789012.dkr.ecr.us-east-2.amazonaws.com/my-redshift-tests-runner:latest \
[email protected] \
--capabilities CAPABILITY_NAMED_IAM

Key takeaways

Here are the core principles that make automated patch testing effective:

  1. Dev/QA on Current track, Production on Trailing: This separation creates the buffer window between when a patch is available and when it reaches production. Without it, there’s no opportunity to catch regressions before they affect users.
  2. Automate the validation: The track split is most effective if the test suite runs after every patch. Event-driven automation helps confirm no patch goes untested during the buffer window.
  3. Test with real drivers: Simulated queries aren’t sufficient. The test suite exercises the Amazon Redshift JDBC and PostgreSQL ODBC drivers that your SQL clients depend on. This validates the same code paths your tools use in production.
  4. Event-driven, not scheduled: Tests run the moment a patch is applied. They don’t run on a fixed cron schedule. Patch applied, then test executed, then results delivered in minutes.
  5. Low operational overhead, minimal cost: The entire solution is serverless (AWS Lambda and AWS Fargate). There are no instances to manage and no agents to install. The Fargate task spins up only when a patch event fires, runs the test suite, and shuts down. You pay only for the compute each test run consumes.

Clean up

When you no longer need the automated test suite, delete the associated resources so you don’t incur ongoing costs.

  1. Delete any created prerequisites, if not needed.
    1. Amazon Redshift cluster (removes the managed secret).
    2. NAT gateway.
    3. VPC.
  2. Empty the Amazon S3 results bucket (AWS CloudFormation cannot delete non-empty buckets).
  3. Delete the image you installed in the Amazon ECR repository in step 1 of getting started.
  4. Delete the AWS CloudFormation stack to remove the Amazon ECS cluster, AWS Fargate task definition, security groups, VPC endpoints, Amazon S3 bucket, Amazon SNS topic, AWS Lambda function, and Amazon EventBridge rules created by the deployment. 
    aws cloudformation delete-stack --stack-name my-redshift-tests

Conclusion

Automated patch testing ensures consistent and predictable performance of your production workloads. By deploying Dev/QA clusters on the Current track with event-driven validation, you gain weeks of advance notice before patches reach production. The solution presented here provides comprehensive testing of JDBC drivers, ODBC drivers, catalog queries, and performance benchmarks. It requires zero manual intervention. Deploy it once, customize it for your workload, and gain confidence that the next Amazon Redshift patch will be validated before it matters.

To learn more about Amazon Redshift, explore the following resources:

About the author

Eva Donaldson

Eva Donaldson

Eva is a Senior Technical Account Manager (TAM) at AWS, specializing in Healthcare & Life Sciences customers. With 20+ years of experience as a data architect, engineer, and team manager, she focuses on designing automated data platforms and solutions that solve real business problems.

AI-powered performance recommendations for Amazon Redshift

Post Syndicated from Steve Phillips original https://aws.amazon.com/blogs/big-data/ai-powered-performance-recommendations-for-amazon-redshift/

Data platform teams running Amazon Redshift collect performance telemetry across system views like SYS_QUERY_HISTORY, SVV_TABLE_INFO, and SVV_ALTER_TABLE_RECOMMENDATIONS, plus Amazon CloudWatch metrics for capacity, query execution, and storage. The challenge is interpretation. Correlating a spike in QueryRuntimeBreakdown commit time with hundreds of small INSERT statements, or connecting high disk spill with undersized compute, takes deep expertise and hours of manual analysis.

In this post, you learn how to build an AI-powered solution that collects the telemetry, pre-computes performance signals, correlates them with CloudWatch, and uses Amazon Bedrock to generate prioritized recommendations. The source code is in the accompanying GitHub repository: sample-ai-performance-advisor-for-amazon-redshift.

The signal-based design is what makes this solution produce precise recommendations rather than generic advice. Instead of dumping raw system view output into the large language model (LLM) prompt, the collector pre-computes boolean and threshold-based findings, pairs them with CloudWatch correlations, and hands the model a structured context. The model then cross-references specific query IDs, table names, and metric values in its output.

Solution overview

Two AWS Lambda functions run on a 24-hour Amazon EventBridge schedule:

  • The collector Lambda runs 13 diagnostic SQL queries against Amazon Redshift Serverless and reads the workgroup’s Workload Management (WLM) configuration. It also collects CloudWatch metrics across capacity, query execution, WLM, connections, and storage. From these inputs, it computes the performance signals. Finally, it writes a telemetry JSON file to Amazon Simple Storage Service (Amazon S3).
  • The analyzer Lambda reads the telemetry from Amazon S3, builds a structured prompt with inline CloudWatch-to-signal correlations. Using the correlations, the analyzer calls Amazon Bedrock (Anthropic Claude Sonnet 4.6), and writes the resulting recommendations JSON back to Amazon S3.
  • An Amazon Simple Notification Service (Amazon SNS) topic sends an email summary of the top recommendations to subscribers.
AWS architecture diagram showing an automated Redshift analysis pipeline within the AWS Cloud. Amazon EventBridge triggers a “Collector” AWS Lambda function, which interacts bidirectionally with AWS Secrets Manager, Amazon Redshift, and Amazon CloudWatch to gather data. The Collector passes results to an “Analyzer” AWS Lambda function, which exchanges data with Amazon Bedrock and reads/writes to Amazon S3. The Analyzer then publishes to Amazon Simple Notification Service (SNS), which delivers an email notification.

Figure 1 – Architecture diagram

Prerequisites

Before deploying the solution, make sure the following are in place.

  • An Amazon Redshift Serverless workgroup with a database and query history.
  • An Amazon Redshift database administrator user (superuser). The collector reads views that only a superuser can query (SVV_TABLE_INFO, SVV_ALTER_TABLE_RECOMMENDATIONS, SVV_MV_INFO, SYS_SERVERLESS_USAGE, SYS_AUTO_TABLE_OPTIMIZATION).
    Store the admin credentials in AWS Secrets Manager and pass the secret ARN to the collector.
    Alternatively, have an existing superuser run ALTER USER "IAMR:redshift-performance-recommendations-role" CREATEUSER;
    once to grant the Lambda role superuser privileges.
  • Amazon Bedrock model access for the model of choice. For this solution, a us.anthropic.claude-* model is recommended for multi-region inference. The solution doesn’t depend on a single model.
  • The AWS Command Line Interface (AWS CLI) installed and configured, and a clone of the GitHub repository.

Create the supporting resources

You need an Amazon S3 bucket, an Amazon SNS topic, an AWS Secrets Manager secret, and an AWS Identity and Access Management (IAM) role before the Lambda functions can run.

Create the Amazon S3 bucket

The Amazon S3 bucket will host the output report.

  • Open the Amazon S3 console and choose Create bucket.
  • Enter a globally unique name (for example, amzn-s3-demo-bucket), keep the default settings, and choose Create bucket.

The collector writes telemetry JSON under the telemetry/ prefix and the analyzer writes recommendations under the recommendations/ prefix.

Create the Amazon SNS topic and subscription

Use Amazon SNS to generate notifications once reports are created.

  • Open the Amazon SNS console and choose Topics, Create topic.
  • Select Standard, and enter the name redshift-performance-recommendations.
  • Choose Create topic.
  • On the topic detail page, choose Create subscription.
  • Select Email as the protocol, enter your email address, and choose Create subscription.
  • Open the confirmation email from AWS Notifications and choose Confirm subscription.
Amazon SNS “Create topic” console page. The Type is set to Standard (selected over FIFO), and the Name field contains “redshift-performance-recommendations.” Annotation arrows highlight the Topics nav item, the Standard topic type, the entered name, and the “Create topic” button in the lower right. Optional sections for Encryption, Access policy, Delivery policy, Message delivery status logging, Tags, and Active tracing are collapsed below.

Figure 2 – Create SNS Topic

Store the admin credentials in AWS Secrets Manager

To avoid using hard-coded credentials, create an AWS Secrets Manager secret to connect to Amazon Redshift.

  • Open the AWS Secrets Manager console and choose Store a new secret.
  • Select Other type of secret, choose the Plaintext tab, and paste the following, replacing <ADMIN_PASSWORD> with the workgroup’s admin password: 
    {"username":"admin","password":"<ADMIN_PASSWORD>"}

  • Choose Next, enter redshift-performance-admin as the secret name, then choose Next, Next, and Store.
  • Copy the secret Amazon Resource Name (ARN) from the secret detail page. You pass it to the collector in a later step.
AWS Secrets Manager “Store a new secret” page, Step 1: Choose secret type. “Other type of secret” is selected, and the Plaintext tab shows the key-value pair {“username”:“admin”,“password”:“”}. The encryption key is set to aws/secretsmanager. Annotation arrows highlight the secret type selection, the plaintext credentials, and the “Next” button in the lower right.

Figure 3 – Create secret

Create the IAM role and attach the policy

The repository includes a trust policy in iam/trust-policy.json (allowing lambda.amazonaws.com to assume the role) and the least-privilege permission policy in iam/lambda-role-policy.json. Replace the <ACCOUNT_ID>, <REGION>, <YOUR_BUCKET>, and SNS topic ARN placeholders in the permission policy with your values, then create the role in the AWS Management Console or with this AWS CLI command:

aws iam create-role --role-name redshift-performance-recommendations-role \
    --assume-role-policy-document file://iam/trust-policy.json

aws iam put-role-policy --role-name redshift-performance-recommendations-role \
    --policy-name redshift-performance-policy \
    --policy-document file://iam/lambda-role-policy.json

The permission policy grants the Amazon Redshift Data API, Amazon S3, Amazon SNS, Amazon Bedrock, AWS Lambda invoke, AWS Secrets Manager, and Amazon CloudWatch Logs permissions that both Lambda functions require.

Deploy the Lambda functions

The collector source is in lambda/collector.py and it loads the SQL files in sql/ at runtime. The deployment package must contain both.

Package the collector

Open a terminal or shell window and execute a command to copy the collector code, supporting SQL into a folder and archive.

mkdir -p build/collector/sql
cp lambda/collector.py build/collector/
cp sql/*.sql build/collector/sql/
(cd build/collector && zip -qr ../collector.zip .)

Create the collector function

Using the AWS Management Console, navigate to AWS Lambda.

  • Choose Create function.

    AWS Lambda “Create function” console page with the “Configure custom execution role” panel open on the right. “Author from scratch” is selected, the function name is “redshift-performance-collector,” and the runtime is Python 3.14. Under Additional settings, the “Custom execution role” toggle is enabled, and the execution role list is set to “redshift-performance-recommendations-role.” Annotation highlights mark the Author from scratch option, function name, runtime, custom execution role toggle, the selected role, the Save button, and the “Create function” button.

    Figure 4 – Create AWS Lambda function

  • Select Author from scratch, enter redshift-performance-collector as the name, and select Python 3.14.
  • Expand Custom settings, toggle Custom execution role, choose an existing role, select redshift-performance-recommendations-role, and choose Save.
  • On the function page, choose Upload from, .zip file, and upload build/collector.zip.
  • In Runtime settings, select Edit, and set the Handler to collector.lambda_handler.

    Lambda console for the “redshift-performance-collector” function, Code tab. The code editor shows collector.py — a Python file that runs diagnostic SQL queries against Amazon Redshift Serverless, collects CloudWatch metrics, writes telemetry to Amazon S3, and invokes the analyzer Lambda. The Runtime settings section below shows the Handler highlighted as “lambda_function.lambda_handler,” with an arrow pointing to the Edit button and the “Upload from .zip file” option highlighted.

    Figure 5 – Set AWS Lambda handler

  • Choose Configuration, Edit, set timeout to 5 minutes, and memory to 256 MB.

    Lambda console for “redshift-performance-collector,” Configuration tab with “General configuration” selected. The panel shows Memory 128 MB, Ephemeral storage 512 MB, and Timeout 0 min 3 sec, with SnapStart set to None. Annotation arrows point to the General configuration menu item and the Edit button.

    Figure 6 – Set AWS Lambda timeout and memory

  • Under Configuration, select Environment variables, and add the following keys:
    • WORKGROUP: your Amazon Redshift Serverless workgroup name.
    • NAMESPACE_NAME: the namespace the workgroup belongs to.
    • DATABASE: dev (or your target database).
    • BUCKET: the Amazon S3 bucket name you created earlier.
    • SECRET_ARN: the AWS Secrets Manager secret ARN you copied earlier.
    • ANALYZER_FN: redshift-performance-analyzer.

Package and create the analyzer

Repeat the same steps for the analyzer, using lambda/analyzer.py with a 15-minute timeout:

(cd lambda && zip -q ../build/analyzer.zip analyzer.py)

Use the Lambda console to create redshift-performance-analyzer with handler analyzer.lambda_handler, timeout 15 minutes, memory 256 MB, the same execution role, and these environment variables:

  • BUCKET: the same Amazon S3 bucket.
  • SNS_TOPIC: the SNS topic ARN.
  • MODEL_ID: us.anthropic.claude-sonnet-4-6.

The analyzer creates the Amazon Bedrock client with read_timeout=600 and max_tokens=16384 to handle large prompts and long responses. Anthropic Claude inference on a full telemetry payload typically takes 2–4 minutes.

How the signals and the prompt work

You don’t write any custom code for signal computation or prompt construction. Both computation and construction live in the repository.

The compute_signals() function in lambda/collector.py scans the telemetry for Boolean and threshold-based anti-patterns. At the table level, it looks for row skew, ghost rows, stale statistics, unsorted data, sub-optimal sort or distribution keys, and oversized VARCHAR columns. It also flags runtime and workload issues such as disk spill, small-insert bursts, high Data Definition Language (DDL) executions, and unoptimized COPY file size. Beyond that, it catches Amazon Redshift Spectrum queries that fail to prune partitions and data sharing materialized views doing full refresh. It also flags WLM configurations that lack Query Monitoring Rules (QMR), such as limits on blocks spilled to disk and query execution time. The full set of signals and thresholds is defined inline in the function. To tune a threshold or add a custom signal, edit this function and redeploy.

The build_prompt() function in lambda/analyzer.py constructs the Amazon Bedrock prompt in four sections. The first section lists the triggered signals. The second adds CloudWatch metrics, annotated with >> CORRELATION lines that pair each signal with its supporting metric. The third includes the filtered supporting data, limited to the table and query rows that triggered a signal. The fourth gives explicit instructions to return a pipe delimited text where every recommendation references specific table names, query IDs, and metric values. This structure is why the model produces targeted output rather than generic best-practice advice.

Schedule daily runs

Use the Amazon EventBridge console to trigger the collector every 24 hours.

  • Open the EventBridge console and choose Schedules under Scheduler, Create schedule.
  • Enter the name redshift-performance-daily for Schedule name, toggle Recurring schedule and Rate-based schedule.
  • Under Rate expression, enter 24 and select hours.
  • For Flexible time window, choose Off, and select Next.
    Amazon EventBridge Scheduler “Create schedule” page, Step 1: Specify schedule detail. The schedule name is “redshift-performance-daily.” Under Schedule pattern, “Recurring schedule” and “Rate-based schedule” are selected, with a rate expression of 24 hours, and the time zone set to (UTC-06:00) America/Denver. Annotation highlights mark the Schedules nav item, the recurring/rate-based selections, the rate expression, and the Next button.

    Figure 7 – Create Amazon EventBridge schedule

     

  • On the Select target page, choose AWS Lambda, select the redshift-performance-collector function, and choose Next.

    EventBridge Scheduler “Create schedule” page, Step 2: Select target. “Templated targets” is selected and the AWS Lambda “Invoke” target is chosen from the grid of target options. In the Invoke section, the Lambda function list is set to “redshift-performance-collector” with an empty JSON payload. Annotation highlights mark the Templated targets toggle, the AWS Lambda Invoke target, the selected function, and the Next button.

    Figure 8 – Select Amazon EventBridge schedule target

  • Accept the defaults for Settings and select Next. EventBridge automatically adds a resource-based permission on the Lambda function so the rule can invoke it.
  • Choose Create schedule.

Run it once and review the output

Invoke the collector manually to confirm the pipeline works end-to-end.

  • In the Lambda console, open the redshift-performance-collector function and choose Test. Create a test event named manual with the body {} and choose Test.

    Lambda console for “redshift-performance-collector,” Test tab. A new test event named “manual” is being configured with Invocation type set to Synchronous, event sharing set to Private, the “Hello World” template selected, and an empty {} Event JSON body. Annotation arrows point to the function in the left nav, the Synchronous option, the event name, the Event JSON field, and the Test button.

    Figure 9 – Test end-to-end workflow

  • The function completes in under a minute. Check the Monitor tab for the invocation log via the CloudWatch live logs link.
  • In the Amazon S3 console, open your bucket. Confirm that the telemetry/ prefix contains a JSON file with the current timestamp.
  • Within 2–4 minutes, the analyzer publishes a message to the SNS topic. Check the email address you subscribed for the summary with the top 10 recommendations. Confirm that the recommendations/ prefix in Amazon S3 contains the full JSON.

Each recommendation has a priority (critical, high, medium, low) and a category (query_optimization, table_design, capacity, wlm, maintenance, or ingestion). It also includes a signal_source that names the signals and CloudWatch metrics that triggered it, a plain-language explanation, a specific SQL or configuration action, and an expected impact estimate.

Email notification from AWS Notifications with the subject “Redshift performance: 3 critical, 5 high, 4 medium, 2 low (8 signals)” highlighted. The body is a plain-text “Amazon Redshift Performance Recommendations” report listing workgroup, namespace, database, analysis time, and 14 recommendations. Two critical items are shown for the game_events table: fixing extreme row-skew via DISTSTYLE ALL, and eliminating non-encoded columns with column compression, each with a category, source, explanation, SQL action, and expected impact.

Figure 10 – Sample analyzer emailed output

Best practices

  • Tune thresholds to your workload. The default thresholds in compute_signals() come from the Amazon Redshift operational review playbook. For high-velocity ingestion or small-cluster environments, consider lowering the small-insert threshold, widening the stale-statistics window, or adding custom signals for your own tables.
  • Keep the signal-to-metric correlations current. When you add a signal, also add a matching correlation in build_correlations(). The inline >> CORRELATION lines are what make the model connect an infrastructure metric to an application-level symptom.
  • Review recommendations before you act. The analyzer produces prioritized suggestions, but VACUUM, ANALYZE, and ALTER TABLE actions change table state. Read the explanation and action on each recommendation, validate the SQL against your schema, and run it during a maintenance window.

Cleaning up

To avoid ongoing charges, delete the resources you created for this solution:

  • The two AWS Lambda functions: redshift-performance-collector and redshift-performance-analyzer.
  • The Amazon EventBridge rule: redshift-performance-daily.
  • The Amazon SNS topic and its email subscription: redshift-performance-recommendations.
  • The Amazon S3 bucket, including the telemetry/ and recommendations/ objects.
  • The AWS Secrets Manager secret: redshift-performance-admin.
  • The IAM role and its inline policy: redshift-performance-recommendations-role.

Conclusion

You now have a daily performance review for Amazon Redshift Serverless that runs entirely on AWS Lambda, stores every run in Amazon S3, and delivers prioritized recommendations by email. The signal-based prompt pattern keeps the Amazon Bedrock cost low and the recommendations specific to your workload.

To learn more, see the following resources:

About the authors

Steve Phillips

Steve Phillips

Steve is a Principal Technical Account Manager and Analytics specialist at AWS in the North America region. Steve currently focuses on data warehouse architectural design, AI/ML data foundations, data lakes, data ingestion pipelines, and cloud distributed architectures.

Richard Raseley

Richard Raseley

Richard is a Senior Technical Account Manager in North America who works with Games customers. He is passionate about applying his background in automation, cloud computing, networking, and storage to help customers build AI solutions.

Getting your SMS short code production-ready with AWS End User Messaging

Post Syndicated from Harshvardhan Chunawala original https://aws.amazon.com/blogs/messaging-and-targeting/getting-your-sms-short-code-production-ready-with-aws-end-user-messaging/

Getting your Short Message Service (SMS) short code production-ready requires you to configure the Amazon Web Services (AWS) infrastructure that controls how your messages are sent, monitored, and protected. You have provisioned your short code, and it is active on carrier networks. In this post, we walk through that setup using AWS End User Messaging SMS, covering 12 configuration steps from compliance through phased traffic migration. Total estimated time is 2 to 4 hours of configuration plus 1 to 3 business days for limit increase approvals.mess

The guide to SMS short codes with AWS End User Messaging covers the application and registration process up through provisioning. This post picks up from that point and provides an operational readiness walkthrough that takes you from “Active” status to confidently sending your first production message, including a final validation step to confirm readiness.

The following diagram shows the end-to-end message flow and event routing architecture covered in this walkthrough.

End-to-end SMS short code architecture showing message flow from sender through AWS End User Messaging SMS to carriers and recipient handsets, with event routing to Amazon CloudWatch, Amazon Simple Notification Service (Amazon SNS), and Amazon Data Firehose destinations

Prerequisites

You need the following to follow along with this walkthrough:

  1. An AWS account with access to the AWS End User Messaging SMS console.
  2. A short code with Active status in the AWS Management Console (carrier provisioning finished).
  3. Permissions to create AWS Identity and Access Management (IAM) roles, Amazon CloudWatch Log Groups, and Amazon Simple Notification Service (Amazon SNS) topics.
  4. AWS Command Line Interface (AWS CLI) v2 or an AWS SDK installed and configured.
  5. Your approved registration documentation, including the service name, keyword responses, and message templates submitted to carriers.

Step 1: Verify your short code is active and delivering

Navigate to the AWS End User Messaging SMS console, choose Phone numbers, and locate your provisioned short code. Confirm that the status shows Active, then send a test message to a phone number you control using the SendTextMessage API or the console test feature. Verify delivery on your handset.

Carrier-side activation can take up to 24 to 48 hours to fully propagate across all networks after provisioning finishes. If the console shows Active but your test message does not arrive, submit a support case so the team can verify propagation status with the carrier.

You can also verify using the AWS CLI:

aws pinpoint-sms-voice-v2 send-text-message \
    --destination-phone-number "+15555550100" \
    --origination-identity "12345" \
    --message-body "Test message from short code" \
    --message-type TRANSACTIONAL \
    --configuration-set-name "prod-otp-shortcode"
# Replace +15555550100 with your test phone number, 12345 with your short
# code, and prod-otp-shortcode with your configuration set name from Step 3.

Step 2: Configure keywords and verify message compliance

US carriers require every short code to respond to HELP and STOP keywords. You defined these during your registration, and this step confirms they are configured correctly in your account.

In the SMS console, choose Phone numbers, select your short code, and choose the Keywords tab. Verify that STOP returns the opt-out response you submitted during registration, and that HELP returns your support contact response (which must include a phone number or email). Add any custom keywords your use case requires, such as YES for double opt-in confirmation flows. You can manage keywords programmatically using the PutKeyword API.

To add or update a keyword programmatically:

aws pinpoint-sms-voice-v2 put-keyword \
    --origination-identity "12345" \
    --keyword "YES" \
    --keyword-message "You have confirmed your subscription to Acme Health Alerts. Msg&data rates may apply. Reply STOP to opt out." \
    --keyword-action AUTOMATIC_RESPONSE
# Replace 12345 with your short code, YES with your custom keyword, and the
# keyword-message text with your approved response.

To verify your current keyword configuration:

aws pinpoint-sms-voice-v2 describe-keywords \
    --origination-identity "12345"
# Replace 12345 with your short code.

Beyond keyword configuration, carrier compliance does not end at registration approval. The content you send in production must stay aligned with what carriers reviewed and approved. Here is what to keep consistent.

Use the exact brand or program name from your approved registration across all keyword responses, confirmation messages, and outbound templates. If carriers approved your registration under “Acme Health Alerts,” every message your short code sends should reference that name. Mixing variations creates inconsistencies that auditors flag during reviews. For example, do not use the company name in one message and the product name in another.

Your HELP, STOP, and confirmation responses must match the templates submitted during registration. Do not add or remove opt-out language, change frequency disclosures, or alter customer care contact details post-approval without updating the registration through a support case. If your organization operates multiple domains, use the domain documented in the registration. For example, you might have one domain for the application and another for marketing. Carrier reviewers cross-reference message content, opt-in screenshots, and privacy policy URLs with what was submitted.

Humans conduct carrier reviews, and message content that is concise and limited to the essentials is reviewed consistently. All messages must remain under 160 characters.

Step 3: Create a configuration set with event destinations

A configuration set controls where your SMS delivery events are streamed and which event types are captured. Without one, you are limited to the basic events that AWS End User Messaging SMS sends to Amazon EventBridge by default. These default events omit recipient details and full carrier response context.

Create a configuration set with a descriptive name such as prod-otp-shortcode or marketing-sc-us. Then create at least one event destination. The three main options are Amazon CloudWatch Logs (for operational monitoring and alarming), Amazon SNS (for real-time event fanout to downstream systems), and Amazon Data Firehose (for durable archival and analytics).

Amazon Data Firehose typically delivers to an Amazon Simple Storage Service (Amazon S3) bucket, where you can query delivery history using Amazon Athena for compliance audits or delivery pattern analysis.

# Create the configuration set
aws pinpoint-sms-voice-v2 create-configuration-set \
    --configuration-set-name "prod-otp-shortcode"

# Add a CloudWatch Logs event destination
aws pinpoint-sms-voice-v2 create-event-destination \
    --configuration-set-name "prod-otp-shortcode" \
    --event-destination-name "otp-delivery-logs" \
    --matching-event-types TEXT_DELIVERED TEXT_FAILED TEXT_QUEUED TEXT_CARRIER_UNREACHABLE TEXT_TTL_EXPIRED \
    --cloud-watch-logs-destination '{
        "IamRoleArn": "arn:aws:iam::123456789012:role/SMSEventsToCloudWatch",
        "LogGroupArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/sms/prod-otp-shortcode"
    }'
# Replace prod-otp-shortcode with your configuration set name, otp-delivery-logs
# with a descriptive destination name, and the ARN values with your IAM role ARN
# (must have logs:PutLogEvents permission) and CloudWatch Log Group ARN.

Important: When sending messages with SendTextMessage, always specify your ConfigurationSetName parameter so events route to the appropriate destination.

Required event types

Event type Description
TEXT_DELIVERED Message successfully delivered to recipient handset.
TEXT_FAILED Message delivery failed.
TEXT_QUEUED Message accepted and queued for delivery.
TEXT_CARRIER_UNREACHABLE Carrier network unreachable.
TEXT_TTL_EXPIRED Message expired before delivery.

For a detailed walkthrough of configuration sets including multi-tenant architectures, see How to send SMS using configuration sets with AWS End User Messaging.

Step 4: Create a phone pool and associate your short code

A pool is a logical container that groups origination identities and controls routing behavior. Creating one gives you deterministic control over which number sends your messages and how opt-outs are enforced.

# Create the pool
aws pinpoint-sms-voice-v2 create-pool \
    --origination-identity "12345" \
    --iso-country-code "US" \
    --message-type TRANSACTIONAL

# Disable shared routes so only your short code is used
aws pinpoint-sms-voice-v2 update-pool \
    --pool-id "pool-1234567890abcdef0" \
    --shared-routes-enabled false
# Replace 12345 with your short code, US with your destination country code,
# and pool-1234567890abcdef0 with the Pool ID returned by create-pool.

Configuration parameters

Parameter Recommended value Rationale
Pool name us-otp-pool Descriptive, environment-prefixed.
SharedRoutesEnabled False Prevents fallback to shared routes; only your short code is used.
Opt-out list Associate one Manages opt-out state per use case.
IsoCountryCode US Restricts to destination country your short code serves.

If you operate multiple use cases on separate short codes, create a dedicated pool for each. For example, use one short code for one-time password (OTP) traffic and another for transactional notifications. This isolation means a recipient opting out of marketing messages does not lose access to authentication codes.

Step 5: Request your throughput increase

Short codes start at a default of 100 messages per second (MPS). If your production volume will exceed this, request an increase before your launch date rather than after traffic is flowing.

Create a case in the AWS Support Center, choose Service limit increase, then choose End User Messaging SMS. Provide your short code phone number, requested MPS, use case description, and expected peak volume. Allow 1 to 3 business days for processing.

To estimate your required MPS:

Required MPS = (Peak hourly volume / 3,600) x 2

Short codes support scaling to thousands of MPS, so start with a value that covers your expected peak and request further increases as traffic grows.

Step 6: Request a spending limit increase

AWS accounts have a default monthly SMS spending limit. To keep delivery uninterrupted at your expected volume, request an increase that accommodates your projected monthly spend before you begin sending.

Create a support case under Service limit increase > End User Messaging SMS > Account Spend Threshold. Provide your estimated monthly spend, use case description, and website URL.

For details, see Requesting increases to your monthly SMS spending quota.

Step 7: Restrict destination countries

If your short code serves a single country (US-only, for example), restrict sending to that country. This protects your account from artificially inflated traffic (SMS pumping). In pumping attacks, messages are routed to international premium-rate numbers, generating significant charges.

In the SMS console, navigate to Account settings, then choose Countries and keep only the countries you intend to send to. The pool-level IsoCountryCode restriction from Step 4 provides an additional enforcement layer at the sending path. Combining account-level country restrictions with pool-level country codes gives you two independent controls that both must be satisfied before a message is sent internationally.

For a detailed walkthrough on SMS fraud prevention controls, see Defending against SMS pumping: new AWS features to help combat artificially inflated traffic.

Step 8: Set up monitoring and alarms

With event destinations configured in Step 3, build proactive alerting that surfaces delivery trends before they affect your end users.

Alarm Metric / Source Threshold
Delivery success rate CloudWatch SMS metrics Alert when below 95%.
Spend threshold CloudWatch billing metric Alert at 80% of monthly limit.
Delivery failures Amazon EventBridge rule on TEXT_FAILED Route to Amazon SNS topic or AWS Lambda.
Carrier unreachable Amazon EventBridge rule on TEXT_CARRIER_UNREACHABLE Route to Amazon SNS topic or AWS Lambda.

Build a CloudWatch dashboard showing messages sent per minute, success versus failure breakdown, and spend accumulation over time.

You can also configure Amazon EventBridge to notify you of registration status changes. AWS End User Messaging SMS publishes events for statuses including REQUIRES_UPDATES, REVIEWING, and PROVISIONING, which is useful if a carrier requests changes during a proactive audit after your short code is already active.

For metric details, see Monitoring SMS activity with Amazon CloudWatch.

Step 9: Track OTP verification success (if applicable)

If your short code delivers OTP or two-factor authentication (2FA) codes, track end-to-end verification success in addition to carrier delivery receipts. A “delivered” status at the carrier level does not confirm the end user received and entered the code.

Tracking verification rates gives you insight into latency patterns when codes expire before arrival, geographic delivery trends, and opportunities to improve conversion. Some use cases involve asynchronous processing where several minutes of computation occur before the SMS is sent. For these, measure the full round-trip from the triggering action to message delivery. This separates application-side latency from carrier-side delivery latency.

For implementation guidance, see Track OTP success with AWS End User Messaging SMS feedback.

Step 10: Set up cost visibility

SMS costs include AWS charges plus per-message carrier surcharges. Setting up cost visibility from day one lets you track spend trends, catch anomalies early, and optimize over time.

Start by activating AWS Cost Explorer and creating a cost allocation tag for your SMS workload. Then configure an AWS Budget with threshold alerts. For example, you might notify at 80% of projected monthly spend. This gives you advance warning of unexpected cost increases, whether from traffic spikes, retry loops, or blocked-country leakage.

Step 11: Plan your traffic migration

A phased rollout validates delivery performance at each stage before you increase volume.

Start with a canary phase (Day 1 to 3) where you route 5 to 10% of traffic to the short code and monitor delivery rates, latency, and event logs. Move to a ramp phase (Day 3 to 7) at 50%, validating throughput and carrier-level delivery across your recipient base. Finish the full migration (Day 7+) at 100%. Decommission your previous origination identity only after confirming stability for at least 48 hours.

Step 12: Validate production readiness and send

Before declaring your short code production-ready, run through the following validation checks:

  1. Confirm your CloudWatch dashboard shows events flowing for TEXT_DELIVERED and TEXT_FAILED (from Step 3).
  2. Send a test message that triggers your STOP keyword. Verify the correct opt-out response is returned and the phone number appears in your opt-out list.
  3. Send a test message that triggers your HELP keyword. Verify the response matches your approved registration.
  4. Check your MPS quota in the support case response (from Step 5). Confirm it matches or exceeds your calculated peak.
  5. Review your country restrictions (from Step 7). Attempt to send a message to a blocked country and confirm it is rejected.
  6. Verify your CloudWatch alarm fires by temporarily lowering the threshold, or by checking that the alarm state is not INSUFFICIENT_DATA.

After all six checks pass, you are ready to begin your phased migration (Step 11) and scale to full production traffic. At this point, your short code is configured, monitored, compliant, and protected.

Automate with a validation script

You can use an AI coding assistant such as Kiro to generate a validation script tailored to your environment. Try a prompt like: “Write a boto3 script that validates my SMS short code is production-ready by checking Active status, HELP/STOP keywords, configuration set existence, and pool association using the pinpoint-sms-voice-v2 client.”

Refine the prompt with specifics from the following reference implementation, such as exact API names, filter parameters, and output format, to match your requirements.

The following script is an example of what that output looks like:

import boto3
import sys

SHORT_CODE = "12345"  # TODO: Replace with your short code (e.g., "67890")
POOL_ID = "pool-1234567890abcdef0"  # TODO: Replace with your pool ID from Step 4
CONFIG_SET_NAME = "prod-otp-shortcode"  # TODO: Replace with your configuration set name from Step 3

client = boto3.client("pinpoint-sms-voice-v2")

# Note: For accounts with many resources, implement NextToken pagination
# on describe_* calls. This script assumes results fit in a single page.


def check_short_code_active():
    """Step 1: Verify short code is Active."""
    response = client.describe_phone_numbers(
        Filters=[
            {"Name": "status", "Values": ["ACTIVE"]},
            {"Name": "number-type", "Values": ["SHORT_CODE"]}
        ]
    )
    numbers = [
        n for n in response["PhoneNumbers"]
        if n["PhoneNumber"] == SHORT_CODE
    ]
    assert len(numbers) > 0, f"Short code {SHORT_CODE} not found or not Active"
    print(f"[PASS] Short code {SHORT_CODE} is Active")


def check_keywords_configured():
    """Step 2: Verify HELP and STOP keywords exist."""
    response = client.describe_keywords(OriginationIdentity=SHORT_CODE)
    keyword_names = [kw["Keyword"].upper() for kw in response["Keywords"]]
    assert "STOP" in keyword_names, "STOP keyword not configured"
    assert "HELP" in keyword_names, "HELP keyword not configured"
    print("[PASS] HELP and STOP keywords configured")


def check_configuration_set():
    """Step 3: Verify configuration set exists."""
    response = client.describe_configuration_sets(
        ConfigurationSetNames=[CONFIG_SET_NAME]
    )
    assert len(response["ConfigurationSets"]) > 0, f"Configuration set {CONFIG_SET_NAME} not found"
    print(f"[PASS] Configuration set '{CONFIG_SET_NAME}' exists")


def check_pool_association():
    """Step 4: Verify pool exists and short code is associated to it."""
    response = client.describe_pools(PoolIds=[POOL_ID])
    assert len(response["Pools"]) > 0, f"Pool {POOL_ID} not found"

    # Verify short code is associated to the pool
    assoc_response = client.list_pool_origination_identities(PoolId=POOL_ID)
    identities = [
        oi["OriginationIdentity"]
        for oi in assoc_response["OriginationIdentities"]
    ]
    assert any(SHORT_CODE in oi for oi in identities), \
        f"Short code {SHORT_CODE} not associated with pool {POOL_ID}"
    print(f"[PASS] Pool '{POOL_ID}' exists and short code is associated")


if __name__ == "__main__":
    checks = [
        check_short_code_active,
        check_keywords_configured,
        check_configuration_set,
        check_pool_association,
    ]
    for check in checks:
        try:
            check()
        except Exception as e:
            print(f"[FAIL] {check.__doc__} - {e}")
            sys.exit(1)
    print("\nAll validation checks passed. Ready for production traffic.")

Cleaning up

If you created test resources while following this walkthrough, you can delete them through the AWS End User Messaging SMS console or with the API to avoid confusion with your production configuration. This includes a test configuration set, test pool, or test event destinations used for validation. Do not delete your production configuration set, pool, or keyword settings.

If you requested a test-level MPS increase or spending limit for validation, update these to your production values through a new support case before going live.

Quick reference checklist

Step Action Key API / Service
1 Verify short code is Active and test delivery SendTextMessage
2 Configure keywords and verify message compliance PutKeyword
3 Create configuration set with event destinations CreateConfigurationSet
4 Create pool and associate short code CreatePool, AssociateOriginationIdentity
5 Request MPS increase for expected throughput AWS Support
6 Request spending limit increase AWS Support
7 Restrict destination countries Console / UpdateAccount
8 Set up CloudWatch alarms and dashboards Amazon CloudWatch
9 Track OTP verification success (if applicable) SMS Feedback events
10 Set up cost visibility AWS Cost Explorer, AWS Budgets
11 Plan phased traffic migration Application-level routing
12 Validate production readiness and send All of the preceding

Conclusion

In this post, we walked through how to configure a newly provisioned SMS short code for production use with AWS End User Messaging SMS. The 12 steps cover keyword verification, message compliance, event monitoring, throughput planning, country restrictions, cost visibility, phased traffic migration, and a final production validation.

You can adapt the sequence to your specific use case and volume profile. For the full registration and application process, see A guide to SMS short codes with AWS End User Messaging. To start configuring, navigate to the AWS End User Messaging SMS console. For the full API reference, see the AWS End User Messaging SMS documentation.

About the author

Integrating Event Source Mappings with AWS Lambda tenant isolation mode

Post Syndicated from Anton Aleksandrov original https://aws.amazon.com/blogs/compute/integrating-event-source-mappings-with-aws-lambda-tenant-isolation-mode/

Building event-driven multi-tenant SaaS applications typically requires compute isolation between tenants to prevent data leakage, maintain security boundaries, and ensure compliance. Traditionally, you had to choose between two approaches: sharing execution environments across tenants (risking cross-tenant contamination of in-memory state) or managing separate Lambda functions per tenant (which introduces operational overhead, increasing costs, and complicating deployments). Both approaches required you to make trade-offs between security, operational complexity, and cost efficiency.

AWS Lambda tenant isolation mode with Event Source Mappings addresses this trade-off. This approach reduces operational complexity, improves your security posture, and removes the need to manage separate functions per tenant, all while maintaining strict compute-level isolation boundaries. You can now build event-driven architectures using services like Amazon SQS and Amazon EventBridge where each tenant’s workloads run in dedicated execution environments, but you manage only a single Lambda function.

In this post, you’ll learn how to propagate tenant identity from event payloads, implement IAM permissions for tenant-isolated invocations, apply validation strategies to verify tenant context, and use a lightweight routing mechanism that invokes tenant-isolated backends. Complete sample code demonstrating this pattern is available in the AWS samples repository.

Understanding Lambda tenant isolation mode

AWS Lambda tenant isolation mode extends Lambda’s execution model by introducing tenant-aware routing of invocations. Instead of reusing execution environments across all invocations of a function, Lambda associates each execution environment with a specific tenant identifier. When a new request is received, Lambda routes it to an existing environment for that specific tenant or creates a new one if none exists.

Tenant Isolation ArchitectureFigure 1. Using Lambda tenant isolation mode for compute isolation

This simplifies how you build multi-tenant SaaS systems, while maintaining isolation boundaries at the compute level. Execution environments are never shared across tenants but still reused within the same tenant for maximum efficiency. That means you can safely cache tenant-specific configurations, such as feature flags or database connection strings, without adding isolation logic manually in your code.

To use the tenant isolation mode, every invocation must include a tenant ID parameter. For synchronous, direct invocations, such as originating from Amazon API Gateway or AWS SDKs, you pass it using the X-Amz-Tenant-Id header, as described in the launch blog and service documentation. Lambda service uses this header to route the invocation to tenant-specific execution environments. Inside your function handler, the tenant ID is available using the context.tenantId property, so you can implement tenant-aware logic.

port const handler = async (event, context) => {
    const tenantId = context.tenantId;

    // Tenant-specific business logic here
    console.log(`Processing request for tenant: ${tenantId}`);
};

Figure 2. Accessing tenant ID from function handler.

When using API Gateway, you can extract the tenant ID value from incoming request metadata, such as HTTP headers, path parameters, query parameters, or JWT claims, and map it directly to the downstream X-Amz-Tenant-Id in the API Gateway integration request configuration. See the launch blog for detailed guidance.

This model works well for direct, synchronous invocations. However, many serverless applications rely on event-driven patterns, where Lambda is invoked through Event Source Mappings.

Using tenant isolation mode with event sources

Many serverless applications use event-driven architectures built on services like Amazon SQS, Amazon EventBridge, Amazon Kinesis, or Amazon DynamoDB Streams. In these cases, Lambda is invoked by an Event Source Mapping (ESM), which polls the event source and invokes your function when new events arrive.

With these services, you’ll commonly find the tenant identity embedded in the event payload or metadata – for example, in an SQS message body or EventBridge event detail. Each event source has its own payload schema. Below are example payloads when using SQS and EventBridge, where you can see the tenantId parameter present in the payload.

SQS message body:

{
    "tenantId": "TenantA",
    "orderId": "ord-12345",
    "eventType": "ORDER_PLACED",
    "payload": { ... }
}

EventBridge event detail:

{
    "source": "com.myapp.orders",
    "detail-type": "OrderPlaced",
    "detail": {
        "tenantId": "TenantA",
        "orderId": "ord-12345"
    }
}

However, event sources don’t provide a built-in mechanism to map message properties to HTTP headers. As a result, if you try to invoke a function with tenant isolation mode enabled directly from an event source mapping, it fails because the tenant ID isn’t propagated as the X-Amz-Tenant-Id header. The following section describes how to address this and integrate ESMs with tenant-isolated Lambda functions.

Propagating tenant identity with Event Source Mappings

To propagate tenant identity from ESM messages, you can introduce a routing component – a lightweight Lambda function that sits between the event source and your tenant-isolated backend function. Your routing function receives events from the ESM, extracts the tenant ID from each message, and invokes your backend function using the Lambda Invoke API, passing the required X-Amz-Tenant-Id header. See the following diagram for an example architecture using SQS ESM.

Lambda with tenant isolated SQS

Figure 3. Propagating tenant ID from SQS messages to Lambda with tenant isolation mode enabled

You don’t need to enable tenant isolation mode on the routing function itself – it acts as a stateless dispatcher. Your multi-tenant backend function, which contains your core business logic, runs with tenant isolation mode enabled and receives properly scoped, tenant-aware invocations. This pattern keeps tenant isolation at the backend layer while preserving a shared event ingestion model.

The following example illustrates a routing function that processes incoming SQS messages, extracts the tenant ID from each message body, and invokes your backend function with the appropriate tenant context. This example assumes MessageGroupId is used to carry the tenant identifier, which ensures messages from the same tenant are processed in order when you’re using FIFO queues.

export const handler = async (event) => {
    for (const record of event.Records) {
        const body = record.body;
        const messageGroupId = record.attributes?.MessageGroupId;

        const command = new InvokeCommand({
            FunctionName: BACKEND_FUNCTION_NAME,
            InvocationType: 'Event',
            TenantId: messageGroupId,
            Payload: Buffer.from(body)
        });

        await lambdaClient.send(command);
    }
}

Figure 4. Routing SQS messages to a Lambda function with tenant isolation mode enabled

The following example illustrates how you can achieve the same routing functionality when processing EventBridge events.

export const handler = async (event) => {
    const tenantId = event.detail?.tenantId;

    if (!tenantId) {
        throw new Error(`Missing tenantId in EventBridge event: ${JSON.stringify(event)}`);
    }

    const command = new InvokeCommand({
        FunctionName: BACKEND_FUNCTION_NAME,
        InvocationType: 'Event',
        TenantId: tenantId,
        Payload: JSON.stringify(event.detail),
    });

    await lambdaClient.send(command);
};

Figure 5. Routing EventBridge events to a Lambda function with tenant isolation mode enabled

IAM permissions

Your routing function’s execution role needs permission to:

  1. Poll the event source: You can apply this policy either to your function execution role or as a resource policy on the event source itself.
  2. Invoke the downstream backend function: Additionally, your router function requires the lambda:InvokeFunction permission scoped to your backend function ARN.

Below is an example execution role policy to allow the router function to poll from an SQS queue

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "sqs:ReceiveMessage",
            "sqs:DeleteMessage",
            "sqs:GetQueueAttributes"
        ],
        "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }]
}

Below is an example execution role policy to allow the router function to invoke the backend function

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": "lambda:InvokeFunction",
        "Resource": "arn:aws:lambda:us-east-1:123456789012:function:my-backend-function"
    }]
}

Figure 6. IAM permissions used for implementing the tenant ID router function mechanism.

Best practices and considerations

When implementing the pattern described in this post, keep these important considerations in mind regarding validation, scaling, and overall system design.

Validate tenant identity before invocation. Tenant identity comes from event payloads, you shouldn’t automatically assume it’s trustworthy. Here’s how to protect your system:

  • Validate incoming payloads and reject messages with missing, malformed, or unauthorized tenant IDs at the routing layer before invoking your backend function
  • Maintain an authoritative tenant registry and validate incoming tenant IDs against it
  • Use dead-letter queues (DLQs) on your SQS queues to capture messages that fail validation for investigation and replay
  • When using EventBridge Pipes, use the enrichment step to validate or normalize tenant IDs before they reach your routing function
  • Enable partial batch response for applicable ESMs, such as SQS, so your routing function can report individual message failures without failing the entire batch

Plan for scaling considerations. Tenant isolation mode creates separate execution environments per tenant. This can increase the number of cold starts compared to shared environments. Each tenant consumes concurrency independently, so monitor your usage and request quota increases as your tenant base grows.

Optimize the routing function. Your routing function introduces an additional invocation segment. Use asynchronous invocation (InvocationType: ‘Event’) to reduce idle waiting time and size your function accordingly.

Understand permission boundaries. Tenants share your backend function’s execution role. If you need fine-grained per-tenant permissions, consider propagating tenant-scoped credentials (for example, using AWS STS AssumeRole) from the upstream segment.

Sample code

A complete, deployable sample project demonstrating this pattern – including SQS routing functions, a tenant-isolated backend function, and AWS SAM infrastructure – is available in this GitHub repository. Follow the instructions in README.md to provision the sample project in your account

Conclusion

Lambda tenant isolation mode introduces cross-tenant compute isolation for your multi-tenant SaaS applications by routing each invocation to a tenant-specific execution environment. When you combine this with event-driven architectures built on services like SQS, EventBridge, and Kinesis, the routing function pattern described in this post allows you to propagate tenant identity from event payloads and invoke your tenant-isolated backend with the correct context.

This approach extends tenant isolation mode to your asynchronous workloads without changing your core business logic. You retain per-tenant execution environment isolation while continuing to use Lambda’s native event source integrations, scaling model, and operational tooling. Together, these patterns provide you with a practical foundation for building secure, scalable, event-driven multi-tenant SaaS applications on AWS.

Next steps: Consider extending this pattern to other event sources like Kinesis Data Streams or DynamoDB Streams. You can also explore combining this approach with AWS Step Functions for orchestrating complex multi-tenant workflows while maintaining tenant isolation boundaries.

Follow below links to learn more:

Building highly available Oracle databases with Amazon FSx for NetApp ONTAP

Post Syndicated from Vignyanand Penumatcha original https://aws.amazon.com/blogs/architecture/building-highly-available-oracle-databases-with-amazon-fsx-for-netapp-ontap/

Oracle databases power mission-critical enterprise applications, making their continuous availability essential for business operations. Traditional Oracle high availability (HA) solutions require complex clustering software, expensive shared storage arrays, and specialized database administration teams. These conventional approaches often introduce single points of failure while demanding significant operational overhead.

Modern cloud architectures offer a transformative approach that combines Amazon FSx for NetApp ONTAP (FSxN) with Amazon EC2 Auto Scaling groups, automated AMI creation, AWS Lambda-driven orchestration, and AWS Systems Manager Parameter Store (SSM Parameter). This solution removes traditional Oracle HA complexities while delivering enterprise-grade availability, automated recovery, and makes sure new instances launch with the latest Oracle configuration.

This post shows how to build a highly available Oracle database architecture using FSxN shared storage, Auto Scaling groups with dynamic AMI updates, and serverless orchestration to help reduce recovery times with current configurations.

Solution overview

The solution uses multiple AWS services working together to create a comprehensive high availability architecture. FSxN Multi-AZ provides persistent shared storage spanning availability zones for Oracle database files, software, and configurations, so that data remains accessible when EC2 instances are replaced. Auto Scaling groups deliver automated instance lifecycle management with the latest AMI configurations, so failed instances are quickly replaced with identical configurations that can immediately access the existing Oracle database files on FSxN. AWS Backup creates AMIs that capture the latest Oracle host configurations including patches and settings, preserving the complete server state for consistent deployments. AWS Lambda extracts the AMI ID from backup recovery points and updates the SSM Parameter, orchestrating the entire configuration management workflow. Systems Manager Parameter Store stores the current AMI ID for Auto Scaling group launch templates, so new instances always launch with the most recent configuration and can immediately connect to the Oracle database on shared storage.

The following diagram shows the complete architecture with all AWS services and their interactions:

AWS architecture diagram showing Oracle Database disaster recovery across two Availability Zones using FSx for ONTAP synchronous replication, AWS Backup automation with EventBridge and Lambda, and Auto Scaling group with SSM Parameter Store for AMI management.

Key benefits include:

  • Recovery Time Objective (RTO): Can help achieve 2–5 minutes with latest Oracle configuration
  • Recovery Point Objective (RPO): Near-zero through synchronous Multi-AZ replication
  • Configuration consistency: New instances launch with identical Oracle host setup
  • Automated AMI management: Scheduled AMI creation with Parameter Store updates

Walkthrough

This walkthrough demonstrates implementing Oracle HA using Amazon FSx for NetApp ONTAP shared storage, AWS Backup-driven AMI creation, Lambda orchestration, and Auto Scaling groups with Parameter Store integration for configuration consistency and automated failover.

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account with appropriate permissions for Amazon FSx, Auto Scaling, EC2, Lambda, and Systems Manager
  • A VPC with subnets in at least two Availability Zones
  • Oracle database software

Keep in mind that customers are responsible for their own Oracle licensing compliance.

  • An EC2 instance with Oracle database installed and configured
  • AWS Identity and Access Management (IAM) roles for AMI creation and cross-service communication
  • Basic knowledge of Oracle database administration and AWS automation

Assumptions

This post is a conceptual illustration of the architecture. Your specific implementation will vary based on your VPC layout, Oracle version, storage requirements, and organizational security policies.

We assume the reader is familiar with:

  • Creating and configuring Amazon FSx for NetApp ONTAP file systems through the AWS console
  • iSCSI concepts including initiators, targets, and multipath I/O
  • Oracle database startup and shutdown procedures
  • AWS Backup, Lambda, and Auto Scaling group fundamentals

For detailed step-by-step instructions on specific AWS services, refer to the additional resources section.

Step 1: Create an Amazon FSx for NetApp ONTAP file system

FSxN Multi-AZ provides the persistent shared storage foundation for this architecture. Unlike Amazon Elastic Block Store (Amazon EBS) volumes, which are bound to a single AZ, FSxN Multi-AZ replicates data synchronously across two AZs with automatic failover. This means that when an EC2 instance is replaced (whether in the same AZ or a different one), the new instance can immediately access the existing Oracle database files without restoring from backup.

To create the file system, navigate to the Amazon FSx console and select Amazon FSx for NetApp ONTAP as the file system type.

The critical configuration choice is selecting Multi-AZ deployment, which places an active file server in one AZ and a standby in another.

Amazon FSx console showing oracle-fsxn-multi-az file system configuration with ONTAP Multi-AZ 1 deployment, 1024 GiB SSD storage, 512 MB/s throughput, spanning us-east-1a preferred and us-east-1b standby subnets.

FSxN console showing Multi-AZ deployment type selection with preferred and standby subnets in separate availability zones.

After the file system is created, you need to set up a Storage Virtual Machine (SVM), which acts as a logical storage container providing data access to your Oracle instances. The SVM creation is done from the FSx console under your file system’s details.With the SVM in place, the next step is configuring iSCSI access. FSxN exposes iSCSI endpoints—these are IP addresses (one per AZ) that your EC2 instances use to connect to the storage over the iSCSI protocol. You can find these endpoint addresses in the FSx console under your SVM’s Endpoints tab.

Amazon FSx Storage Virtual Machine configuration page showing oracle-svm with Created lifecycle state, NFS, iSCSI, and management endpoints for Oracle Database storage connectivity.

SVM Endpoints tab showing iSCSI endpoint IP addresses for each availability zone. These addresses are used in the EC2 instance’s iSCSI discovery configuration.

The iSCSI setup involves creating iGroups (which define which EC2 instances can access the storage) and LUNs (logical storage units mapped to those groups) through the NetApp ONTAP CLI. On the EC2 side, you configure the iSCSI initiator to discover and connect to the FSxN endpoints, then mount the resulting block devices. Using multipath I/O with both endpoints makes sure that Oracle data remains accessible even during an AZ failover. For detailed iSCSI configuration steps, see mounting iSCSI LUNs on Linux clients.

A dedicated security group is required for FSxN access. At minimum, the security group must allow inbound traffic on ports 111 (NFS portmapper), 635 (NFS mountd), 2049 (NFS), 3260 (iSCSI), 4045–4046 (NFS lock), 443 (HTTPS for management), and 22 (SSH for ONTAP CLI). Restrict the source to only your Oracle EC2 instances’ security group.

Step 2: Set up AWS Backup for EC2 instance protection

AWS Backup captures the complete state of your Oracle EC2 instance. The key design choice here is using tag-based resource selection rather than specifying instance IDs directly. Because Auto Scaling groups replace instances (and generate new instance IDs), tag-based selection makes sure that any new instance with the correct tags are automatically included in the backup plan.Configure a backup plan with a frequency appropriate for your environment and set the resource assignment to select EC2 instances matching your application tag (for example, ‘Application: Oracle’).

AWS Backup console showing blog-test backup plan with hourly backup rule targeting Oracle EC2 instances identified by the Application:oracle-db tag.

AWS Backup resource assignment configured with tag-based selection. Any EC2 instances tagged with the application tag are automatically included in the backup plan.

Step 3: Configure Lambda for AMI management

When AWS Backup completes an EC2 backup, it creates an AMI as the recovery point. An Amazon EventBridge rule detects this completion event and triggers a Lambda function. The function extracts the AMI ID from the backup recovery point, updates the SSM Parameter Store parameter with the new AMI ID, and cleans up older AMIs to control storage costs.

AWS Lambda function configuration for oracle-backup-handler showing Python 3.11 runtime, EventBridge trigger, and description indicating it processes AWS Backup completion events and updates AMI in SSM.

Lambda function overview showing the EventBridge trigger, Python 3.11 runtime, and function description indicating its role in processing backup completions and updating AMI references in SSM.

This event-driven approach means the latest AMI is available without manual intervention. The Lambda function needs IAM permissions for EC2 (to manage AMIs), SSM (to update the parameter), and Backup (to read recovery point metadata).

Amazon EventBridge rule oracle-backup-completion configured to trigger the oracle-backup-handler Lambda function when AWS Backup completes an EC2 backup job, with event pattern filtering for COMPLETED state.

EventBridge rule configured to match AWS Backup job completion events for EC2 resources, with the Lambda function as the target.

Step 4: Configure the Systems Manager Parameter Store

The SSM Parameter Store holds the current AMI ID that the Auto Scaling group’s launch template references. The parameter is created with the aws:ec2:image data type, which enables the launch template’s resolve:ssm: functionality, a feature that allows the launch template to dynamically resolve the AMI ID at instance launch time without requiring a template version update.

AWS Systems Manager Parameter Store showing /oracle/ec2/ami-id parameter with AMI value ami-0a705a7d5523c555, version 857, last modified by the oracle-backup-lambda-role on April 25, 2026.

SSM Parameter Store showing the /oracle/ec2/ami-id parameter with aws:ec2:image data type. The “Last modified user” confirms the Lambda function is automatically updating this parameter after each backup cycle.

When Lambda updates this parameter after each backup cycle, the next instance launched by the Auto Scaling group will automatically use the latest AMI. This removes the operational burden of manually updating launch template versions.

Step 5: Set up an Auto Scaling Group with dynamic AMI

The launch template references the SSM parameter using the resolve:ssm: prefix for the AMI ID field. This is the mechanism that ties the entire automation pipeline together. The mechanism backups trigger AMI creation, AMI IDs flow into Parameter Store, and the launch template resolves the latest AMI at launch time.

EC2 Launch Template oracle-db-launch-template version 75 showing AMI ID resolved from SSM parameter resolve:ssm:/oracle/ec2/ami-id with r7i.large instance type for Oracle Database deployment.

Launch template AMI configuration showing the ‘resolve:ssm:’ prefix, which dynamically retrieves the latest AMI ID from Parameter Store at instance launch time.

The Auto Scaling group is configured with minimum, maximum, and desired capacity all set to 1. This is not traditional auto-scaling, it’s a self-healing pattern. The sole purpose is to detect when the Oracle instance becomes unhealthy and automatically launch a replacement. The health check grace period should be set to at least 300 seconds (5 minutes) to allow Oracle sufficient time to start before health checks begin evaluating the new instance.

The launch template also includes a User Data script that runs on each new instance. This script configures the iSCSI initiator, discovers and connects to the FSxN endpoints, mounts the Oracle data volumes, and starts the Oracle database through a systemd service. This automation makes sure that a replacement instance is fully operational without manual intervention.

EC2 Auto Scaling group oracle-db-asg configuration showing desired capacity of 1, scaling limits 1-1, r7i.large instance type, oracle-db-launch-template with Latest version, spanning two availability zone subnets.

Auto Scaling group configured with min=max=desired=1 across two availability zones, providing self-healing capability.

Test the complete workflow

To validate the architecture, simulate an instance failure by terminating the current Oracle EC2 instance.

The expected sequence is:

  1. The Auto Scaling group detects the instance is unhealthy (within approximately 30 seconds)
  2. A new instance launches from the latest AMI resolved from Parameter Store (approximately 2 minutes)
  3. The User Data script connects to FSxN using iSCSI and starts Oracle (approximately 2–3 minutes)
  4. The Oracle database is available and accepting connections (total elapsed: approximately 5 minutes)

Auto Scaling group Activity History showing the self-healing sequence — the unhealthy instance is terminated, and a replacement is launched automatically within seconds.

The new instance automatically inherits the application tags from the Auto Scaling group, which means AWS Backup includes it in the next backup cycle without manual configuration.

Cleaning up

To avoid incurring future charges, delete the resources:

  • Delete Lambda functions and EventBridge rules
  • Remove Parameters from Systems Manager Parameter Store
  • Delete AWS Backup plans and backup vault
  • Deregister created AMIs
  • Terminate Auto Scaling group instances
  • Delete the Amazon FSx for NetApp ONTAP file system

Conclusion

This architecture facilitates Oracle high availability with configuration consistency by combining FSxN persistent shared storage with automated AMI management and AWS Backup protection. The Lambda-driven AMI management from backup recovery points and Parameter Store integration helps make sure that replacement instances launched by Auto Scaling groups always use the latest Oracle host configuration and can immediately connect to the existing Oracle database files stored on FSxN. Replacements occur only when health checks fail. Organizations can target high availability while maintaining configuration consistency across instance replacements. The automated AMI management alleviates configuration drift and makes sure that disaster recovery scenarios restore Oracle instances with identical host-level configurations that can immediately access the persistent Oracle database on shared storage. Healthy instances continue running unchanged, with replacements occurring only, when necessary, because of health check failures.Next steps include implementing cross-Region AMI replication, adding AMI validation testing, and developing custom health checks that verify both Oracle database and host configuration consistency.

Additional resources

Multi-Region event-driven failover architecture with Amazon EventBridge and Route 53

Post Syndicated from Napoleone Capasso original https://aws.amazon.com/blogs/compute/multi-region-event-driven-failover-architecture-with-amazon-eventbridge-and-route-53/

Multi-Region Event-Driven Failover Architecture with Amazon EventBridge and Route 53

Event-driven architectures enable applications to respond to events in real-time, providing scalability and loose coupling between components. However, ensuring high availability across multiple AWS regions requires careful design of failover mechanisms. This post demonstrates how to build a resilient multi-region event-driven architecture using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53 health-based failover.

Overview

Organizations building event-driven applications need to achieve high availability and disaster recovery capabilities. This architecture provides automatic failover between AWS regions while maintaining regional independence for event processing. The solution uses Amazon Route 53 health checks to monitor regional Amazon API Gateway endpoints and automatically routes traffic to healthy regions without manual intervention.

The architecture delivers several key benefits. Regional independence reduces latency by processing events in the same region where they originate. Amazon DynamoDB global tables provide automatic data replication across regions, ensuring data availability during regional failures. The solution provides robust failover capabilities while maintaining architectural simplicity.

Organizations with strict availability requirements can find this solution particularly valuable. All event processing remains within AWS regions, and failover occurs automatically based on health check results. The architecture supports both planned maintenance windows and unplanned regional outages, providing flexibility for operational needs.

Solution overview

The solution implements an active-passive multi-region architecture where events flow through Amazon API Gateway to regional Amazon EventBridge buses. Amazon Route 53 health checks monitor the primary region and automatically route traffic to the secondary region during failures. Each region processes events independently, while Amazon DynamoDB Global Tables replicate data across regions.

The following diagram provides an overview of the solution:

The above diagram depicts the multi-region architecture running across two AWS regions. The Route 53 DNS service serves as the main entry point for the application, with health checks monitoring both regions. Each region contains an identical stack with Amazon API Gateway, Amazon EventBridge, Amazon SQS, and AWS Lambda. The Amazon DynamoDB Global Table replicates data between regions automatically.

Solution deployment

To deploy this solution, follow the instructions in the GitHub repository and clone the repository. The solution deploys in two AWS regions. Ensure valid SSL certificates exist in AWS Certificate Manager (ACM) in both regions for the custom domain.

Prerequisites

For this walkthrough, the following resources are needed:

  • AWS Account: An AWS account with permissions to create and manage Amazon API Gateway, Amazon EventBridge, Amazon SQS, AWS Lambda, Amazon DynamoDB, Amazon Route 53, AWS IAM, and AWS CloudFormation resources
  • AWS Serverless Application Model (SAM): The AWS SAM CLI installed, as the templates use the SAM transform for Lambda and API Gateway resource definitions
  • Domain Name: A registered domain with a Route 53 hosted zone- SSL Certificates: ACM certificates for the custom domain in both deployment regions
  • AWS CLI: The AWS CLI installed and configured with credentials for the target AWS account
  • Region Selection: Two AWS regions for deployment

Walkthrough

The AWS CloudFormation templates from the sample GitHub repository create a secure, multi-region architecture that provides automatic failover for event-driven applications. The templates provision regional API Gateway endpoints, EventBridge buses, SQS queues, Lambda functions, and an Amazon DynamoDB Global Table. The solution establishes health monitoring through Route 53 health checks and configures DNS failover routing. The templates use AWS Serverless Application Model (SAM) transform to simplify Lambda and API Gateway resource definitions.

Step 1: Deploy the primary stack

The primary stack creates the foundational resources in the primary region. This includes the Amazon EventBridge bus, Amazon API Gateway with custom domain, health check, AWS Lambda function, Amazon SQS queue, and Amazon DynamoDB Global Table. The stack creates an EventBridge bus that receives events from API Gateway:

EventBus: 
Type: AWS::Events::EventBus 
Properties: 
Name: !Ref EventBusName

The API Gateway uses AWS service integration to forward events directly to EventBridge:

x-amazon-apigateway-integration: 
type: "aws" 
uri: !Sub "arn:aws:apigateway:${AWS::Region}:events:path//" 
credentials: !GetAtt ApiGatewayEventBridgeRole.Arn 
httpMethod: "POST"

The health check monitors the API Gateway endpoint to determine regional availability:

DomainHealthCheck: 
Type: AWS::Route53::HealthCheck 
Properties: 
HealthCheckConfig: 
Type: HTTPS 
ResourcePath: /Prod/health FullyQualified
DomainName: !Sub ${Api}.execute-api.${AWS::Region}.amazonaws.com 
Port: 443 
RequestInterval: 30 
FailureThreshold: 3

The Route 53 DNS record configures failover routing with the PRIMARY designation:

ApiDnsRecord:
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref HostedZoneId
Name: !Ref CustomDomainName
Type: A
SetIdentifier: primary-region
Failover: PRIMARY
HealthCheckId: !Ref DomainHealthCheck

The DynamoDB Global Table creates replicas in both regions:

DataTable: 
Type: AWS::DynamoDB::GlobalTable 
Properties: 
BillingMode: PAY_PER_REQUEST 
Replicas: 
- Region: !Ref AWS::Region 
- Region: !Ref SecondaryRegion

Note the `DataTableName` output value for use in the secondary stack deployment. The `CustomDomainURL` output provides the endpoint to invoke the solution.

Step 2: Deploy the secondary stack

The secondary stack creates identical resources in the secondary region , except for the Amazon DynamoDB table which references the existing Global Table. The secondary stack creates its own Amazon EventBridge bus, Amazon API Gateway, health check, AWS Lambda function, and Amazon SQS queue. The Route 53 DNS record uses the SECONDARY designation

Step 3: Event processing flow

Events flow through the processing pipeline in each region. API Gateway receives events and forwards them to EventBridge using the PutEvents API. EventBridge evaluates event rules and routes matching events to SQS queues. Lambda functions poll the SQS queues and process events in batches. AWS Lambda writes processed data to the DynamoDB Global Table, which replicates across regions.

The Lambda function processes events from the queue and writes to DynamoDB:

def handler(event, context): 
for record in event.get('Records', []): 
body = json.loads(record['body']) 
detail = body.get('detail', {}) 
event_id = body.get('id', '') 
item = { 'id': event_id, 'detail': detail, 'timestamp': datetime.utcnow().isoformat() } 
table.put_item(Item=item)

Testing

Fetch the custom domain URL and test it by sending an event:

curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v

The response includes an `X-Region` header indicating which region processed the request. Under normal conditions, this shows the primary region.

To test failover:

  1. Remove the base path mapping for the primary region:
aws apigateway delete-base-path-mapping \ --domain-name api.example.com \ --base-path '(none)' \ --region {primary-region}
  1. Delete the primary API Gateway stage:

aws apigateway delete-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --region {primary-region}

  1. Wait 2-3 minutes for the health check to fail. The Route 53 health check performs checks every 30 seconds with a failure threshold of 3, requiring 90 seconds to detect the failure.
  2. Send another request to the API endpoint:
curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v
  1. Verify the failover: The `X-Region` header now shows the secondary region, confirming successful failover.

Verify event processing in the secondary region:

  1. Check the Lambda logs for successful processing:

aws logs tail /aws/lambda/<secondary-lambda-name> --region {secondary region}

You should see log entries similar to:

Processing message: 
{"version":"0",
"id":"abc12345-...",
"source":"demo.event",
"detail-type":"POSTED",...} 
Event Source: demo.event
Detail Type: POSTED
Successfully wrote item to DynamoDB: abc12345-... 
Successfully read item from DynamoDB: 
{'id': 'abc12345-...', 
'source': 'demo.event', 
'detailType': 'POSTED', 
'detail': 
{'data': {'IsHelloWorldExample': 'true'}, 
...}, 
'timestamp': '2025-01-15T18:30:00.000000', 
'processed': True}
  1. Verify the data in Amazon DynamoDB:

aws dynamodb scan \ --table-name <table-name> \ --region {secondary region}```

The scan results should include items with the event details:

{ "Items": 
[ { "id": {"S": "abc12345-..."}, 
"source": {"S": "demo.event"}, 
"detailType": {"S": "POSTED"},
"detail": 
{"M": {"data": 
{"M": 
{"IsHelloWorldExample": 
{"S": "true"}}}}}, 
"timestamp": {"S": "2025-01-15T18:30:00.000000"},
"processed": {"BOOL": true} } ], 
"Count": 1 }
  1. Restore the primary region – recreate the stage:

aws apigateway create-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --deployment-id <deployment-id> \ --region {primary region}

  1. Restore the primary region – recreate the base path mapping:

aws apigateway create-base-path-mapping \ --domain-name api.example.com \ --rest-api-id <primary-api-id> \ --stage Prod \ --region {primary region}

You can find the “deployment-id” by running: aws apigateway get-deployments \ --rest-api-id <primary-api-id> \ --region {primary region}

After 2-3 minutes, the health check passes and Route 53 routes traffic back to the primary region.

Cleanup

To remove the solution and avoid ongoing charges, delete the CloudFormation stacks in the correct order. Delete the secondary stack first, then the primary stack. This order is important because the Amazon DynamoDB Global Table is owned by the primary stack. Warning: Deleting these stacks permanently removes all resources including the Amazon DynamoDB global table and any event data stored in it. Back up any data you need before proceeding. This action cannot be undone. The following resources incur costs while deployed:

  • Amazon API Gateway (REST API)
  • Amazon Route 53 health checks and DNS records
  • Amazon DynamoDB global table (with cross-region replication)
  • AWS Lambda function invocations and duration
  • Amazon SQS queue operations
  • Amazon CloudWatch Logs storage

Delete the secondary stack:

aws cloudformation delete-stack --stack-name secondary-stack --region {secondary region}

Wait for the secondary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name secondary-stack --region {secondary region}

Delete the primary stack:

aws cloudformation delete-stack --stack-name primary-stack --region {primary region}

Wait for the primary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name primary-stack --region {primary region}

This removes all resources including the Amazon EventBridge buses, Amazon API Gateways, AWS Lambda functions, Amazon SQS queues, Amazon DynamoDB Global Table, Amazon Route 53 health checks, DNS records and IAM roles.

Conclusion

This post demonstrates how to establish a resilient multi-region architecture for event-driven applications using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53. The solution uses Route 53 health-based failover, a powerful capability that automatically routes traffic to healthy regions based on health check results. This architecture significantly enhances application availability by providing automatic failover during regional outages while maintaining regional independence for event processing.

Automating identity lifecycle and security with AWS Directory Service APIs

Post Syndicated from Ali Alzand original https://aws.amazon.com/blogs/security/automating-identity-lifecycle-and-security-with-aws-directory-service-apis/

Managing identities and access across complex environments has become more critical than ever. AWS Directory Service for Managed Microsoft Active Directory, also known as AWS Managed Microsoft AD, has added new capabilities to manage users and groups. Now, you can perform create, read, update, and delete (CRUD) operations on users and groups directly through AWS Command Line Interface (AWS CLI), APIs, and the AWS Management Console. You can use this powerful capability to automate identity lifecycle management and enhance security in your AWS environment. By using these APIs, collectively known as the Directory Service Data APIs, you can perform operations such as:

  • Listing users and groups
  • Retrieving user and group details
  • Disabling and enabling user accounts
  • Resetting user passwords
  • Managing group memberships

These APIs provide new possibilities for automating identity management tasks and integrating Active Directory management into your existing workflows and applications.

The introduction of these APIs brings several key benefits:

  • Automation of the identity lifecycle: You can now programmatically manage user accounts throughout their lifecycle—from creation to deletion—enabling streamlined onboarding and offboarding processes.
  • Enhanced security: By integrating these APIs with security services like Amazon GuardDuty, you can create automated responses to potential security threats, such as disabling accounts with inappropriate access.
  • Improved compliance: You can use automated user management to help enforce consistent policies and help maintain compliance with various regulatory requirements.
  • Operational efficiency: You can automate routine tasks such as user provisioning, deprovisioning, and group management, reducing manual effort and the potential for human error.
  • Integration capabilities: By using these APIs, you can seamlessly integrate with existing identity management systems, custom applications, and third-party tools.
  • Cost optimization: By automating processes and reducing manual intervention, you can potentially help your organization optimize operational costs associated with identity management.

In this post, we explore these new APIs and demonstrate how you can use them to create an automated solution for detecting and responding to unexpected behavior by Active Directory users. We walk through a practical example that combines GuardDuty, AWS Step Functions, Amazon EventBridge, and the new AWS Directory Service APIs to create a robust security automation workflow.

Solution overview

To demonstrate the power of these new APIs, let’s explore a practical solution that automates the detection and response to unexpected behavior by Active Directory users. This solution combines several AWS services to create a robust security automation workflow:

    1. GuardDuty continuously monitors for unexplained behavior of Active Directory users from AWS Managed Microsoft AD. For the example in this post, we’re using Backdoor:Runtime/C&CActivity.B!DNS
    2. An EventBridge rule detects GuardDuty findings related to these users and triggers a Step Functions workflow. 
      {
  "detail-type": ["GuardDuty Finding"],
  "source": ["aws.guardduty"],
  "detail": {
    "type": ["Backdoor:Runtime/C&CActivity.B!DNS"]
  }
}

    3. The Step Functions workflow will:
      1. Extract the Active Directory username from the instance using a run command.
      2. Start an automation that will disable the account using the DisableUser API.
Figure 1: Diagram of the Step Functions workflow showing the process of Systems Manager finding the username and starting the automation to disable the account

Figure 1: Diagram of the Step Functions workflow showing the process of Systems Manager finding the username and starting the automation to disable the account

  1. Finally, another EventBridge rule will monitor the DisableUser API call. It will send an email to the user using Amazon Simple Notification Service (Amazon SNS) notifications. 
    {
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.ds"],
  "detail": {
    "eventSource": ["ds.amazonaws.com"],
    "eventName": ["DisableUser"]
  }
}

This solution delivers automated, near real-time remediation of potential security threats — significantly reducing exposure windows and containing the impact of unauthorized account access.

The following figure shows a high-level architecture diagram of the solution.

Figure 2: Diagram showing the workflow of what happens when potentially damaging activity is detected

Figure 2: Diagram showing the workflow of what happens when potentially damaging activity is detected

Note: The solution must be deployed in the primary AWS Region of your directory.

Prerequisites

To complete the walkthrough in this post, you must have the following prerequisites in place.

GuardDuty

GuardDuty is an automated threat detection service that continuously monitors for unexpected activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon Simple Storage Service (Amazon S3).

To activate GuardDuty:

  1. Go to the GuardDuty console.
    1. If you’re activating GuardDuty for the first time, under Try threat detection with GuardDuty, select All Features and then choose Get Started.
    2. If you’ve used GuardDuty before, select Runtime Monitoring and then choose Enable under Runtime Monitoring.
Figure 3: Runtime Monitoring enabled

Figure 3: Runtime Monitoring enabled

AWS Managed Microsoft AD

AWS Managed Microsoft AD provides a fully managed service for Microsoft Active Directory (AD) in the AWS Cloud. When you create your directory, AWS deploys two domain controllers that are exclusively yours in separate Availability Zones for high availability. For use cases that require even higher resilience and performance in a specific AWS Region or during specific hours, you can scale AWS Managed Microsoft AD by deploying additional domain controllers to meet your needs. These domain controllers can help load balance, increase overall performance, or provide additional nodes to protect against temporary availability issues. Using AWS Managed Microsoft AD, you can define the correct number of domain controllers for your directory based on your use case.

To deploy a new AWS Managed Microsoft AD:

  1. Go to the Directory Service console.
  2. Choose Set up directory and select AWS Managed Microsoft AD.
  3. Select Standard Edition and enter a directory DNS name and password.
  4. Select a virtual private cloud (VPC). For this example, use the Default VPC.
  5. Choose Create directory.

Create a test Active Directory user

You will use this test user account to sign in to an EC2 instance and initiate a command that simulates unexplained activity that results in this account being disabled.

To create the test user, you can use AWS CloudShell or the AWS CLI from your local machine. Run the following commands, replacing the --directory-id value with your own:

# Create the test user
aws ds-data create-user \
 --directory-id "your-directory-id" \
 --sam-account-name "TestUser" \
 --given-name "Test" \
 --surname "User"

Then

# Set a password for the test user 
aws ds reset-user-password \
 --directory-id "your-directory-id" \
 --user-name "TestUser" \
 --new-password "YourSecurePassword123!"

In this example, the password is set to YourSecurePassword123!. If you need to replace it with a password that meets your organization’s requirements, see Resetting and enabling an AWS Managed Microsoft AD user’s password. For more information on creating users, see Creating an AWS Managed Microsoft AD user in the AWS Directory Service documentation.

Test EC2 instance

To generate alerts on GuardDuty, you need a domain joined Linux EC2 instance. If you don’t have a domain joined EC2 Linux instance, follow these instructions for joining a Linux instance to an Active Directory domain. This instance will be used to simulate suspicious activity that triggers a GuardDuty finding and initiates the automated remediation workflow.

Implement the solution

Let’s walk through the steps to implement this solution in your AWS environment.

Deploy the solution

  1. Download the CloudFormation template
  2. Navigate to the CloudFormation console in the AWS account.
  3. For Create Stack, choose with new resources (standard).
  4. For Template source, choose Upload a template file. Choose Choose file and select the template you downloaded in step 1.
  5. Choose Next.
  6. For Stack name, enter a stack name (such as CRUD-API-MAD).
  7. In the Parameters area, do the following:
    1. For DirectoryID, enter the AWS Active Directory ID.
    2. For NotificationEmail, enter the email address to send the notification to.
  8. On the Configure stack options page, choose Next.
  9. Select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.

After the page is refreshed, the status of your stack should be CREATE_IN_PROGRESS. When the status changes to CREATE_COMPLETE, proceed to the next section.

Test

To simulate a threat, use a GuardDuty test domain that GuardDuty will recognize as a command and control server.

  1. Go to the Amazon EC2 console.
  2. Choose Instances from the navigation pane.
  3. Select the test EC2 instance that you created earlier.
  4. Choose Connect, select the Session Manager tab, and choose Connect.
  5. Authenticate with your test user by entering su followed by the test user with the domain name that you created earlier. For example su [email protected], then enter the password.
  6. Enter the command curl guarddutyc2activityb.com.
    You will receive an error because the page won’t resolve, but GuardDuty will have detected concerning events.
  7. Go to the GuardDuty console and select Findings from the navigation pane.
  8. Within 3–5 minutes, you should see a high severity finding for Backdoor:Runtime/C&CActivity.B!DNS.
  9. This will then trigger the automation to disable the account.
    Figure 4: Account successfully disabled

    Figure 4: Account successfully disabled

  10. After the account is disabled, an email notification will be sent notifying an administrator that the account was disabled (it might take up to 5 minutes to receive the notification).

    Figure 5: AWS notification message showing the username has been disabled

    Figure 5: AWS notification message showing the username has been disabled

Note: You must archive the GuardDuty finding before running this test again, because the EventBridge rule only runs once against a GuardDuty finding with the same details. To archive the finding, select the check box next to the Backdoor:Runtime/C&CActivity.B!DNS finding, choose Actions (top right), and select Archive.

Conclusion

The new AWS Directory Service APIs for AWS Managed Microsoft AD provide powerful capabilities for programmatically managing Active Directory users and groups. By using these APIs in conjunction with services such as Amazon GuardDuty and AWS Step Functions, you can create sophisticated automation workflows that enhance your security posture and streamline identity management processes.

The solution we’ve explored in this post demonstrates just one of many possible use cases for these new APIs. As you integrate these capabilities into your own environments, you will probably discover numerous opportunities to improve efficiency, security, and compliance in your identity management practices.

For a solution that uses PowerShell Active Directory cmdlets with AWS Systems Manager Run Command to disable users, see How to automatically disable users in AWS Managed Microsoft AD based on GuardDuty findings.

For more information about AWS Directory Service and its APIs, visit the AWS Directory Service documentation.

We’re excited to see how you’ll use these new APIs to innovate and improve your identity management workflows. If you have any questions or want to share your own use cases, leave a comment below or reach out to AWS Support.

Remember, the cloud journey is all about continuous improvement and innovation. Keep exploring, keep learning, and keep pushing the boundaries of what’s possible with AWS.

Ali Alzand

Ali Alzand

Ali is a Senior Infrastructure Migration & Modernization Specialist Solutions Architect at AWS who helps enterprise customers migrate, modernize, and operate their Microsoft workloads on AWS. He specializes in Infrastructure as Code, automating at scale with AWS Systems Manager, EC2 Image Builder, and CloudFormation. He also designs event-driven architectures building responsive, loosely coupled solutions with EventBridge and Lambda. Outside of work, Ali enjoys grilling with friends and discovering new cuisines around town.

Kevin Sookhan

Kevin Sookhan

Kevin is a Specialist Solutions Architect at Amazon Web Services with over 20 years of experience working with Microsoft technologies. He has expertise in running Microsoft workloads on AWS with specialization in helping customers with their migrations, cost optimization, and infrastructure architecture.

Serverless ICYMI Q1 2026

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/serverless-icymi-q1-2026/

Stay current with the latest serverless innovations that can improve your applications. In this 32nd quarterly recap, discover the most impactful AWS serverless launches, features, and resources from Q1 2026 that you might have missed.

In case you missed our last ICYMI, check out what happened in Q4 2025.

2026 Q1 calendar

2026 Q1 calendar

Serverless with Mama J




Serverless with Mama J

If you really want to know whether you understand something, try explaining it to your mom!

That’s exactly what Eric Johnson did. His mom, everyone calls her Mama J, wanted to know what serverless actually means and why it matters. So he walked her through it: what servers do, why they’re a headache to manage, and how AWS Lambda lets you skip all that by running code only when it’s needed, scaling automatically, and charging you nothing when nobody’s using it.

Watch the video on the AWS Developers YouTube channel.

Build serverless apps faster with AI

AWS is providing a growing set of AI-powered tools to bring serverless expertise directly into your coding assistants. From Model Context Protocol (MCP) servers and Anthropic Claude plugins to Kiro Powers. These tools provide contextual guidance for architecture decisions, implementation patterns, and deployment automation across the full serverless development lifecycle.

For more information on the tools available, see the resources page.

Serverless Patterns Collection

The open source Serverless Patterns Collection on Serverless Land now provides a direct link to download pattern .zip files. You can also clone the whole repo and explore more patterns.

Serverless Patterns .zip download

Serverless Patterns .zip download

AWS Lambda

Build fault-tolerant, long-running applications using familiar programming patterns using AWS Lambda durable functions. You can use Lambda durable functions to write multi-step workflows in your preferred programming language, using built-in methods that automatically handle progress checkpointing and error recovery. This can improve your architecture so that you can focus on your business logic and optimize costs by charging only for active compute time.

You can build durable functions in Python and TypeScript and there is a durable execution SDK for Java in preview with the code available on GitHub.

Eric Johnson has a new video deep dive showing how to upload videos and scan them with AI. Learn how to coordinate multiple AWS services like Amazon Rekognition and Amazon Transcribe, implement human-in-the-loop approval workflows, and crate a live dashboard for real-time updates.

To find out how durable functions work, see the blog post which also provides testing and best practices guidance. You can also watch the re:Invent Breakout Session video: Deep Dive on AWS Lambda durable functions (CNS380)

Lambda now supports the .NET 10 runtime, including support for file-based apps. Developers can take advantage of the latest .NET 10 performance improvements, new language features, and improved startup times for Lambda functions.

You can now see Availability Zone (AZ) metadata in function execution environments. This allows you to determine the AZ ID (e.g., use1-az1) of the AZ your function is running in. This helps build functions that can make AZ-aware routing decisions, such as preferring same-AZ endpoints for downstream services to reduce cross-AZ latency. Operators can also implement AZ-aware resilience patterns like AZ-specific fault injection testing.

Payload size increase

AWS has increased the maximum payload size from 256 KB to 1 MB for a number of services such as asynchronous Lambda invocations, Amazon SQS, and Amazon EventBridge. This gives you more room to build and maintain context-rich event-driven systems and reduce the need for complex workarounds such as data chunking or external large object storage.

This blog post explores a real-world example using rich event context in agentic event-driven architectures

Payload size increase workflow

Payload size increase workflow

Amazon Bedrock

Amazon Bedrock expanded its model availability with a new set of fully managed open-weight models spanning frontier reasoning and agentic coding. Other model releases include Anthropic Claude Opus 4.6 and Claude Sonnet 4.6, and NVIDIA Nemotron 3 Super. You can invoke them through the unified Amazon Bedrock API without managing any underlying infrastructure, making it straightforward to experiment and swap models as your workload evolves.

Amazon Bedrock AgentCore is the infrastructure layer for securely deploying and operating AI agents. It works with popular open source frameworks, including Strands Agents, LangGraph and CrewAI, giving you the flexibility to build with your preferred tools without vendor lock-in.

AgentCore Gateway now includes semantic tool search, so you can discover the right tool for a task using natural language queries instead of manually browsing a catalogue. It also adds custom KMS encryption, debugging messages, and resource tagging to give you stronger governance over tool integrations.

Policy in Bedrock AgentCore allows you to define precise boundaries on agent actions and run continuous quality monitoring. This helps you maintain predictable, auditable agent behavior in production without embedding guardrail logic inside each individual agent.

AgentCore Runtime now supports stateful MCP server features, allowing agents to maintain session context across tool calls for richer, more coherent multi-step interactions.

Strands Agents

Strands Agents SDK

Strands Agents SDK

Strands Agents is an open source SDK for building and running AI agents in just a few lines of code, working with models available in Amazon Bedrock. Strands Labs is a new dedicated GitHub organization for experimental agent projects, including robotics and code agents. This gives you early access to cutting-edge agentic techniques before they reach production frameworks. See the introduction blog post for more information.

AWS Step Functions

AWS Step Functions introduces an enhanced TestState API that enables API-based testing for validating workflows before deployment. The new API supports testing individual states in isolation or complete workflows end-to-end, making it easier to verify state machine logic without incurring runtime costs.

By integrating TestState API testing into CI/CD pipelines, you can validate workflow logic before deployment, reducing the risk of production issues. Find complete code examples and testing framework in the GitHub repository.

Amazon EventBridge

Amazon EventBridge Scheduler now provides resource count metrics to help you monitor quota usage. These new metrics make it easier to track the number of schedules and schedule groups in your account and proactively manage service quotas.

Amazon DynamoDB

You can replicate Amazon DynamoDB table data across multiple AWS accounts and Regions. This enhances resiliency through account-level isolation, supports tailored security and data-perimeter controls. You can align workloads by business unit or environment and simplify governance requirements.

Amazon DynamoDB global replication

Amazon DynamoDB global replication

Amazon ECS

Amazon ECS Managed Instances can now integrate with Amazon EC2 Capacity Reservations. This allows you to make sure there is capacity availability for your container workloads while benefiting from the management automation of ECS Managed Instances.

ECS also now supports Network Load Balancer (NLB) for linear and canary deployment strategies. This helps you perform gradual traffic shifting using NLBs, providing more flexibility in deployment pipelines for latency-sensitive applications.

Serverless blog posts

January

February

March

Serverless Office Hours

Join our livestream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. Watch episodes on-demand at serverlessland.com/office-hours.

January

February

March

Still looking for more?

The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Developer Advocacy team to see the latest news, follow conversations, and interact with the team.

And finally, visit Serverless Land  for your serverless needs.

AWS Outposts monitoring and reporting: A comprehensive Amazon EventBridge solution

Post Syndicated from Matt Price original https://aws.amazon.com/blogs/compute/aws-outposts-monitoring-and-reporting-a-comprehensive-amazon-eventbridge-solution/

Organizations using AWS Outposts racks commonly manage capacity from a single AWS account and share resources through AWS Resource Access Manager (AWS RAM) with other AWS accounts (consumer accounts) within AWS Organizations. In this post, we demonstrate one approach to create a multi-account serverless solution to surface costs in shared AWS Outposts environments using Amazon EventBridge, AWS Lambda, and Amazon DynamoDB. This solution reports on instance runtime and allocated storage for Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Services (Amazon RDS), and Amazon Elastic Block Store (Amazon EBS) services running on Outposts racks. In turn, teams can track the cost of infrastructure associated with their workloads across AWS accounts. This solution is a framework that can be customized to meet your organization’s specific business objectives.

Solution overview

The following is the Terraform-based reference architecture used to represent the solution, including EventBridge, DynamoDB, and Lambda across a multi-account environment. Relevant launch events are tracked in EventBridge that invoke Lambda functions, which are logged in DynamoDB tables (see sample code). This allows reporting on captured event data through the AWS SDK for Python (Boto3). AWS architecture diagram showing data collection and workload account integration with EventBridge, CloudTrail, and Outposts
Figure 1: Reference architecture for reporting solution on AWS Outposts 

Prerequisites

The following prerequisites are necessary to implement this solution:

Walkthrough

The following sections walk you through how to deploy this solution.

Deploying in data collection account

Step 1: Create a bucket in-Region to hold the Terraform state file in the data collection account.

aws s3 mb s3://state-bucket-name

Step 2: Clone the repository.On your local machine, clone the repository that contains the sample by running the following command:

git clone https://github.com/aws-samples/sample-outposts-monitoring-and-reports.git

Navigate to the cloned repository by running the following command:cd sample-outposts-monitoring-and-reports/data_collection

Step 3: Edit the providers.tf to configure the AWS provider.



provider "aws" {
  region = ""
}

Step 4: Edit the backend.tf to provide the Terraform state bucket and Outposts anchored AWS Region.

terraform {
  backend "s3" {
    bucket = ""
    key    = "terraform.tfstate"
    region = ""
  }
}

Step 5: Modify the variables.tf.From the root directory of the cloned repository, modify the variables.tf file with the target Region and workload accounts as shown in the following example. The target Region is the collection destination.

variable "aws_region" {
  description = "AWS region for resources"
  type        = string
  default     = ""
}

variable "allowed_account_id" {
  description = "AWS account ID allowed to put events to the event bus"
  

}

Initialize the configuration directory of the data collection account to download and install the providers defined in the configuration by running the following command:

terraform init

All resources are deployed with minimal permissions to serve as an example. We recommend viewing all configurations to make sure that they meet your organizational security policies. Step 6: Deploy infrastructure in the data collection account.Run terraform plan on the configuration to and review which resources are created:

terraform plan

When you have reviewed the plan, run the following command and enter “yes” to accept the changes and deploy:

terraform apply

Deployment should take less than 5 minutes. If you receive any errors, review the previously mentioned steps to ensure that you followed them in their entirety. If the errors persist, reach out to AWS Support for additional guidance.

Deploying in workload account

The data collection account receives events from EventBridge and performs intelligent analysis and storage from the AWS Outposts resource data.Step 1: Navigate to the workload account directory by running the following command:

cd ../workload_account

Step 2: Edit variables.tf to set up the Region and event bus Amazon Resource Name (ARN). 

variable "aws_region" {
  description = "AWS region for resources"
  type        = string
  default     = ""
}

variable "event_bus_arn" {
  description = "target event bus arn"
  type        = string
  default     = ""
}

Edit the code to update the event bus name.

Step 3: Run the following command to create the backend.tf and create the Terraform state bucket for each workload account.

./init-backend.sh

This is an idempotent operation that creates a file from the template and a bucket with a fixed name including the account ID if it doesn’t exist. 

Step 4: Initialize the configuration directory of the Data Collection Account to download and install the providers defined in the configuration by running the following command:

terraform init

Step 5: Deploy the infrastructure in the Data Collection Account.Run a terraform plan on the configuration and review which resources are created:

terraform plan

After you have reviewed the plan, run the following command and enter “yes” to accept the changes and deploy:

terraform apply

Deployment should take less than 5 minutes. If you receive any errors, follow the troubleshooting steps in the previous section.

At this point, any Amazon EC2 or Amazon RDS instances and Amazon EBS volumes are logged to the DynamoDB tables in the data collection account. Repeat Steps 3–5 for each workload account running resources on AWS Outposts with appropriate account credentials. If you’re deploying at scale and using AWS Control Tower consider using AWS Control Tower Account Factory for Terraform (AFT).

Running monthly reports

With this solution in place, reports can be generated on demand. These reports can be customized by modifying the Python example scripts shown to support your needs. Reports can be created from a local machine with credentials that have access to the DynamoDB tables in the data collection account. The examples were created from the source directory of the data collection account git repository. Run the following command to view the report for Amazon RDS usage in September 2025:

./rds_runtime_calculator.py --year 2025 --month 9 --output rds_report.csv

Spreadsheet showing RDS database instances with configuration details, storage allocation, and operational status in us-west-2 region

Figure 2: Example of RDS runtime report 

 

Run the following command to view the report for Amazon EBS usage in September 2025:

./ebs_volume_reporter.py --year 2025 --month 9 --output ebs_report.csv

 

EBS volume tracking table showing volume configurations, lifecycle hours, and active/deleted status in us-west-2

Figure 3: Example of EBS usage report 

 

Run the following command to view the report for Amazon EC2 usage in September 2025:

./ec2_runtime_calculator.py --month 9 --year 2025 --output ec2_report.csv

EC2 instance tracking table showing c5.large instances with runtime hours and running/stopped status on AWS Outposts

Figure 4: Example of EC2 runtime report 

 

Cleaning up

Complete the following steps to clean up the resources that were deployed by this solution. For each workload account, complete the following:

cd sample-outposts-monitoring-and-reports/workload_account
terraform destroy

Enter “yes” to proceed. You can then manually empty and remove the terraform state S3 bucket for that account.

For the data collection, complete the following:

cd ../data_collection
terraform destroy

Enter “yes” to proceed. You can then manually empty and remove the terraform state S3 bucket for that account.

Conclusion

Customers who have shared multi-account Outposts deployments can use this solution to create account level reporting for Outposts resources using real-time event capture and processing, state analysis and categorization, historical usage metrics, and serverless architecture. Teams can use this to visualize and report on the costs of running their workloads on Outposts. The event-driven design supports accurate tracking while maintaining low operational overhead. The solution scales effectively across multiple Outposts and accounts, providing a unified view of hybrid infrastructure. Keep in mind that you can extend the functionality described here to meet your business objectives.

Deploy this solution today using the GitHub repository to gain financial insights to share with the tenants of your Outposts workload accounts. Reach out to your AWS account team, or fill out this form to learn more about Outposts.

Build a multi-tenant configuration system with tagged storage patterns

Post Syndicated from Koshal Agrawal original https://aws.amazon.com/blogs/architecture/build-a-multi-tenant-configuration-system-with-tagged-storage-patterns/

In modern microservices architectures, configuration management remains one of the most challenging operational concerns. Two gaps emerge as organizations scale: handling tenant metadata that changes faster than cache TTL allows, and scaling the metadata service itself without creating a performance bottleneck.

Traditional caching strategies force an uncomfortable trade-off: either accept stale tenant context (risking incorrect data isolation or feature flags), or implement aggressive cache invalidation that sacrifices performance and increases load on your metadata service. When tenant counts grow into the hundreds or thousands, this metadata service itself becomes a scaling challenge, particularly when different configuration types have vastly different access patterns.

The challenge intensifies when you need to support different storage backends for different configuration types. Some require high-frequency access patterns suited for Amazon DynamoDB, while others benefit from the hierarchical organization and built-in versioning of AWS Systems Manager Parameter Store. Traditional solutions often force engineering teams into a corner: either build multiple configuration services (increasing operational overhead), or compromise on performance by using a single storage backend that isn’t optimized for every use case.

In this post, we demonstrate how you can build a scalable, multi-tenant configuration service using the tagged storage pattern, an architectural approach that uses key prefixes (like tenant_config_ or param_config_) to automatically route configuration requests to the most appropriate AWS storage service. This pattern maintains strict tenant isolation and supports real-time, zero-downtime configuration updates through event-driven architecture, alleviating the cache staleness problem.

What you’ll learn:

  • Implementing a multi-tenant data model with DynamoDB and Parameter Store
  • Using the Strategy pattern for flexible storage backend switching
  • Building tenant isolation through JSON Web Token (JWT) claims
  • Creating an event-driven auto-refresh mechanism with Amazon EventBridge and AWS Lambda
  • Implementing zero-downtime configuration updates with gRPC (a high-performance communication protocol) streaming
  • Addressing the cache TTL problem for rapidly-changing tenant metadata

By the end of this post, you’ll understand how to architect a configuration service that handles complex multi-tenant requirements while optimizing for both performance and operational simplicity.

Solution overview

The architecture uses four AWS services orchestrated through a NestJS-based gRPC service to create a reliable, event-driven configuration management system. Let’s first understand the overall architecture before diving into each component’s implementation details.

Architecture components

The following diagram shows the end-to-end architecture of the Multi-Tenant Configuration Service deployed on AWS, from how client requests enter the system to how configuration data is retrieved from the right storage backend.

WS microservices architecture diagram showing ECS Fargate services, API Gateway, Cognito auth, DynamoDB, and CloudWatch monitoring

Figure 1: Multi-Tenant Configuration Service Architecture

Client applications authenticate via Amazon Cognito and pass through AWS WAF before reaching Amazon API Gateway. Traffic is then routed through a VPC Link to an Application Load Balancer, which distributes requests across two core microservices running on Amazon Elastic Container Service (Amazon ECS) on AWS Fargate within private subnets :

  • Order Service— handles incoming REST requests and delegates configuration lookups to the Config Service via gRPC
  • Config Service— exposes a gRPC API and uses a Config Strategy Factory to dynamically select the appropriate storage backend (DynamoDB or Parameter Store) based on the request

Service discovery is managed by AWS Cloud Map, while Amazon CloudWatch centralizes logs and metrics across services.

The system is organized into four interconnected layers, each addressing a specific aspect of the configuration management challenge:

1. Storage layer – multi-backend strategy

The storage layer strategically uses two complementary AWS services, each optimized for different configuration access patterns and requirements.

  • Amazon DynamoDB: Stores tenant-specific configurations. These are settings unique to each customer, such as payment gateway preferences or feature flags. With single-digit millisecond latency, DynamoDB handles high-frequency reads efficiently. The schema uses composite keys (TENANT#{tenantId} as partition key, CONFIG#{configType} as sort key) for efficient tenant-scoped queries and built-in multi-tenant isolation at the data model level.
  • AWS Systems Manager Parameter Store: manages shared parameters. These are configuration values used across multiple services or tenants, such as API endpoints, database connection strings, and region-specific settings. Unlike tenant-specific configs that change frequently, these parameters are relatively static but benefit from hierarchical organization. The path structure (/config-service/{tenantId}/{service}/{parameter}) enables bulk retrieval operations, reducing the number of API calls needed during service initialization from dozens to a single request.

2. Service layer – gRPC with strategy pattern

A NestJS-based microservice implements the configuration retrieval logic using gRPC for high-performance, type-safe communication. This choice significantly reduces network bandwidth and improves response times for service-to-service communication where compatibility with web browsers isn’t a requirement.

At the core is a Strategy Pattern implementation that determines the optimal storage backend based on configuration key prefixes. This pattern simplifies the addition of new storage backends (like Amazon Simple Storage Service (Amazon S3) for large configuration files) without modifying the core service logic.

3. Authentication layer – Amazon Cognito

User authentication flows through Amazon Cognito with custom attributes:

  • custom:tenantId (immutable) – Tenant identifier embedded in JWT
  • custom:role (mutable) – User role for authorization

Critical security design: The service never accepts tenantId from request parameters. Instead, it extracts the tenant context from validated JWT tokens, making sure requests cannot access other tenants’ data even if they attempt to manipulate request payloads.

4. Event-driven refresh layer

Traditional configuration updates present a dilemma: how do you keep services synchronized without compromising performance or causing downtime?

Polling approaches continuously check for changes, generating unnecessary API calls that cost money even when nothing changes. They also introduce delays. Services don’t see updates until the next poll cycle, which could be seconds or minutes later.

Service restart approaches cause downtime, drop active connections, and disrupt user sessions. For SaaS applications serving customers 24/7, restart-based updates are unacceptable.

The event-driven refresh layer addresses both problems by implementing a reactive architecture where Amazon EventBridge monitors Parameter Store for changes and triggers AWS Lambda to update the service’s local cache. This achieves configuration updates within seconds while users experience no interruption.

Technical implementation

The following sections detail the implementation, starting with the data model, which serves as the backbone for tenant isolation and efficient querying.

A. Multi-tenant data model

The foundation of tenant isolation begins with the data model. Using DynamoDB’s composite key structure, we achieve both tenant isolation and efficient querying without requiring separate tables per tenant.

DynamoDB schema design:

The following example shows a tenant-specific configuration stored in DynamoDB, illustrating how composite keys enable both isolation and efficient access:

{
  "pk": "TENANT#acme-corp",
  "sk": "CONFIG#payment-gateway",
  "config": {
    "providers": [
      {
        "name": "Stripe",
        "apiEndpoint": "https://api.stripe.com",
        "retryPolicy": "exponential"
      }
    ]
  },
  "isActive": true,
  "version": 2,
  "createdAt": "2024-01-15T10:30:00Z",
  "updatedAt": "2024-02-20T14:45:00Z"
}

Key schema decisions:

  1. Partition key pattern: TENANT#{tenantId} makes sure tenant data is co-located, enabling efficient tenant-scoped queries while maintaining logical separation.
  2. Sort key pattern: CONFIG#{configType} allows querying specific configuration types within a tenant’s data. The CONFIG# prefix enables future expansion with other entity types (for example, METADATA#, AUDIT#).
  3. Soft deletion: The isActive boolean flag supports soft deletion, maintaining audit trails while excluding inactive configurations from queries.
  4. Versioning: The version field tracks configuration changes, supporting rollback capabilities and change history.

Parameter store organization:

Parameters follow a hierarchical structure that mirrors the multi-tenant model. This example demonstrates the path structure:

/config-service/
├── acme-corp/
│   ├── api/
│   │   ├── api-key
│   │   └── endpoint
│   └── database/
│       └── connection-string
└── globex-inc/
    ├── api/
    │   ├── api-key
    │   └── endpoint
    └── database/
        └── connection-string

This structure provides several benefits:

  • Bulk retrieval using path prefix (GetParametersByPath API)
  • Clear ownership and access control through AWS Identity and Access Management (AWS IAM) policies
  • Environment separation (dev/staging/prod) at the path level
  • Automatic parameter versioning and change tracking

Advanced: Multi-dimensional tenant context
For organizations with multiple services requiring different configuration scopes, consider introducing a second dimension in the partition key:

PK = "TENANT#acme-corp|SERVICE#order-service"
SK = "CONFIG#payment-gateway"

This multi-dimensional approach enables service-level isolation where the Order service sees only billing API configurations while the Reporting service doesn’t have access to payment gateway settings. It also provides efficient service-scoped queries, retrieve configurations for a specific service with PK = TENANT#acme-corp|SERVICE#order-service and SK begins with CONFIG#. The second dimension can represent business units, geographic regions, or a logical boundary that aligns with access control requirements, making this pattern particularly valuable when fine-grained access control beyond tenant-level isolation is needed. For detailed guidance on multi-tenant DynamoDB modelling patterns, see amazon-dynamodb-data-modeling-for-multi-tenancy-part-2.

B. Strategy pattern for storage flexibility

The system decides which storage backend to use for each configuration request. The Strategy Pattern is a design approach that allows a program to choose different behaviors at runtime based on context. Think of it like a traffic controller that examines each request and directs it to the appropriate service.

Why use the strategy pattern?

Without the Strategy Pattern, handling multiple storage backends would require complex conditional logic throughout the code base. Different tenant metadata has vastly different access patterns. Routing to optimized backends alleviates both DynamoDB cost explosions (for rarely-changing configs) and Parameter Store throttling (for high-frequency reads), addressing the scaling gap. A naive implementation might look something like this and it’s worth pausing to understand why this approach breaks down.

// Without Strategy Pattern - complex and hard to maintain
async getConfig(key: string, tenantId: string) {
  if (key.startsWith('tenant_config_')) {
    // DynamoDB logic here
    const pk = `TENANT#${tenantId}`;
    const sk = `CONFIG#${key.slice(14)}`;
    return await this.dynamoDB.query({...});
  } else if (key.startsWith('param_config_')) {
    // Parameter Store logic here
    const path = `/config-service/${tenantId}/${key.slice(13)}`;
    return await this.ssm.getParameter({...});
  }
  // More conditions as backends are added...
}

Every time you add a new storage backend, say, AWS Secrets Manager or Amazon S3, you’re forced to reach back into this function and bolt on another else if. The storage logic becomes tightly coupled to your service layer, making it harder to test each backend in isolation and nearly impossible to swap one out without risking regressions elsewhere.

Implementation strategy

The Strategy Pattern encapsulates storage-specific logic into separate, interchangeable strategy classes. This code demonstrates how the factory examines keys and selects strategies:

@Injectable()
export class ConfigStrategyFactory {
  private keyStrategyMap = new Map<string, ConfigStrategy>([
    ['tenant_config_', this.dynamoDBConfigStrategy],
    ['param_config_', this.ssmConfigStrategy],
  ]);
  getStrategy(key: string): ConfigStrategy {
    for (const [prefix, strategy] of this.keyStrategyMap.entries()) {
      if (key.startsWith(prefix)) {
        return strategy;
      }
    }
    throw new ValidationException(`Invalid key format: ${key}`);
  }
}

Key prefix mapping:

  • tenant_config_* → Routes to Amazon DynamoDB for tenant-specific, high-frequency access patterns
  • param_config_* → Routes to AWS Systems Manager Parameter Store for shared, hierarchical parameters

With this approach, adding a new storage backend requires only:

  • Creating a new strategy class implementing the ConfigStrategy interface
  • Adding one line to the keyStrategyMap with the new prefix and strategy
  • No changes to existing strategies or calling code

This design helps protect technology investments. As requirements evolve and new AWS services become relevant, the system adapts without major rewrites.

Multi-layer caching strategy

Different configurations benefit from different caching approaches. The pattern implements different caching strategies optimized for each configuration type’s access patterns and business requirements:

  • High-frequency tenant configurations (accessed thousands of times per minute) use application-level caching with short Time-To-Live (TTL) values. This significantly reduces database queries while maintaining reasonably fresh data.
  • Shared parameters (accessed frequently but change rarely) use in-memory caching with event-driven invalidation. The cache only refreshes when EventBridge detects an actual change, alleviating unnecessary API calls.

Cache Security Considerations

The implementation uses a shared in-memory Map with tenant-prefixed keys (tenantId:serviceName:configKey). Cached values are configuration metadata (API endpoints, feature flags, thresholds), not sensitive data like credentials or PII. Sensitive values remain in Parameter Store with SecureString encryption and are retrieved on-demand, not cached. Even in edge cases, downstream access controls (JWT validation, DynamoDB composite keys) act as the final enforcement boundary.

For teams handling more sensitive configuration payloads, consider Amazon ElastiCache (Redis OSS) or Valkey with key-prefix isolation and encryption at rest/in transit, though this adds 1-3ms network latency versus sub-millisecond in-memory access.

C. Authentication and tenant isolation

Tenant isolation is enforced at multiple layers, starting with JWT-based authentication and custom authorization guards.

Cognito JWT validation flow:

  1. Client authenticates with Cognito and receives JWT token
  2. Request includes JWT in Authorization: Bearer {token} header
  3. CognitoJwtGuard validates token signature against Cognito JSON Web Key Sets (JWKS) endpoint
  4. Guard extracts custom:tenantId claim and attaches to request context
  5. TenantAccessGuard verifies user has access to requested tenant
  6. Service layer uses validated tenantId for data operations

This implementation demonstrates the secure approach to tenant context extraction:

async retrieveConfig(req: RetrieveConfigRequest): Promise<RetrieveConfigResponse> {
  // tenantId is extracted from validated JWT token, never from request parameters
  const tenantId = (req as any).tenantId;
  if (!tenantId) {
    throw new UnauthorizedException('Tenant ID not found in authentication context');
  }
  const strategy = this.strategyFactory.getStrategy(req.key);
  const data = await strategy.getConfig(req.serviceName, req.key, tenantId);
  return { data };
}

Why this approach helps prevent unauthorized access:

Consider what happens if an unauthorized user tries to access another tenant’s configuration:

  1. User authenticates as Tenant A and receives JWT with custom:tenantId: "tenant-a"
  2. User attempts to manipulate request to access Tenant B’s data
  3. The service extracts tenantId from the JWT (still “tenant-a”), ignoring request parameters
  4. Query uses the JWT’s tenant ID, so user only sees Tenant A’s data

Advanced: Infrastructure-level credential isolation

The current design enforces tenant isolation at the application layer through JWT extraction and DynamoDB composite keys. The ECS task uses a shared IAM execution role, meaning tenant requests operate under the same AWS credentials. While this approach is sufficient for most multi-tenant applications, teams with stricter compliance requirements (HIPAA, PCI-DSS, FedRAMP) may need infrastructure-level isolation.

For enhanced isolation, consider implementing a Token Vending Machine (TVM) pattern with AWS Security Token Service (STS) to issue temporary, tenant-scoped IAM credentials. This provides infrastructure-level isolation with per-tenant AWS CloudTrail audit trails and principle of least privilege enforcement. However, TVM adds operational complexity (credential caching, STS API costs, token refresh logic) and latency (50-100ms per operation).

Consider this as a next step when compliance auditors require infrastructure-level separation rather than a baseline requirement.

This design helps prevent cross-tenant access attempts at the infrastructure level, addressing a common security issue.

D. Zero-downtime auto-refresh mechanism

Configuration updates in production systems present a classic operations challenge. This event-driven approach addresses the cache TTL trade-off entirely, configurations update in real-time without polling or staleness windows.

EventBridge integration flow:

1. Parameter Store Change
         ↓
2. EventBridge Rule (matches /config-service/* changes)
         ↓
3. Lambda Function (extracts tenantId from path)
         ↓
4. Service Discovery (AWS Cloud Map queries for healthy instances)
         ↓
5. gRPC Refresh Call (direct service-to-service invocation)
         ↓
6. In-Memory Cache Update (zero-downtime)
         ↓
7. Updated Configuration Active (no connection drops)

Key benefits:

  1. Zero downtime: No service restarts required. Connections remain active
  2. Reactive updates: Only triggers when changes occur (no wasteful polling)
  3. Cost efficient: Minimizes SSM API calls through caching and event-driven refresh
  4. Audit trail: EventBridge provides complete change history and monitoring

When to use this pattern?

The tagged storage pattern isn’t universally applicable. Like most architectural approaches, it has ideal use cases where the benefits significantly outweigh the implementation complexity. Consider this pattern when your application matches these characteristics:

  • Multi-tenant SaaS requiring strict tenant isolation and regulatory compliance benefit significantly. The pattern’s infrastructure-level isolation through JWT claims and data model design provides security commitments that application-level isolation cannot match.
  • Microservices architectures with complex configuration requirements across dozens of services find value in the centralized management and flexible storage routing.
  • Organizations managing configurations across multiple storage backends and environments (dev, staging, production, DR) appreciate the hierarchical organization and path-based access control that Parameter Store provides, combined with DynamoDB’s performance for high-frequency access.
  • High-throughput applications (1000+ requests/second) needing sub-millisecond response times use DynamoDB Accelerator (DAX) for in-memory caching. While DynamoDB offers excellent single-digit millisecond latency, DAX delivers microsecond read latency, typically 5-10x faster for cached data. This makes a substantial difference at scale.
  • Teams prioritizing operational simplicity value the event-driven refresh mechanism that avoids manual deployment coordination.

Getting started

Ready to implement the Tagged Storage Pattern in your organization?

Start with a pilot project focusing on a single microservice and gradually expand the pattern across your architecture. The modular design means that you can realize benefits incrementally while building confidence in the approach.

Implementation steps:

  1. Design your data model: Define DynamoDB schema and Parameter Store hierarchy
  2. Set up Amazon Cognito: Configure user pool with custom tenant attributes
  3. Build the service layer: Implement Strategy Pattern for storage routing
  4. Add event-driven refresh: Configure EventBridge rules and Lambda function
  5. Test tenant isolation: Verify JWT validation and cross-tenant access deterrence
  6. Deploy and monitor: Establish CloudWatch dashboards and operational procedures

You can find the complete code for this solution, including AWS CloudFormation templates, deployment and testing scripts, in the GitHub – Configuration Management Service.

To avoid incurring ongoing charges, delete the resources you created during this walkthrough. For detailed cleanup instructions including step-by-step commands and verification steps, see the Infrastructure Cleanup Guide.

Conclusion

Building a multi-tenant configuration service requires careful consideration of storage patterns, security boundaries, and operational requirements. The tagged storage pattern demonstrated in this post provides a flexible, scalable foundation that addresses these challenges through:

  1. Intelligent storage routing: The Strategy Pattern provides optimal backend selection per configuration type, allowing DynamoDB for tenant-specific settings and SSM Parameter Store for shared parameters.
  2. Zero-downtime updates: Event-driven architecture through EventBridge and Lambda avoids service restarts and polling overhead so that configurations refresh immediately upon changes.
  3. Strong tenant isolation: JWT-based authentication with custom claims makes sure tenant boundaries are enforced at the infrastructure level, not application logic, helping prevent cross-tenant access attempts.
  4. Operational simplicity: In-memory caching, combined with event-driven refresh, can reduce API costs while maintaining microsecond response times.
  5. Cost efficiency: Pay-per-request billing, aggressive caching, and Spot instances help keep operational costs minimal even at scale.

Additional resources

About the authors

Mastering millisecond latency and millions of events: The event-driven architecture behind the Amazon Key Suite

Post Syndicated from Ali Ufuk Yucel original https://aws.amazon.com/blogs/architecture/mastering-millisecond-latency-and-millions-of-events-the-event-driven-architecture-behind-the-amazon-key-suite/

Background

Amazon Key empowers customers to securely manage access to their homes and businesses through innovative solutions. Through a suite of consumer and business products, the Amazon Key team is transforming how customers receive deliveries and manage access to their spaces. Our In-Garage Delivery service offers a secure and convenient solution for receiving Amazon packages and groceries directly inside customers’ garages. For property managers and building owners, Amazon Key provides comprehensive access management solutions that enable safe and efficient delivery operations in apartment buildings and gated communities, enhancing both security and convenience for residents.

In this post, we explore how the Amazon Key team used Amazon EventBridge to modernize their architecture, transforming a tightly coupled monolithic system into a resilient, event-driven solution. We explore the technical challenges we faced, our implementation approach, and the architectural patterns that helped us achieve improved reliability and scalability. The post covers our solutions for managing event schemas at scale, handling multiple service integrations efficiently, and building an extensible architecture that accommodates future growth.

Opportunities

Service Coupling and System Fragility

Our legacy architecture faced significant challenges stemming from its tightly coupled design, where service interactions created a complex web of dependencies impacting system stability and scalability. Making service modifications was particularly challenging, as adding or removing services required careful consideration of numerous interdependencies. An incident highlighted this vulnerability when an issue in Service-A triggered a cascade of failures across many upstream services, with increased timeouts leading to retry attempts and ultimately resulting in service deadlocks. System fragility was further demonstrated when problems with a single device vendor, despite being responsible only for specific delivery operations, caused widespread degradation across multiple system services.

Loose Event Schemas

Our old event management infrastructure lacked explicit schema definitions and employed a loosely-typed data architecture, leading to several critical issues. Events were difficult to maintain as use cases expanded, and the absence of formal schema documentation impacted transparency and team collaboration. The design made it almost impossible to implement backward-incompatible changes, such as removing unused fields or events for performance optimization. Without a repository for schema management, team-to-team collaboration for schema modifications (adding fields, removing fields, deprecating fields, or marking fields as required) became challenging. The system also lacked organized validation logic, making it difficult for publishers to identify invalid events before they entered the system. Additionally, the loosely typed schemas lost important semantic context, such as inheritance and composition relationships between different event schemas.

Inconsistent Event Routing and Management

The event routing logic was manually managed and lacked the sophistication needed for growing use cases. The system only supported basic validation of events, primarily checking for required fields, with limited capability for extending validation rules or implementing more complex routing logic. Features that were commonly available in off-the-shelf solutions, such as parallel publishing to multiple subscribers, required significant custom development and ongoing maintenance effort. The implementation only supported a limited number of subscribers to the event pipeline, with no sustainable pathway for adding more consumers. While attempts were made to reduce coupling through SNS/SQS pairs between services, these solutions were implemented on an ad-hoc basis, lacking standardization and creating additional maintenance overhead. This approach led to redundant work and failed to abstract away common functionality, resulting in an inefficient and hard-to-maintain system.These challenges collectively highlighted the need for a more robust and flexible architectural approach that could better serve the system’s evolving needs while improving reliability, maintainability, and scalability.

Design

Given our requirements and the architectural challenges we faced, we implemented a single-bus, multi-account pattern to optimize our system architecture. In this design, each service team maintains complete ownership and autonomy over their application stack, enabling independent development and deployment cycles. Meanwhile, our DevOps team manages a centralized infrastructure stack that encompasses event bus rules, target configurations, and service integrations. This separation of concerns provides several key benefits:

  1. Clear ownership boundaries: Service teams can focus on their core business logic while leveraging a standardized event infrastructure.
  2. Centralized governance: The DevOps team facilitates consistent event routing patterns, security controls, and monitoring across service integrations.
  3. Simplified operations: A single event bus reduces operational complexity while maintaining logical separation through well-defined routing rules.
  4. Enhanced security: The multi-account structure provides natural isolation boundaries while still enabling controlled cross-account event flows.
  5. Streamlined compliance: Centralized management of data exchange patterns makes it easier to implement and maintain compliance requirements.

While EventBridge provided the foundation, we developed additional components to meet our specific requirements.  Our team built three key components: a schema repository serving as the single source of truth for event definitions, a client library that handles schema validation and provides developer-friendly abstractions, and an infrastructure library offering reusable components for subscriber integration.

Event Schema Repository

Amazon EventBridge’s schema discovery and documentation capabilities provide powerful solutions for managing event-driven architectures. The service automatically captures event structures in the schema registry, maintaining versions as events evolve over time. While EventBridge provides developers with tools to implement validation using external solutions or custom application code, it currently does not include native schema validation capabilities. For our organization’s large-scale event-driven architecture, schema validation was a critical requirement. We evaluated two implementation approaches: a centralized validation service or client-side validation at the publisher/subscriber level. The centralized approach would have required managing additional infrastructure, scaling considerations, and introduced latency through extra network hops. After analyzing these factors alongside our requirements for schema governance and team autonomy, we implemented a custom schema repository with client-side validation.

This architecture prioritizes developer experience through immediate validation feedback while maintaining our standards for schema versioning and release management. The repository serves as the foundation for our event-driven architecture, providing essential capabilities for data governance and quality control. By acting as the single source of truth for event definitions, it enables standardized validation across clients, enforces data quality checks, establishes clear ownership boundaries, and maintains comprehensive audit trails for schema changes. Publishers and subscribers leverage these schemas to maintain data consistency and compatibility as their services evolve. The repository has become instrumental in facilitating efficient cross-team collaboration through self-service schema discovery, documentation, and automated validation during development. It maintains a comprehensive registry of event publishers and their corresponding subscribers, providing clear visibility into event flow patterns and dependencies across the system. Teams can quickly manage schema evolution with clear deprecation policies and migration paths, while the system helps detect breaking changes early in the development cycle. This collaborative approach has significantly improved team velocity and reduced integration issues between services.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "$id": "/resource/event/schema/EventV1.json",
    "title": "EventV1",
    "description": "Schema for a simple event.",
    "type": "object",
    "properties": {
        "id": {
            "description": "Id of the event.",
            "type": "string"
        },
        "type": {
            "description": "Type of the event.",
            "$ref": "EventType.json"
        },
        "time": {
            "description": "Time at which the event occurred. It uses ISO 8601 Date Time Format. Reference: https://www.iso.org/iso-8601-date-and-time-format.html",
            "type": "string",
            "format": "date-time"
        },
        "publisher": {
            "description": "Publisher of the event.",
            "$ref": "../core/Publisher.json"
        }
    },
    "required": [
        "id",
        "type",
        "time",
        "publisher"
    ]
}

Client Library

The client library serves as a crucial component for both publishers and subscribers, streamlining their integration with the central event bus. At its core, the library leverages our Event Schema Repository, generating code bindings at build time to provide developers with type-safe and intuitive interfaces for event creation and handling. This approach significantly enhances developer productivity by offering straightforward and convenient methods to construct events and interact with the bus, reducing the likelihood of errors and improving code readability.

A key feature of the client library is its built-in validation mechanism. By utilizing the schemas from our local repository, the library performs thorough validation of events before they are published. This proactive approach catches potential issues early in the development cycle, making sure that only well-formed events conforming to the agreed-upon schemas make it to the event bus. Once validated, the library handles the serialization process and manages the actual publishing of events to the bus, abstracting and simplifying data transformation and transport.

For subscribers, the client library offers equally valuable functionality. It seamlessly handles the deserialization of incoming events, presenting them to the subscribing services in a readily usable format. This feature saves development time and reduces the risk of parsing errors, allowing teams to focus on business logic rather than data handling intricacies. By providing these comprehensive capabilities, our client library has become an indispensable tool in our event-driven network, promoting consistency, reliability, and efficiency across our microservices architecture.

Subscriber Constructs Library

We developed a subscriber constructs library using AWS Cloud Development Kit (CDK) to simplify and standardize the integration process with our central event bus. This library abstracts the setup and management of underlying infrastructure required for event consumption, enabling teams to focus on their core business logic rather than infrastructure configuration details.

The library automates the creation of essential components required for reliable event processing. It provisions a dedicated event bus within the subscriber’s account, establishes the necessary IAM roles and permissions for secure cross-account communication with the central event bus, and configures standardized monitoring and alerting for event processing. This automation not only reduces the potential for configuration errors but also facilitates consistent implementation of our architectural patterns across different teams.

/**
 * Subscriber implementation to provision necessary AWS infrastructure.
 *
 */
const subscription = new Subscription(scope, id, {
    name: "DeliveryService", // Name of your application
    application: {
       region: Region.US_EAST_1, // Region of your Application
    },
});

Conclusion

Amazon Key team’s journey to modernize their architecture and build a resilient, event-driven solution exemplifies the powerful benefits of leveraging AWS EventBridge and adopting a well-designed event-driven architecture. By addressing the challenges of service coupling, loose event schemas, and inconsistent event routing, the team was able to transform their system into a more reliable, scalable, and maintainable resource. The key architectural patterns and components they implemented have had a significant impact on their ability to deliver innovative solutions to their customers.

Reliability and Scale:

  • Built a decoupled event system processing 2000 events/second with 99.99% success rate
  • Achieved consistent 80ms p90 latency from ingestion to target invocation across 14M subscriber calls
  • Avoided the need for new infrastructure for event exchange through standardized event routing
  • Enabled migration of existing complex interdependencies to event-driven architecture

Developer Experience:

  • Reduced service integration time for new use cases from five days to one day (80% improvement)
  • New event onboarding on the Custom Event Schema repository now takes four hours, down from 48 hours
  • Publisher/subscriber integration completed in eight hours, previously took 40 hours
  • Standardized client library addressed 90% of common integration errors

Security and Governance :

  • Single control plane manages 100% of event bus infrastructure
  • Automated security compliance checks catch 100% of unauthorized data exchange patterns
  • Real-time monitoring dashboard tracks every event flow and schema change
  • Schema repository provides complete audit trail for system modifications

The solutions developed by the Amazon Key team provide a blueprint for other organizations looking to modernize their architectures and leverage the power of event-driven design patterns. By adopting similar architectural patterns and components, such as the schema repository and client libraries, other organizations can be empowered to achieve similar benefits.

About the authors

AWS Weekly Roundup: Amazon Bedrock agent workflows, Amazon SageMaker private connectivity, and more (February 2, 2026)

Post Syndicated from Betty Zheng (郑予彬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-bedrock-agent-workflows-amazon-sagemaker-private-connectivity-and-more-february-2-2026/

Over the past week, we passed Laba festival, a traditional marker in the Chinese calendar that signals the final stretch leading up to the Lunar New Year. For many in China, it’s a moment associated with reflection and preparation, wrapping up what the year has carried, and turning attention toward what lies ahead.

Looking forward, next week also brings Lichun, the beginning of spring and the first of the 24 solar terms. In Chinese tradition, spring is often seen as the season when growth begins and new cycles take shape. There’s a common saying that “a year’s plans begin in spring,” capturing the idea that this is a time to set one’s direction and start fresh.

Last week’s launches
Here are the launches that got my attention this week:

  • Amazon Bedrock enhances support for agent workflows with server-side tools and extended prompt caching – Amazon Bedrock introduced two updates that improve how developers build and operate AI agents. The Responses API now supports server-side tool use, so agents can perform actions such as web search, code execution, and database updates within AWS security boundaries. Bedrock also adds a 1-hour time-to-live (TTL) option for prompt caching, which helps improve performance and reduce the cost for long-running, multi-turn agent workflows. Server-side tools are available with OpenAI GPT OSS 20B and 120B models, and the 1-hour prompt caching TTL is generally available for select Claude models by Anthropic in Amazon Bedrock.
  • Amazon SageMaker Unified Studio adds private VPC connectivity with AWS PrivateLink – Amazon SageMaker Unified Studio now supports AWS PrivateLink, providing private connectivity between your VPC and SageMaker Unified Studio without routing customer data over the public internet. With SageMaker service endpoints onboarded into a VPC, data traffic remains within the AWS network and is governed by IAM policies, supporting stricter security and compliance requirements.
  • Amazon S3 adds support for changing object encryption without data movement – Amazon S3 now supports changing the server-side encryption type of existing encrypted objects without moving or re-uploading data. Using the UpdateObjectEncryption API, you can switch from SSE-S3 to SSE-KMS, rotate customer -managed AWS Key Management Service (AWS KMS) keys, or standardize encryption across buckets at scale with S3 Batch Operations while preserving object properties and lifecycle eligibility.
  • Amazon Keyspaces introduces table pre-warming for predictable high-throughput workloads – Amazon Keyspaces (for Apache Cassandra) now supports table pre-warming, which helps you proactively set warm throughput levels so tables can handle high read and write traffic instantly without cold-start delays. Pre-warming helps reduce throttling during sudden traffic spikes, such as product launches or sales events, and works with both on-demand and provisioned capacity modes, including multi-Region tables. The feature supports consistent, low-latency performance while giving you more control over throughput readiness.
  • Amazon DynamoDB MRSC global tables integrate with AWS Fault Injection Service – Amazon DynamoDB multi-Region strong consistency (MRSC) global tables now integrate with AWS Fault Injection Service. With this integration, you can simulate Regional failures, test replication behavior, and validate application resiliency for strongly consistent, multi-Region workloads.

Additional updates
Here are some additional projects, blog posts, and news items that I found interesting:

  • Building zero-trust access across multi-account AWS environments with AWS Verified Access – This post walks through how to implement AWS Verified Access in a centralized, shared-services architecture. It shows how to integrate with AWS IAM Identity Center and AWS Resource Access Manager (AWS RAM) to apply zero trust access controls at the application layer and reduce operational overhead across multi-account AWS environments.
  • Amazon EventBridge increases event payload size to 1 MB – Amazon EventBridge now supports event payloads up to 1 MB, an increase from the previous 256 KB limit. This update helps event-driven architectures carry richer context in a single event, including complex JSON structures, telemetry data, and machine learning (ML) or generative AI outputs, without splitting payloads or relying on external storage.
  • AWS MCP Server adds deployment agent SOPs (preview) – AWS introduced deployment standard operating procedures (SOPs) that AI agents can deploy web applications to AWS from a single natural language prompt in MCP -compatible integrated development environments (IDEs) and command line interfaces (CLIs) such as Kiro, Cursor, and Claude Code. The agent generates AWS Cloud Development Kit (AWS CDK) infrastructure, deploys AWS CloudFormation stacks, and sets up continuous integration and continuous delivery (CI/CD) workflows following AWS best practices. The preview supports frameworks including React, Vue.js, Angular, and Next.js.
  • AWS Network Firewall adds generation AI traffic visibility with web category filtering – AWS Network Firewall now provides visibility into generative AI application traffic through predefined web categories. You can use these categories directly in firewall rules to govern access to generative AI tools and other web services. When combined with TLS inspection, category-based filtering can be applied at the full URL level.
  • AWS Lambda adds enhanced observability for Kafka event source mappings – AWS Lambda introduced enhanced observability for Kafka event source mappings, providing Amazon CloudWatch Logs and metrics to monitor event polling configuration, scaling behavior, and event processing state. The update improves visibility into Kafka-based Lambda workloads, helping teams diagnose configuration issues, permission errors, and function failures more efficiently. The capability supports both Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Apache Kafka event sources.
  • AWS CloudFormation 2025 year in review – This year-in-review post highlights CloudFormation updates delivered throughout 2025, with a focus on early validation, safer deployments, and improved developer workflows. It covers enhancements such as improved troubleshooting, drift-aware change sets, stack refactoring, StackSets updates, and new -IDE and AI -assisted tooling, including the CloudFormation language server and the Infrastructure as Code (IaC) MCP server.

Upcoming AWS events
Check your calendars so that you can sign up for this upcoming event:

AWS Community Day Romania (April 23–24, 2026) – This community-led AWS event brings together developers, architects, entrepreneurs, and students for more than 10 professional sessions delivered by AWS Heroes, Solutions Architects, and industry experts. Attendees can expect expert-led technical talks, insights from speakers with global conference experience, and opportunities to connect during dedicated networking breaks, all hosted at a premium venue designed to support collaboration and community engagement.

If you’re looking for more ways to stay connected beyond this event, join the AWS Builder Center to learn, build, and connect with builders in the AWS community.

Check back next Monday for another Weekly Roundup.

–betty

Serverless ICYMI Q4 2025

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/serverless-icymi-q4-2025/

Stay current with the latest serverless innovations that can transform your applications. In this 31st quarterly recap, discover the most impactful AWS serverless launches, features, and resources from Q4 2025 that you might have missed.

In case you missed our last ICYMI, check out what happened in Q3 2025.

2025 Q4 calendar

2025 Q4 calendar

Serverless at re:Invent 2025

This post covers the biggest serverless announcements from re:Invent 2025, highlighting key feature updates that can improve your applications, and shares valuable resources to keep you informed.

AWS re:Invent 2025 had more than 60,000 in-person attendees and more than 2 million online viewers for the keynotes. The event featured 3,500 sessions from 3,000 speakers, which included information on 530 AWS service and feature announcements.

Keynote Igniting the serverless movement

Keynote Igniting the serverless movement

The serverless content consisted of two tracks: Containers and Serverless (CNS) and Application Integration (API). These tracks included 150 unique sessions watched in-person by more than 16,000 attendees. There were developer-focused experiences including a Road to re:Invent Hackathon, AWS Builder Loft, and Builders Arena. Serverlesspresso, the coffee shop powered by serverless technology, operated in two locations during the event: the Expo Hall and the certification lounge.

Serverless and developer community photo

Serverless and developer community photo

Find a curated list of serverless videos on Serverless Land YouTube.

AWS Lambda durable functions

Managing state across multi-step serverless workflows has traditionally required complex external orchestration tools. AWS Lambda durable functions expand how developers can use Lambda. You can now build reliable multi-step applications and AI workflows directly within Lambda.

AWS Lambda durable functions code

AWS Lambda durable functions code

Durable functions automatically checkpoint progress by saving the current state and completed steps at key points during execution. This allows them to suspend execution for up to one year during long-running tasks and recover from failures by resuming from the last checkpoint rather than restarting from the beginning, all without requiring additional infrastructure management.

Developers can now build in Python or TypeScript, wrap calls in steps with automatic retries and checkpointing. You can use waits to suspend execution for minutes, hours, or even up to a year without paying for idle compute. Durable functions use a replay mechanism to maintain state and handle failures gracefully. The replay mechanism works by re-executing your function code from checkpoints when recovering from failures, ensuring state consistency without data loss. This also means you don’t need complex external orchestration tools for many use cases. This can be helpful for AI workflows and multi-step applications where you need reliable state management without managing external infrastructure.

For more information, read the launch blog post and watch the re:Invent Breakout Session video: Deep Dive on AWS Lambda durable functions (CNS380)

AWS Lambda Managed Instances

Lambda now offers Lambda Managed Instances, a new compute option that combines Amazon EC2 flexibility with fully managed infrastructure. AWS automatically handles instance provisioning, scaling, and maintenance while allowing access to the full range of EC2 capabilities, including Graviton4, network-optimized instances, and other specialized compute options.

AWS Lambda Managed Instances configuration

AWS Lambda Managed Instances configuration

Your functions run on dedicated EC2 capacity from your account, in your own Amazon Virtual Private Cloud (Amazon VPC). AWS still manages the operational overhead, including OS patching, load balancing, and auto-scaling. This gives you access to specialized hardware options while maintaining the serverless operational model. You can further improve costs by using EC2 pricing models, including Compute Savings Plans and Reserved Instances for Lambda workloads. Each instance can handle multiple concurrent requests, making this particularly valuable for high-volume, steady-state workloads where predictable pricing and specific hardware requirements matter.

For more information, read the launch blog post and watch the re:Invent Breakout Session video: Lambda Managed Instances: EC2 Power with Serverless Simplicity (CNS382).

Other Lambda announcements

Multi-tenant SaaS applications face challenges like data leakage between tenants and noisy neighbor effects where one tenant’s workload impacts others. They also struggle with implementing custom isolation mechanisms. Tenant isolation mode addresses these by processing function invocations in separate execution environments for each tenant. This manages tenant-level compute environment isolation automatically.

AWS Lambda tenant isolation

AWS Lambda tenant isolation

Lambda adds Provisioned Mode for Amazon SQS event-source mappings, providing predictable performance and reduced cold starts for high-throughput SQS processing workloads.

You can now send up to 1 MB of data in asynchronous Lambda invocations, increased from 256 KB, helping you build more complex data processing scenarios.

Lambda functions now support IPv6 networking, so you don’t need NAT Gateways when accessing the internet or other AWS services from VPC-connected functions.

Lambda internet connectivity through a NAT Gateway (IPv4) and Lambda internet connectivity through an egress-only internet gateway (IPv6).

Lambda internet connectivity through a NAT Gateway (IPv4) and Lambda internet connectivity through an egress-only internet gateway (IPv6).

Lambda Rust support is now generally available, moving from experimental status. This is backed by AWS Support and the Lambda availability SLA.

Lambda has expanded its runtime support by adding Python 3.14, Node.js 24, and Java 25 as both managed runtimes and container base images, providing access to the latest language features and ensuring long-term support.

Amazon ECS

Amazon Elastic Container Service (Amazon ECS) Express Mode streamlines the deployment and management of containerized applications by automating the infrastructure setup that traditionally slows down developers.

Amazon ECS Express Mode deployment

Amazon ECS Express Mode deployment

This means you can focus on building applications while deploying with confidence using AWS best practices. Express Mode lets you deploy production-ready containerized web applications and APIs with a single command. This automatically handles domains, networking, load balancing, AWS Identity and Access Management (IAM) roles, and auto-scaling through simplified APIs. When your applications evolve and require advanced features, you can seamlessly configure and access the full capabilities of the resources, including Amazon ECS. Learn more from the launch blog post.

Amazon ECS announced a public preview of a fully managed MCP server, enabling AI-powered experiences for development and operations. The Model Context Protocol (MCP) server provides enterprise-grade capabilities like automatic updates and patching, centralized security through AWS IAM integration, comprehensive audit logging via AWS CloudTrail, and the proven scalability, reliability, and support of AWS.

Amazon Elastic Container Registry (ECR) managed container image signing enhances your security posture and eliminates the operational overhead of setting up signing. Container image signing allows you to verify that images are from trusted sources. ECR automatically signs images as they are pushed using the identity of the entity pushing the image. Signing operations are logged through CloudTrail for full auditability.

Amazon API Gateway

Amazon API Gateway allows you to improve the responsiveness of your REST APIs by progressively streaming response payloads back to the client. With this new capability, you can use streamed responses to enhance user experience when building LLM-driven applications (such as AI agents and chatbots), improve time-to-first-byte (TTFB) performance for web and mobile applications, stream large files, and perform long-running operations while reporting incremental progress using protocols such as server-sent events (SSE).

Amazon API Gateway streaming

API Gateway introduces private integration with Application Load Balancers (ALBs). You can use this to expose your VPC-based applications securely through REST APIs without exposing your ALBs to the public internet.

You can also now configure enhanced TLS security policies on API endpoints and custom domain names, providing you with greater control over the security posture of your APIs.

Amazon EventBridge

Amazon EventBridge introduced an enhanced visual rule builder that helps developers discover and subscribe to events from custom applications and over 200 AWS services. The console-based interface integrates the EventBridge schema registry with a comprehensive event catalog and intuitive drag-and-drop canvas that simplifies building event-driven applications. Developers can browse and search through events with readily available sample payloads and schemas without having to hunt through individual service documentation. The schema-aware visual builder guides developers through creating event filter patterns and rules, reducing syntax errors and accelerating development time.

EventBridge also allows targeting SQS fair queues.

AWS Step Functions

AWS Step Functions allows for enhanced local testing through the TestState API, providing programmatic access to comprehensive testing capabilities without deploying to AWS. This helps you build automated test suites that validate your workflow definitions locally on your development machines. Test error handling patterns, data transformations, and mock service integrations using your preferred testing frameworks.

There is also a new metrics dashboard, giving you visibility into your workflow operations at both the account and state machine levels.

Other announcements

Savings Plans flexible pricing model extends to AWS managed database services with the launch of Database Savings Plans. This helps reduce database costs by up to 35% when committing to a consistent amount of usage ($/hour) over a 1-year term. Savings automatically apply each hour to eligible usage across supported database services, and additional usage beyond the commitment is billed at on-demand rates.

Amazon DynamoDB now supports multi-attribute composite keys in global secondary indexes. You no longer need to concatenate values into synthetic keys manually, which sometimes results in the need to backfill data before adding new indexes. Instead, you can create primary keys using up to eight existing attributes, making it easier to model diverse access patterns and adapt to new query requirements.

Amazon Bedrock introduced AgentCore with quality evaluations and policy controls for deploying trusted AI agents at scale.

Bedrock also added 18 fully managed open weight models, expanding AI model options for developers.

The Strands Agents SDK is an open source framework that takes a model-driven approach to building and running AI agents in just a few lines of code. TypeScript support is now available in preview so you can choose between Python and TypeScript for building Strands Agents.

Amazon S3 Vectors became generally available. S3 Vectors delivers purpose-built, cost-optimized vector storage for AI agents, inference, Retrieval Augmented Generation (RAG), and semantic search at billion-vector scale.

Serverless blog posts

October

November

Serverless Office Hours

Join our livestream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. Episodes are available on-demand at serverlessland.com/office-hours.

October

November

December

Still looking for more?

The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Serverless Developer Advocacy team to see the latest news, follow conversations, and interact with the team.

And finally, visit Serverless Land for all your serverless needs.

More room to build: serverless services now support payloads up to 1 MB

Post Syndicated from Anton Aleksandrov original https://aws.amazon.com/blogs/compute/more-room-to-build-serverless-services-now-support-payloads-up-to-1-mb/

To support cloud applications that increasingly depend on rich contextual data, AWS has raised the maximum payload size from 256 KB to 1 MB for asynchronous AWS Lambda function invocations, Amazon Simple Queue Service (Amazon SQS), and Amazon EventBridge. Developers can use this enhancement to build and maintain context-rich event-driven systems and reduce the need for complex workarounds such as data chunking or external large object storage.

Overview

Modern cloud applications rely on context-rich, structured data to drive intelligent behavior. Large language model (LLM) prompts, telemetry signals, personalization data, machine learning (ML) outputs, and user interaction logs are no longer simple strings. Instead, they’re typically complex, nested JSON or YAML objects carrying meaningful context. Previously, developers working with serverless services such as Amazon SQS, Lambda (asynchronous invocations and Amazon SQS event-source mapping), or EventBridge had to carefully manage their data to fit within the 256 KB payload size limit. This commonly meant chunking larger payloads, externalizing payloads to object stores such as Amazon S3, or using data compression. These workarounds added complexity and latency, creating edge cases that were difficult to monitor and debug.

With the recent launches, you can now transmit payloads up to 1 MB, significantly reducing the need for complex data chunking and architectural workarounds. This increased capacity streamlines design patterns, reduces operational overhead, and makes event-driven systems more intuitive to build and maintain. Developers can now include richer data in single payloads—from detailed LLM prompts and full system states to comprehensive context and complete transaction histories.

The new 1 MB payload size limit applies to asynchronous Lambda function invocations, whether you trigger them using either SQS event-source mapping, AWS Command Line Interface (AWS CLI), AWS SDKs, Lambda Invoke API, or AWS services such as EventBridge. The increased limit also extends to all messages and events flowing through Amazon SQS queues and EventBridge Event Buses.

Getting started

There’s nothing you need to do to get started. This enhancement is automatically applied to all new and existing Lambda functions, SQS queues, and EventBridge Event Buses.

If you were previously chunking data at 256KB (or lower) threshold, then you might need to make changes to your service configurations or business logic code to start using the new limit. For example, if you’ve explicitly set Amazon SQS MaximumMessageSize attribute, then you might need to adjust it to a new desired value. Larger payloads might also result in higher costs, as described in the following section.

Real-world example: rich event context in agentic event-driven architectures

Event-driven architectures allow services to operate independently without centralized coordination. In these systems, comprehensive event context is essential. With the increased 1 MB payload limit, events can now carry more comprehensive data—from user profiles and order details to historical interactions. This enables services such as inventory, shipping, and notifications to act autonomously.

Consider the following example. In hospitality and quick-service industries, customer satisfaction depends on timely, thoughtful service recovery. When a guest submits negative feedback through a survey, review, or complaint form, service teams must gather context, interpret the issue, and craft a response. Traditionally, this meant manually piecing together visit logs, loyalty data, and prior complaints. Now, this can be fully automated using an AI agent powered by AWS serverless services and Amazon Bedrock, as shown in the following figure.

Figure 1: Customer feedback processing pipeline

The workflow:

  1. Receive: A new review is submitted through the Review application and emitted as an event to EventBridge Event Bus.
  2. Detect: Event Bus delivers the event to downstream Feedback analysis agent. The agent running in a Lambda function recognizes the review as low-rating or complaint.
  3. Enrich: The agent collects the guest’s visit metadata, booking details, loyalty activity, and complaint history using attached MCP tools into a single structured JSON payload (up to 1 MB).
  4. Queue: The payload is sent to an SQS queue for further asynchronous processing by downstream components.
  5. Generate: A separate Lambda function polls messages from Amazon SQS and invokes an Amazon Bedrock model to analyze the full complaint context, draft a personalized response, suggest a gesture (such as a refund or credit), and classify issue severity.
  6. Deliver: The message is logged and sent to the customer, and to the service team for further analysis.

This use case demonstrates the importance of having a rich context: current and previous visits details, loyalty tier, prior interactions, and feedback history. Previously, teams had to offload pieces of context to Amazon S3 and reference them externally, adding latency and architectural complexity. The new 1 MB payload size means that all this information can be transported together, improving the serverless agentic workflow efficiency and streamlining maintenance.

Best practices when using large payloads

The following sections outline best practices that you should apply when using larger payloads.

Performance considerations

Monitor Lambda function memory usage carefully when working with larger payloads, because parsing and processing complex JSON objects can increase memory usage and execution duration. Test your systems thoroughly under load, especially for high-throughput applications, by benchmarking with realistic payload sizes and traffic patterns. Although the payload limit has increased to 1 MB, the Lambda 15-minute timeout and memory limits remain unchanged. When applicable, you can use compression to process even larger datasets efficiently, but remember to account for the added CPU overhead of compression and decompression in your performance calculations. Read the Monitoring best practices for event delivery with Amazon EventBridge post for more best practices to tune your event-driven architectures performances.

Operational guidelines

Configure dead-letter-queues (DLQ) to make sure that failed messages are retained for inspection and troubleshooting. This becomes especially important with larger payloads, because debugging complex data structures necessitates access to the complete message context. Implement robust error handling and retries to manage transient failures, particularly when processing rich payload content that may contain nested structures or complex relationships.

To further optimize throughput, you can batch similar smaller events together into a single payload. However, avoid mixing unrelated events and maintain clear boundaries between different business domains and processes.

Always make sure that your downstream dependencies are capable of handling larger payloads.

When to use external storage

Even with the increased 1 MB payload limit, there are scenarios where patterns such as claim check remain a sound architectural choice. These patterns involve storing a full payload in an external system, such as Amazon S3, and passing a lightweight reference through your event stream. This approach continues to provide value when payloads exceed the new limit, when data needs to be reused by multiple consumers, or when strict governance, traceability, and security requirements are involved. For example, audit logs, image metadata, or large ML inference inputs may still surpass the 1 MB boundary, even when compressed. Instead of risking truncation or fragmentation, a claim check enables consistent, scalable access to the complete data set.

You can use open source libraries such as the Kafka sink connector for EventBridge and Amazon SQS Extended Client Library (available for Python and Java) that abstract complexities of storing large objects in external storage.

Cost management

Although larger payloads enable richer context in your applications, logging full payloads can increase storage and processing costs. Services such as CloudWatch Logs charge based on data volume, thus implementing selective logging, payload truncation, or sampling becomes crucial for high-volume events. Consider logging only essential fields or implementing smart sampling strategies based on business importance.

For full payload archival and retention, evaluate cost-effective storage solutions such as Amazon S3 with appropriate lifecycle policies. This can include moving older logs to cheaper storage tiers or implementing automated cleanup procedures for non-critical data. Balance your retention needs with cost optimization by defining clear policies for what data needs to be kept and for how long.

Review the pricing pages for AWS Lambda, Amazon EventBridge, and Amazon SQS to learn about the costs of delivering and processing events and messages.

Conclusion

The increase in maximum payload size from 256 KB to 1 MB enables developers to build more efficient distributed architectures. You can use this enhancement to transport richer context in event and message payloads, reducing the need for complex workarounds that previously added architectural complexity and operational overhead. This added room to transmit rich context means that you can streamline your workflows, improve observability, and reduce architectural complexity whether using choreography or orchestration patterns.

Go to the developer guides for AWS Lambda, Amazon EventBridge, and Amazon SQS, to learn more about how to take advantage of this update.

To learn more about serverless architectures, visit Serverless Land.

Automate and orchestrate Amazon EMR jobs using AWS Step Functions and Amazon EventBridge

Post Syndicated from Senthil Kamala Rathinam original https://aws.amazon.com/blogs/big-data/automate-and-orchestrate-amazon-emr-jobs-using-aws-step-functions-and-amazon-eventbridge/

Many enterprises are adopting Apache Spark for scalable data processing tasks such as extract, transform, and load (ETL), batch analytics, and data enrichment. As data pipelines evolve, the need for flexible and cost-efficient execution environments that support automation, governance, and performance at scale also evolve in parallel. Amazon EMR provides a powerful environment to run Spark workloads, and depending on workload characteristics and compliance requirements, teams can choose between fully managed options like Amazon EMR Serverless or more customizable configurations using Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2).

In use cases where infrastructure control, data locality, or strict security postures are essential, such as in financial services, healthcare, or government, running transient EMR on EC2 clusters becomes a preferred choice. However, orchestrating the full lifecycle of these clusters, from provisioning to job submission and eventual teardown, can introduce operational overhead and risk if done manually.

To streamline this process, the AWS Cloud offers built-in orchestration capabilities using AWS Step Functions and Amazon EventBridge. Together, these services help you automate and schedule the entire EMR job lifecycle, reducing manual intervention while optimizing cost and compliance. Step Functions provides the workflow logic to manage cluster creation, Spark job execution, and cluster termination, and EventBridge schedules these workflows based on business or operational needs.

In this post, we discuss how to build a fully automated, scheduled Spark processing pipeline using Amazon EMR on EC2, orchestrated with Step Functions and triggered by EventBridge. We walk through how to deploy this solution using AWS CloudFormation, processes COVID-19 public dataset data in Amazon Simple Storage Service (Amazon S3), and store the aggregated results in Amazon S3. This architecture is ideal for periodic or scheduled batch processing scenarios where infrastructure control, auditability, and cost-efficiency are critical.

Solution overview

This solution uses the publicly available COVID-19 dataset to illustrate how to build a modular, scheduled architecture for scalable and cost-efficient batch processing for time-bound data workloads.The solution follows these steps:

  1. Raw COVID-19 data in CSV format is stored in an S3 input bucket.
  2. A scheduled rule in EventBridge triggers a Step Functions workflow.
  3. The Step Functions workflow provisions a transient Amazon EMR cluster using EC2 instances.
  4. A PySpark job is submitted to the cluster to calculate COVID-19 hospital utilization data to compute monthly state-level averages of inpatient and ICU bed utilization, and COVID-19 patient percentages.
  5. The processed results are written back to an S3 output bucket.
  6. After successful job completion, the EMR cluster is automatically deleted.
  7. Logs are persisted to Amazon S3 for observability and troubleshooting.

By automating this workflow, you alleviate the need to manually manage EMR clusters while gaining cost-efficiency by running compute only when needed. This architecture is ideal for periodic Spark jobs such as ETL pipelines, regulatory reporting, and batch analytics, especially when control, compliance, and customization are required.The following diagram illustrates the architecture for this use case.

The infrastructure is deployed using AWS CloudFormation to provide consistency and repeatability. AWS Identity and Access Management (IAM) roles grant least‑privilege access to Step Functions, Amazon EMR, EC2 instances, and S3 buckets, and optional AWS Key Management Service (AWS KMS) encryption can secure data at rest in Amazon S3 and Amazon CloudWatch Logs. By combining a scheduled trigger, stateful orchestration, and centralized logging, this solution delivers a fully automated, cost‑optimized, and secure way to run transient Spark workloads in production.

Prerequisites

Before you get started, make sure you have the following prerequisites:

Set up resources with AWS CloudFormation

To provision the required resources using a single CloudFormation template, complete the following steps:

  1. Sign in to the AWS Management Console as an admin user.
  2. Clone the sample repository to your local machine or AWS CloudShell and navigate into the project directory. 
    git clone https://github.com/aws-samples/sample-emr-transient-cluster-step-functions-eventbridge.git
cd sample-emr-transient-cluster-step-functions-eventbridge

  3. Set an environment variable for the AWS Region where you plan to deploy the resources. Replace the placeholder with your Region code, for example, us-east-1. 
    export AWS_REGION=<YOUR AWS REGION>

  4. Deploy the stack using the following command. Update the stack name if needed. In this example, the stack is created with the name covid19-analysis. 
    aws cloudformation deploy \
--template-file emr_transient_cluster_step_functions_eventbridge.yaml \
--stack-name covid19-analysis \
--capabilities CAPABILITY_IAM \
--region $AWS_REGION

You can monitor the stack creation progress on the AWS CloudFormation console on the Events tab. The deployment typically completes in under 5 minutes.

After the stack is successfully created, go to the Outputs tab on the AWS CloudFormation console and note the following values for use in later steps:

  • InputBucketName
  • OutputBucketName
  • LogBucketName

Set up the COVID-19 dataset

With your infrastructure in place, complete the following steps to set up the input data:

  1. Download the COVID-19 data CSV file from HealthData.gov to your local machine.
  2. Rename the downloaded file to covid19-dataset.csv.
  3. Upload the renamed file to your S3 input bucket under the raw/ folder path.

Set up the PySpark Script

Complete the following steps to set up the PySpark script:

  1. Open AWS CloudShell from the console.
  2. Confirm that you are working inside the sample-emr-transient-cluster-step-functions-eventbridge directory before running the next command.
  3. Copy the PySpark script needed for this walkthrough into your input bucket: 
    aws s3 cp covid19_processor.py s3://<InputBucketName>/scripts/

This script processes COVID-19 hospital utilization data stored as CSV files in your S3 input bucket. When running the job, provide the following command-line arguments:

  • --input – The S3 path to the input CSV files
  • --output – The S3 path to store the processed results

The script reads the raw dataset, standardizes various date formats, and filters out records with invalid or missing dates. It then extracts key utilization metrics such as inpatient bed usage, ICU bed usage, and the percentage of beds occupied by COVID-19 patients and calculates monthly averages grouped by state. The aggregated output is saved as timestamped CSV files in the specified S3 location.

This example demonstrates how you can use PySpark to efficiently clean, transform, and analyze large-scale healthcare data to gain actionable insights on hospital capacity trends during the pandemic.

Configure a schedule in EventBridge

The Step Functions state machine is by default scheduled to run on December 31, 2025, as a one-time execution. You can update the schedule for recurring or one-time execution as needed. Complete the following steps:

  1. On the EventBridge console, choose Schedules under Scheduler in the navigation pane.
  2. Select the schedule named <StackName>-covid19-analysis and choose Edit.
  3. Set your preferred schedule pattern.
    1. If you want to run the schedule one time, select One-time schedule for Occurrence and enter a date and time.
    2. If you want to run this on a recurring basis, select Recurring schedule. Specify the schedule type as either Cron-based schedule or Rate-based schedule as needed.
  4. Choose Next twice and choose Save schedule.

Start the workflow in Step Functions

Based on your EventBridge schedule, the Step Functions workflow will run automatically. For this walkthrough, complete the following steps to trigger it manually:

  1. On the Step Functions console, choose State machines in the navigation pane.
  2. Choose the state machine that begins with Covid19AnalysisStateMachine-*.
  3. Choose Start execution.
  4. In the Input section, provide the following JSON (provide the log bucket and output bucket names with the appropriate values captured earlier): 
    {
  "LogUri": "s3://<LogBucketName>/logs/",
  "OutputS3Location": "s3://<OutputBucketName>/processed/"
}

  5. Choose Start execution to initiate the workflow.

Monitor the EMR job and workflow execution

After you start the workflow, you can track both the Step Functions state transitions and the EMR job progress in real time on the console.

Monitor the Step Functions state machine

Complete the following steps to monitor the Step Functions state machine:

  1. On the Step Functions console, choose State machines in the navigation pane.
  2. Choose the state machine that begins with Covid19AnalysisStateMachine-*.
  3. Choose the running execution to view the visual workflow.

    Each state node will update as it progresses—green for success, red for failure.

  4. To explore a step, choose its node and inspect the input, output, and error details in the side pane.

The following screenshot shows an example of a successfully executed workflow.

Monitor the EMR cluster and EMR step

Complete the following steps to monitor the EMR cluster and EMR step status:

  1. While the cluster is active, open the Amazon EMR console and choose Clusters in the navigation pane.
  2. Locate the Covid19Cluster transient EMR cluster.
    Initially, it will be in Starting status.

    On the Steps tab, you can see your Spark submit step listed. As the job progresses, the step status changes from Pending to Running to finally Completed or Failed.

  3. Choose the Applications tab to view the application UIs, in which you can access the Spark History Server and YARN Timeline Server for monitoring and troubleshooting.

Monitor CloudWatch logs

To enable CloudWatch logging and enhanced monitoring for your EMR on EC2 cluster, refer to Amazon EMR on EC2 – Enhanced Monitoring with CloudWatch using custom metrics and logs. This guide explains how to install and configure the CloudWatch agent using a bootstrap action, so you can stream system-level metrics (such as CPU, memory, and disk usage) and application logs from EMR nodes directly to CloudWatch. With this setup, you can gain real-time visibility into cluster health and performance, simplify troubleshooting, and retain critical logs even after the cluster is terminated.

For this walkthrough, check the logs in the S3 log output location.

Confirm cluster deletion

When the Spark step is complete, Step Functions will automatically delete the Amazon EMR cluster. Refresh the Clusters page on the Amazon EMR console. You should see your cluster status change from Terminating to Terminated within a minute.

By following these steps, you gain full end-to-end visibility into your workflow from the moment the Step Functions state machine is triggered to the automatic shutdown of the EMR cluster. You can monitor execution progress, troubleshoot issues, confirm job success, and continuously optimize your transient Spark workloads.

Verify job output in Amazon S3

When the job is complete, complete the following steps to check the processed results in the S3 output bucket:

  1. On the Amazon S3 console, choose Buckets in the navigation pane.
  2. Open the output S3 bucket you noted earlier.
  3. Open the processed folder.
  4. Navigate into the timestamped subfolder to view the CSV output file.
  5. Download the CSV file to view the processed results, as shown in the following screenshot.

Monitoring and troubleshooting

To monitor the progress of your Spark job running on a transient EMR on EC2 cluster, use the Step Functions console. It provides real-time visibility into each state transition in your workflow, from cluster creation and job submission to cluster deletion. This makes it straightforward to track execution flow and identify where issues might occur.During job execution, you can use the Amazon EMR console to access cluster-level monitoring. This includes YARN application statuses, step-level logs, and overall cluster health. If CloudWatch logging is enabled in your job configuration, driver and executor logs stream in near real time, so you can quickly detect and diagnose errors, resource constraints, or data skew within your Spark application.

After the workflow is complete, regardless of whether it succeeds or fails, you can perform a detailed post-execution analysis by reviewing the logs stored in the S3 bucket specified in the LogUri parameter. This log directory includes standard output and error logs, along with Spark history files, offering insights into execution behavior and performance metrics.

For continued access to the Spark UI during job execution, you can use persistent application UIs on the EMR console. These links remain accessible even after the cluster is stopped, enabling deeper root-cause analysis and performance tuning for future runs.

This visibility into both workflow orchestration and job execution can help teams optimize their Spark workloads, reduce troubleshooting time, and build confidence in their EMR automation pipelines.

Clean up

To avoid incurring ongoing charges, clean up the resources provisioned during this walkthrough:

  1. Empty the S3 buckets:
    1. On the Amazon S3 console, choose Buckets in the navigation pane.
    2. Select the input, output, and log buckets used in this tutorial.
    3. Choose Empty to remove all objects before deleting the buckets (optional).
  2. Delete the CloudFormation stack:
    1. On the AWS CloudFormation console, choose Stacks in the navigation pane.
    2. Select the stack you created for this solution and choose Delete.
    3. Confirm the deletion to remove associated resources.

Conclusion

In this post, we showed how to build a fully automated and cost-effective Spark processing pipeline using Step Functions, EventBridge, and Amazon EMR on EC2. The workflow provisions a transient EMR cluster, runs a Spark job to process data, and stops the cluster after the job completes. This approach helps reduce costs while giving you full control over the process. This solution is ideal for scheduled data processing tasks such as ETL jobs, log analytics, or batch reporting, especially when you need detailed control over infrastructure, security, and compliance settings.

To get started, deploy the solution in your environment using the CloudFormation stack provided and adjust it to fit your data processing needs. Check out the Step Functions Developer Guide and Amazon EMR Management Guide to explore further.

Share your feedback and ideas in the comments or connect with your AWS Solutions Architect to fine-tune this pattern for your use case.

About the authors

Senthil Kamala Rathinam

Senthil Kamala Rathinam

Senthil is a Solutions Architect at Amazon Web Services, specializing in Data and Analytics for banking customers across North America. With deep expertise in Data and Analytics, AI/ML, and Generative AI, he helps organizations unlock business value through data-driven transformation. Beyond work, Senthil enjoys spending time with his family and playing badminton.

Shashi Makkapati

Shashi Makkapati

Shashi is a Senior Solutions Architect serving banking customers across North America. He specializes in data analytics, AI/ML, and generative AI, focusing on innovative solutions that transform financial organizations. Shashi is passionate about leveraging technology to solve complex business challenges in the banking sector. Outside of work, he enjoys traveling and spending quality time with his family.

Enhance Amazon EMR observability with automated incident mitigation using Amazon Bedrock and Amazon Managed Grafana

Post Syndicated from Yu-Ting Su original https://aws.amazon.com/blogs/big-data/enhance-amazon-emr-observability-with-automated-incident-mitigation-using-amazon-bedrock-and-amazon-managed-grafana/

Maintaining high availability and quick incident response for Amazon EMR clusters is important in data analytics environments. In this post, we show you how to build an automated observability system that combines Amazon Managed Grafana with Amazon Bedrock to detect and remediate EMR cluster issues. We demonstrate how to integrate real-time monitoring with AI-powered remediation suggestions, combining Amazon Managed Grafana for visualization, Amazon Bedrock for intelligent response recommendations, and AWS Systems Manager for automated remediation actions on Amazon Web Services (AWS).

Solution overview

This solution helps you improve EMR cluster observability through a comprehensive four-layer architecture—comprising monitoring, notification, remediation, and knowledge management—to provide the following features:

  • Real-time monitoring of EMR clusters using Amazon Managed Service for Prometheus and Amazon Managed Grafana
  • Automated first-aid remediation through Systems Manager
  • AI-powered incident response suggestions using Amazon Bedrock
  • Integration with the AWS Premium Support knowledge base
  • Historical incident data archival and analysis

The implementation of this architecture delivers the following key benefit:

  • Reduced Mean time to resolution (MTTR)
  • Proactive incident prevention
  • Automated first-response actions
  • Knowledge base enrichment through machine learning

The following diagram illustrates the solution architecture.

End-to-end AWS monitoring solution diagram integrating Knowledge Center, Support, CloudWatch metrics with EventBridge rules and Lambda processing

The architecture comprises the following core components:

  • Monitoring layer – The monitoring layer uses Amazon Managed Service for Prometheus and Amazon CloudWatch to capture real-time metrics from EMR clusters. Amazon Managed Grafana serves as the visualization layer, offering comprehensive dashboards for Apache YARN, HDFS, Apache HBase, and Apache Hudi performance monitoring. Advanced alerting mechanisms trigger notifications based on predefined query results.
  • Notification layer – To provide timely and reliable alert delivery, the notification layer uses Amazon Simple Notification Service (Amazon SNS) for distribution and Amazon Simple Queue Service (Amazon SQS) for message queuing. This architecture prevents message delays and provides a robust trigger mechanism for AWS Lambda functions.
  • Remediation layer – The remediation layer enables automatic issue resolution through:
    • Lambda functions for orchestration
    • Systems Manager for script execution
    • Amazon Bedrock (amazon.nova-lite-v1:0) for generating intelligent response recommendations
  • Knowledge management layer – To maintain an up-to-date knowledge base, the solution:

We provide an AWS CloudFormation template to deploy the solution resources.

Prerequisites

Before starting this walkthrough, make sure you have access to the following AWS resources and configurations:

  • An AWS account
  • Access to the US East (N. Virginia) AWS Region
    • Add access to Amazon Bedrock foundation models (amazon.nova-lite-v1:0)

  • Amazon EMR version 6.15.0 (used in this demo)
  • Archived technical or troubleshooting articles
  • AWS IAM Identity Center enabled with at least one role that can become a Grafana administrator
  • (Optional) AWS Premium Support with a business support plan or higher for enhanced troubleshooting capabilities

Throughout this walkthrough, we provide detailed instructions to set up and configure these prerequisites if you haven’t already done so.

Configure resources using AWS CloudFormation

Complete the following steps to configure your resources:

  1. Launch the CloudFormation stack:

launch stack

  1. Provide emrobservability as the stack name.
  2. Select a virtual private cloud (VPC) and assign a public subnet.
  3. For EMRClusterName, enter a name for your cluster (default: emrObservability).
  4. Enter an existing Amazon S3 location as the Apache HBase root directory location (for example, s3://mybucket/my/hbase/rootdir/).
  5. For MasterInstanceType and CoreInstanceType, enter your instance types (default: m5.xlarge for both).
  6. For CoreInstanceCount, enter your instance count (default: 2).
  7. For SSHIPRange, use CheckIp and enter your IP (for example, 10.1.10/32).
  8. Choose the release label (default: 6.15.0).
  9. For KeyName, enter a key name to SSH to Amazon Elastic Compute Cloud (Amazon EC2) instances.
  10. For LatestAmiId, enter your AMI (default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2).
  11. For KBS3Bucket, enter a name for your S3 bucket (for example, mykbbucket).
  12. For SubscriptionEndpoint, enter an email address to receive notifications and responses (for example, [email protected]).

Accept subscription confirmation

Accept the subscription confirmation sent to the email address you specified in the CloudFormation stack parameters. The following screenshot shows an example of the email you receive.

AWS email confirmation for SNS topic subscription to QA Lambda function responses with opt-out instructions

Prepare the knowledge base

Complete the following steps to populate the S3 bucket with archived technical articles and cases:

  1. On the Lambda console, choose Functions in the navigation pane.
  2. Choose the function CustomFunctionCopyKCArticlesToS3Bucket.

AWS Lambda console displaying Functions page with CustomFunctionCopyKCArticlesToS3Bucket function details

  1. Manually invoke the function by choosing Test on the Test tab.

AWS Lambda Test tab interface with event configuration options

  1. Verify successful execution by checking the CloudWatch logs.

AWS Lambda successful function execution result with null output

  1. Repeat the process for the Lambda function CustomFunctionCopyCasesToS3Bucket.

Lambda function interface displaying CustomFunctionCopyCasesToS3Bucket configuration with CloudFormation ID and description panel

AWS Lambda test interface showing Test event configuration options and action buttons

AWS Lambda function execution success message with null response and SHA-256 code

  1. Confirm the S3 bucket has been populated with archived technical articles and cases.

Amazon S3 bucket interface showing two folders with action buttons and search functionality

Sync data to the Amazon Bedrock knowledge base

Complete the following steps to sync the data to your knowledge base:

  1. On the Lambda console, choose Functions in the navigation pane.
  2. Choose the function KBDataSourceSync.

AWS Lambda console displaying filtered functions with CloudFormation tags, Python runtime versions, and modification timestamps

  1. Manually invoke the function by choosing Test on the Test tab.

This task might take 10–15 minutes to complete.

AWS Lambda console test configuration panel with CloudWatch integration and event creation controls

  1. Verify successful execution by checking the CloudWatch logs.

Lambda function execution results showing successful completion status and details

Configure your Amazon Managed Grafana workspace

Complete the following steps to configure your Amazon Managed Grafana workspace:

  1. On the Amazon Managed Grafana console, choose Workspaces in the navigation pane.
  2. Open your workspace.
  3. Choose Assign new user or group.

Amazon Grafana workspace showing IAM configuration notice and user assignment button

  1. Select your IAM Identity Center role and choose Assign users and groups.

Amazon Grafana IAM Identity Center user assignment panel with search and selection controls

  1. On the Admin dropdown menu, choose Make admin.

Amazon Grafana user list showing assigned viewer with admin action options

  1. Enable Grafana alerting, then choose Save changes.

Amazon Grafana alerting configuration panel showing disabled status with navigation tabs and edit button

Amazon Grafana configuration panel showing enabled alerting and plugin management settings

  1. Wait 10 minutes for the workspace to become active.
  2. When it’s active, sign in to the Grafana workspace. (For more information, refer to Connect to your workspace.)

Configure data sources

Add and configure the following data sources:

  1. For Service, choose CloudWatch, then select your Region and add CloudWatch as a data source.

  1. Choose Amazon Managed Service for Prometheus as a second data source and select your Region.

  1. Validate CloudWatch connectivity:
    1. Run test queries (for example, Namespace: AWS/EC2, Metric name: CPUUtilization, Statistic: Maximum).
      Amazon Managed Gragana interface showing CPU utilization query setup for EC2 instance.
    2. Verify CloudWatch metric retrieval.
      Line graph showing CPU utilization over time with peak at 40%.
  1. Validate Amazon Managed Service for Prometheus connectivity:
    1. Run test queries (for example, Metric: hadoop_hbase_numregionservers, Label filters: cluster_id = <Amazon EMR cluster ID>).
      Amazon Managed Grafana query interface showing Hadoop HBase metric configuration.
    2. Verify Prometheus metric retrieval.
      Amazon Managed Grafana monitoring dashboard showing a graph with HBase Region Server amount from 0 to 2

Confirm SNS notification channels

Complete the following steps to confirm your SNS notification is set up:

  1. On the Amazon SNS console, choose Topics in the navigation pane.
  2. Locate and note the ARNs for -LambdaFunctionTopic and -QALambdaFunctionTopic.

AWS SNS Topics list showing 4 topics with names, types, and ARNs

AWS SNS Topics console showing filtered search results for "LambdaFunctionTopic"

AWS SNS Topics console showing filtered search results for "QALambdaFunctionTopic"

  1. Choose Contact points under Alerting.

  1. Create the first contact point:
    1. For Name, enter SNS_SSM.
    2. For Integration, choose AWS SNS.
    3. For Topic, enter the ARN for LambdaFunctionTopic.
    4. For Auth Provider, choose Workspace IAM role.
    5. For Alert Message format, choose JSON.

  1. Create the second contact point:
    1. For Name, enter SNS_QA.
    2. For Integration, choose AWS SNS.
    3. For Topic, enter the ARN for QALambdaFunctionTopic.
    4. For Auth Provider, choose Workspace IAM role.
    5. For Alert Message format, choose JSON.

Create alert rules

Complete the following steps to set up two critical alert rules:

  1. Choose Alert rules under Alerting.

  1. Set up alerting if the Apache HBase region server status is abnormal:
    1. For Alert name, enter HBase region server down.
    2. For Data source, choose Amazon Managed Service for Prometheus.
    3. For Metric, choose hadoop_hbase_numregionservers.
      Alert rule configuration interface for HBase region server monitoring
    4. For Threshold, configure to alert if the region server count is less than 2 for 3 minutes.
      Amazon Managed Grafana alert rule configuration interface with expressions setup
    5. For Evaluation interval, set to 1 minute.
      New evaluation group creation modal showing P0_RegionServer name input and 1m interval settingHBase alert configuration panel showing P0_RegionServer group and 3m pending period
    6. For Contact point, choose SNS_SSM.
      Amazon Managed Grafana alert configuration interface showing labels and notifications setup with AWS SNS integration
  1. Create a second alert for if Amazon EC2 CPU utilization is abnormal:
    1. For Alert name, enter EC2 CPU utilization too high.
    2. For Data source, choose Amazon CloudWatch.
    3. For Namespace, choose AWS/EC2.
    4. For Metric name, choose CPUUtilization
    5. For Statistic, choose Maximum.
      Amazon CloudWatch query interface for setting up EC2 CPU utilization alert conditions
    6. For Threshold, configure to alert if CPU utilization is more than 95% for 3 minutes.
      Amazon Managed Grafana alert interface with Reduce and Threshold expressions for alert condition management
    7. For Evaluation interval, configure to 1 minute.
      New evaluation group configuration modal showing CPU utilization monitoring setup with 1-minute interval
      AWS Managed Grafana alert rule configuration screen showing evaluation behavior settings
    8. For Contact point, choose SNS_QA.Amazon Managed Grafana alert configuration showing customizable labels, contact point selection for SNS_QA integration
  1. On the alert rule creation page, scroll to 5. Add annotations and for Summary, add a clear description of the alert, for example, CPU utilization on EC2 instance is too high.

Alert configuration summary field with "CPU utilization on EC2 instance is too high" warning message

Apache HBase region server incident test

To confirm the system is working as expected, complete the following Apache HBase region server incident test:

  1. SSH into an EMR core instance.
  2. Stop the Apache HBase region server using systemctl:
 # Stop HBase region server service 
 sudo systemctl stop hbase-regionserver.service

  1. Verify the service status:
 # Check the current state of HBase region server service 
 sudo systemctl status hbase-regionserver.service
  1. Observe Amazon Managed Grafana alert progression:
    1. Monitor alert status changes.
      Alert dashboard showing HBase region server alert status in pending state
      Alert dashboard showing HBase region server alert in firing state
    2. Verify SNS message generation.
    3. Confirm SQS message queuing.
    4. Track the Lambda function triggered for remediation.

Terminal output showing HBase RegionServer service status and daemon processes

HBase monitoring interface displaying region server status with health indicators and action buttons

CPU utilization stress test

Complete the following CPU utilization stress test:

  1. SSH into the EMR primary instance.
  2. Install stress testing tools:
 sudo amazon-linux-extras install epel -y
 sudo yum install stress -y

  1. Verify the installation:
 stress --version

  1. Generate high CPU load using the stress command and the following command structure:
 sudo stress [options]

For our Amazon EMR test, use the following command:

 # For m5.xlarge instances (4 vCPUs) sudo stress --cpu 4

-c 4 in the command creates 4 CPU-bound processes (one for each vCPU).The following are instance type vCPUs for your reference:

  • m5.xlarge: 4 vCPUs
  • m5.2xlarge: 8 vCPUs
  • m5.4xlarge: 16 vCPUs
  1. Monitor system response:
    1. Observe Amazon Managed Grafana alert status changes.
      Amazon Managed Grafana dashboard header showing rules status
    2. Verify Amazon Bedrock recommendation generation.
    3. Check SNS email notification delivery.
      AWS SNS notification email showing troubleshooting steps for high CPU usageCode snippet showing CPU usage troubleshooting steps in red text

Best practices and considerations

Monitoring infrastructure requires precise alert prioritization and threshold configuration. Alert aggregation techniques prevent notification overload by consolidating event streams and reducing redundant alerts. Operational teams must maintain dashboards through consistent updates and metric integration, providing real-time visibility into system performance and health.

Security implementations focus on least-privilege AWS Identity and Access Management (IAM) roles, restricting access to critical resources and minimizing potential breach vectors. Data protection strategies involve encryption protocols for information at rest and in transit, using AES-256 standards. Automated security audit processes scan automation scripts, identifying potential vulnerabilities through code analysis and runtime inspection.

Performance optimization in serverless architectures uses Lambda extensions to cache knowledge base content, reducing latency and improving response times. Retry mechanisms for API calls implement exponential backoff strategies, mitigating transient network exceptions and enhancing system resilience. Execution time monitoring of Lambda functions enables detection of anomalies through statistical analysis, providing insights into potential system-wide incidents or performance degradations.

Clean up

To avoid incurring future charges, delete the resources by deleting the parent stack on the AWS CloudFormation console.

Conclusion

This solution provides a robust framework for automated EMR cluster monitoring and incident response. By combining real-time monitoring with AI-powered remediation suggestions and automated execution, organizations can significantly reduce MTTR for common Amazon EMR issues while building a knowledge base for future incident response.

Try out this solution for your own use case, and leave your feedback in the comments section.

About the authors

Author Yu-ting Su, Sr. Hadoop System Engineer, AWS Support Engineering. Yu-Ting is a Sr. Hadoop Systems Engineer at Amazon Web Services (AWS). Her expertise is in Amazon EMR and Amazon OpenSearch Service. She’s passionate about distributing computation and helping people to bring their ideas to life.

AWS Weekly Roundup: Kiro, AWS Lambda remote debugging, Amazon ECS blue/green deployments, Amazon Bedrock AgentCore, and more (July 21, 2025)

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-kiro-aws-lambda-remote-debugging-amazon-ecs-blue-green-deployments-amazon-bedrock-agentcore-and-more-july-21-2025/

I’m writing this as I depart from Ho Chi Minh City back to Singapore. Just realized what a week it’s been, so let me rewind a bit. This week, I tried my first Corne keyboard, wrapped up rehearsals for AWS Summit Jakarta with speakers who are absolutely raising the bar, and visited Vietnam to participate as a technical keynote speaker in AWS Community Day Vietnam, an energetic gathering of hundreds of cloud practitioners and AWS enthusiasts who shared knowledge through multiple technical tracks and networking sessions.

What I presented was a keynote titled “Reinvent perspective as modern developers”, featuring serverless, containers, and how we can cut the learning curves and be more productive with Amazon Q Developer and Kiro. I got a chance to discuss with a couple of AWS Community Builders and community developers, who shared how Amazon Q Developer actually addressed their challenges on building applications, with several highlighting significant productivity improvements and smoother learning curves in their cloud development journeys.

As I head back to Singapore, I’m carrying with me not just memories of delicious cà phê sữa đá (iced milk coffee), but also fresh perspectives and inspirations from this vibrant community of cloud innovators.

Introducing Kiro
One of the highlights from last week was definitely Kiro, an AI IDE that helps you deliver from concept to production through a simplified developer experience for working with AI agents. Kiro goes beyond “vibe coding” with features like specs and hooks that help get prototypes into production systems with proper planning and clarity.

Join the waitlist to get notified when it becomes available.

Last week’s AWS Launches
In other news, last week we had AWS Summit in New York, where we released several services. Here are some launches that caught my attention:

Console to IDE Integration

ECS Blue-Green Deployments

AWS Free Tier Enhanced Benefits

  • Monitor and debug event-driven applications with new Amazon EventBridge logging — Amazon EventBridge now provides enhanced logging capabilities that offer comprehensive event lifecycle tracking with detailed information about successes, failures, and status codes. This new observability feature addresses microservices and event-driven architecture monitoring challenges by providing visibility into the complete event journey.

EventBridge Enhanced Logging

S3 Vectors Overview

  • Amazon EKS enables ultra-scale AI/ML workloads with support for 100k nodes per cluster — Amazon EKS now supports up to 100,000 worker nodes in a single cluster, enabling customers to scale up to 1.6 million AWS Trainium accelerators or 800K NVIDIA GPUs. This industry-leading scale empowers customers to train trillion-parameter models and advance AGI development while maintaining Kubernetes conformance and familiar developer experience.

EKS Ultra-Scale Performance Improvements

From AWS Builder Center
In case you missed it, we just launched AWS Builder Center and integrated community.aws. Here are my top picks from the posts:

Upcoming AWS events
Check your calendars and sign up for upcoming AWS and AWS Community events:

  • AWS re:Invent – Register now to get a head start on choosing your best learning path, booking travel and accommodations, and bringing your team to learn, connect, and have fun. If you’re an early-career professional, you can apply to the All Builders Welcome Grant program, which is designed to remove financial barriers and create diverse pathways into cloud technology.
  • AWS Builders Online Series – If you’re based in one of the Asia Pacific time zones, join and learn fundamental AWS concepts, architectural best practices, and hands-on demonstrations to help you build, migrate, and deploy your workloads on AWS.
  • AWS Summits — Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Taipei (July 29), Mexico City (August 6), and Jakarta (June 26–27).
  • AWS Community Days — Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Singapore (August 2), Australia (August 15), Adria (September 5), Baltic (September 10), and Aotearoa (September 18).

You can browse all upcoming AWS led in-person and virtual developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Donnie

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Join Builder ID: Get started with your AWS Builder journey at builder.aws.com

Monitor and debug event-driven applications with new Amazon EventBridge logging

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/monitor-and-debug-event-driven-applications-with-new-amazon-eventbridge-logging/

Starting today, you can use enhanced logging capability in Amazon EventBridge to monitor and debug your event-driven applications with comprehensive logs. These new enhancements help improve how you monitor and troubleshoot event flows.

Here’s how you can find this new capability on the Amazon EventBridge console:

The new observability capabilities address microservices and event-driven architecture monitoring challenges by providing comprehensive event lifecycle tracking. EventBridge now generates detailed log entries every time a matched event against rules is published, delivered to subscribers, or encounters failures and retries.

You gain visibility into the complete event journey with detailed information about successes, failures, and status codes that make identifying and diagnosing issues straightforward. What used to take hours of trial-and-error debugging now takes minutes with detailed event lifecycle tracking and built-in query tools.

Using Amazon EventBridge enhanced observability
Let me walk you through a demonstration that showcases the logging capability in Amazon EventBridge.

I can enable logging for an existing event bus or when creating a new custom event bus. First, I navigate to the EventBridge console and choose Event buses in the left navigation pane. In Custom event bus, I choose Create event bus.

I can see this new capability in the Logs section. I have three options to configure the Log destination: Amazon CloudWatch Logs, Amazon Data Firehose Stream, and Amazon Simple Storage Service (Amazon S3). If I want to stream my logs into a data lake, I can select Amazon Kinesis Data Firehose Stream. Logs are encrypted in transit with TLS and at rest if a customer-managed key (CMK) is provided for the event bus. CloudWatch Logs supports customer-managed keys, and Data Firehose offers server-side encryption for downstream destinations.

For this demo, I select CloudWatch logs and S3 logs.

I can also choose Log level, from Error, Info, or Trace. I choose Trace and select Include execution data because I need to review the payloads. You need to be mindful as logging payload data may contain sensitive information, and this setting applies to all log destinations you select. Then, I configure two destinations, one each for CloudWatch log group and S3 logs. Then I choose Create.

After logging is enabled, I can start publishing test events to observe the logging behavior.

For the first scenario, I’ve built an AWS Lambda function and configured this Lambda function as a target.

I navigate to my event bus to send a sample event by choosing Send events.

Here’s the payload that I use:

{
  "Source": "ecommerce.orders",
  "DetailType": "Order Placed",
  "Detail": {
    "orderId": "12345",
    "customerId": "cust-789",
    "amount": 99.99,
    "items": [
      {
        "productId": "prod-456",
        "quantity": 2,
        "price": 49.99
      }
    ]
  }
}

After I sent the sample event, I can see the logs are available in my S3 bucket.

I can also see the log entries appearing in the Amazon CloudWatch logs. The logs show the event lifecycle, from EVENT_RECEIPT to SUCCESS. Learn more about the complete event lifecycle on TBD:DOC_PAGE.

Now, let’s evaluate these logs. For brevity, I only include a few logs and have redacted them for readability. Here’s the log from when I triggered the event:

{
    "resource_arn": "arn:aws:events:us-east-1:123:event-bus/demo-logging",
    "message_timestamp_ms": 1751608776896,
    "event_bus_name": "demo-logging",
// REDACTED FOR BREVITY //
    "message_type": "EVENT_RECEIPT",
    "log_level": "TRACE",
    "details": {
        "caller_account_id": "123",
        "source_time_ms": 1751608775000,
        "source": "ecommerce.orders",
        "detail_type": "Order Placed",
        "resources": [],
        "event_detail": "REDACTED FOR BREVITY"
    }
}

Here’s the log when the event was successfully invoked:

{
    "resource_arn": "arn:aws:events:us-east-1:123:event-bus/demo-logging",
    "message_timestamp_ms": 1751608777091,
    "event_bus_name": "demo-logging",
// REDACTED FOR BREVITY //
    "message_type": "INVOCATION_SUCCESS",
    "log_level": "INFO",
    "details": {
// REDACTED FOR BREVITY //
        "total_attempts": 1,
        "final_invocation_status": "SUCCESS",
        "ingestion_to_start_latency_ms": 105,
        "ingestion_to_complete_latency_ms": 183,
        "ingestion_to_success_latency_ms": 183,
        "target_duration_ms": 53,
        "target_response_body": "<REDACTED FOR BREVITY>",
        "http_status_code": 202
    }
}

The additional log entries include rich metadata that makes troubleshooting straightforward. For example, on a successful event, I can see the latency timing from starting to completing the event, duration for the target to finish processing, and HTTP status code.

Debugging failures with complete event lifecycle tracking
The benefit of EventBridge logging becomes apparent when things go wrong. To test failure scenarios, I intentionally misconfigure a Lambda function’s permissions and change the rule to point to a different Lambda function without proper permissions.

The attempt failed with a permanent failure due to missing permissions. The log shows it’s a FIRST attempt that resulted in NO_PERMISSIONS status.

{
    "message_type": "INVOCATION_ATTEMPT_PERMANENT_FAILURE",
    "log_level": "ERROR",
    "details": {
        "rule_arn": "arn:aws:events:us-east-1:123:rule/demo-logging/demo-order-placed",
        "role_arn": "arn:aws:iam::123:role/service-role/Amazon_EventBridge_Invoke_Lambda_123",
        "target_arn": "arn:aws:lambda:us-east-1:123:function:demo-evb-fail",
        "attempt_type": "FIRST",
        "attempt_count": 1,
        "invocation_status": "NO_PERMISSIONS",
        "target_duration_ms": 25,
        "target_response_body": "{\"requestId\":\"a4bdfdc9-4806-4f3e-9961-31559cb2db62\",\"errorCode\":\"AccessDeniedException\",\"errorType\":\"Client\",\"errorMessage\":\"User: arn:aws:sts::123:assumed-role/Amazon_EventBridge_Invoke_Lambda_123/db4bff0a7e8539c4b12579ae111a3b0b is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:123:function:demo-evb-fail because no identity-based policy allows the lambda:InvokeFunction action\",\"statusCode\":403}",
        "http_status_code": 403
    }
}

The final log entry summarizes the complete failure with timing metrics and the exact error message.

{
    "message_type": "INVOCATION_FAILURE",
    "log_level": "ERROR",
    "details": {
        "rule_arn": "arn:aws:events:us-east-1:123:rule/demo-logging/demo-order-placed",
        "role_arn": "arn:aws:iam::123:role/service-role/Amazon_EventBridge_Invoke_Lambda_123",
        "target_arn": "arn:aws:lambda:us-east-1:123:function:demo-evb-fail",
        "total_attempts": 1,
        "final_invocation_status": "NO_PERMISSIONS",
        "ingestion_to_start_latency_ms": 62,
        "ingestion_to_complete_latency_ms": 114,
        "target_duration_ms": 25,
        "http_status_code": 403
    },
    "error": {
        "http_status_code": 403,
        "error_message": "User: arn:aws:sts::123:assumed-role/Amazon_EventBridge_Invoke_Lambda_123/db4bff0a7e8539c4b12579ae111a3b0b is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:123:function:demo-evb-fail because no identity-based policy allows the lambda:InvokeFunction action",
        "aws_service": "AWSLambda",
        "request_id": "a4bdfdc9-4806-4f3e-9961-31559cb2db62"
    }
}

The logs provide detailed performance metrics that help identify bottlenecks. The ingestion_to_start_latency_ms: 62 shows the time from event ingestion to starting invocation, while ingestion_to_complete_latency_ms: 114 represents the total time from ingestion to completion. Additionally, target_duration_ms: 25 indicates how long the target service took to respond, helping distinguish between EventBridge processing time and target service performance.

The error message clearly states what failed, lambda:InvokeFunction action, why it failed, (no identity-based policy allows the action), which role was involved (Amazon_EventBridge_Invoke_Lambda_1428392416), and which specific resource was affected, which was indicated by the Lambda function Amazon Resource Name (ARN).

Debugging API Destinations with EventBridge Logging
One particular use case that I think EventBridge logging capability will be helpful is to debug issues with API destinations. EventBridge API destinations are HTTPS endpoints that you can invoke as the target of an event bus rule or pipe. HTTPS endpoints help you to route events from your event bus to external systems, software-as-a-service (SaaS) applications, or third-party APIs using HTTPS calls. They use connections to handle authentication and credentials, making it easy to integrate your event-driven architecture with any HTTPS-based service. 

API destinations are commonly used to send events to external HTTPS endpoints and debugging failures from the external endpoint can be a challenge. These problems typically stem from changes to the endpoint authentication requirements or modified credentials.

To demonstrate this debugging capability, I intentionally configured an API destination with incorrect credentials in the connection resource.

When I send an event to this misconfigured endpoint, the enhanced logging shows the root cause of this failure.

{
    "resource_arn": "arn:aws:events:us-east-1:123:event-bus/demo-logging",
    "message_timestamp_ms": 1750344097251,
    "event_bus_name": "demo-logging",
    //REDACTED FOR BREVITY//,
    "message_type": "INVOCATION_FAILURE",
    "log_level": "ERROR",
    "details": {
        //REDACTED FOR BREVITY//,
        "total_attempts": 1,
        "final_invocation_status": "SDK_CLIENT_ERROR",
        "ingestion_to_start_latency_ms": 135,
        "ingestion_to_complete_latency_ms": 549,
        "target_duration_ms": 327,
        "target_response_body": "",
        "http_status_code": 400
    },
    "error": {
        "http_status_code": 400,
        "error_message": "Unable to invoke ApiDestination endpoint: The request failed because the credentials included for the connection are not authorized for the API destination."
    }
}

The log provides immediate clarity about the failure. The target_arn shows this involves an API destination, the final_invocation_status indicates SDK_CLIENT_ERROR, and the http_status_code of 400 , which points to a client-side issue. Most importantly, the error_message explicitly states that: Unable to invoke ApiDestination endpoint: The request failed because the credentials included for the connection are not authorized for the API destination.

This complete log sequence provides useful debugging insights because I can see exactly how the event moved through EventBridge — from event receipt, to ingestion, to rule matching, to invocation attempts. This level of detail eliminates guesswork and points directly to the root cause of the issue.

Additional things to know
Here are a couple of things to note:

  • Architecture support – Logging works with all EventBridge features including custom event buses, partner event sources, and API destinations for HTTPS endpoints.
  • Performance impact – Logging operates asynchronously with no measurable impact on event processing latency or throughput.
  • Pricing – You pay standard Amazon S3, Amazon CloudWatch Logs or Amazon Data Firehose pricing for log storage and delivery. EventBridge logging itself incurs no additional charges. For details, visit the Amazon EventBridge pricing page .
  • Availability – Amazon EventBridge logging capability is available in all AWS Regions where EventBridge is supported.
  • Documentation — For more details, refer to the Amazon EventBridge monitoring and debugging Documentation.

Get started with Amazon EventBridge logging capability by visiting the EventBridge console and enabling logging on your event buses.

Happy building!
— Donnie