Tag Archives: AWS Lambda

Build high-performance apps with AWS Lambda Managed Instances

Post Syndicated from Debasis Rath original https://aws.amazon.com/blogs/compute/build-high-performance-apps-with-aws-lambda-managed-instances/

High-performance applications such as CPU-intensive processing, memory-heavy analytics, and steady-state data pipelines often require more predictable compute resources than standard AWS Lambda configurations provide. AWS Lambda Managed Instances (LMI) addresses this by letting you run Lambda functions on selected Amazon EC2 instance types while preserving the Lambda programming model. You can choose over 400 Amazon Elastic Compute Cloud (Amazon EC2) instance types from general purpose, compute optimized, or memory optimized instance families to match workload requirements. AWS Lambda continues to manage infrastructure operations such as instance lifecycle management, operating system patching, runtime updates, request routing, and automatic scaling. This approach gives your teams greater control over compute characteristics, EC2 pricing model and reduces operational overhead of managing servers or clusters.

In this post, you will learn how to configure AWS Lambda Managed Instances by creating a Capacity Provider that defines your compute infrastructure, associating your Lambda function with that provider, and publishing a function version to provision the execution environments. We will conclude with production best practices including scaling strategies, thread safety, and observability for reliable performance.

Figure 1. Creating Function on LMI

Figure 1. Creating Function on LMI

Creating Capacity Providers

A Capacity Provider defines the infrastructure blueprint for running LMI functions on Amazon EC2. It specifies instance types, network placement, and scaling behavior. To create a Capacity Provider, you need two parameters: an IAM role (Capacity Provider Operator Role) granting Lambda permissions to launch and manage instances and your VPC configuration with subnets and security groups. Create this role in your account with the AWSLambdaManagedEC2ResourceOperator managed policy following the Principle of Least Privilege (granting only the minimum permissions necessary).

This command creates a Capacity Provider with instance types and scaling configuration:

aws lambda create-capacity-provider \
  --capacity-provider-name my-lmi-capacity \
  --vpc-config SubnetIds=subnet-abc123,subnet-def456,SecurityGroupIds=sg-xyz789 \
  --permissions-config CapacityProviderOperatorRoleArn=arn:aws:iam::123456789012:role/LMIOperatorRole \
  --instance-requirements Architectures=x86_64,AllowedInstanceTypes=c5.2xlarge,r5.4xlarge \
  --capacity-provider-scaling-config MaxVCpuCount=50,ScalingMode=Auto \
  --region us-east-1

This command returns a Capacity Provider ARN that you’ll use to create your LMI function. Your functions behavior depends on four main configurations in the capacity provider:

Instance selection

Lambda currently supports three Amazon EC2 instance families (.large and up): C (compute optimized) for CPU-heavy work, M (general purpose) for balanced workloads, and R (memory optimized) for large datasets. Choose x86 (Intel/AMD) or ARM (Graviton) architectures. If you don’t specify instance types, Lambda defaults to appropriate instances based on your function’s memory and CPU configuration. This is the recommended starting point unless you have specific performance requirements. When you need more control, use AllowedInstanceTypes to specify only the instance types that Lambda can use or use ExcludedInstanceTypes to exclude specific types while allowing all other instance types. You can’t use both parameters together.

VPC and networking

Configure multiple subnets across Availability Zones. Lambda creates a minimum Amazon EC2 fleet of three instances distributed across your configured Availability Zones to maintain availability and resiliency. Egress traffic from functions, including Amazon CloudWatch Logs, transits through the Amazon EC2 instance’s network interface in your Amazon Virtual Private Cloud (Amazon VPC). As functions send logs and metrics to CloudWatch, you will need internet access through a NAT Gateway or VPC endpoints with AWS PrivateLink for Amazon CloudWatch. This only affects egress traffic; function invoke requests don’t flow through your VPC. Security groups attached to your instances should allow only the traffic your function code needs. With LMI, configure VPC once at the Capacity Provider level instead of per function, simplifying management for multiple LMI functions. Standard Lambda functions continue to use their own VPC configurations. This Capacity Provider VPC configuration applies only to LMI functions.

Figure 2. LMI Networking

Figure 2. LMI Networking

Scaling configuration

Set MaxVCpuCount to cap compute capacity and control costs. New invocations throttle when you reach this limit until capacity frees up. Lambda monitors CPU utilization and scales instances automatically. Choose automatic scaling mode where Lambda tunes thresholds based on load patterns, or manual mode where you set a target CPU utilization percentage. Multiple functions can share the same Capacity Provider to reduce costs through better resource utilization, though you might want separate providers for functions with different performance or isolation requirements.

Security

Lambda encrypts Amazon Elastic Block Store (Amazon EBS) volumes attached to EC2 instances with a service-managed key by default. You can provide your own AWS Key Management Service (AWS KMS) key for encryption. Place instances in private subnets with restrictive security groups for enhanced security.

Creating Lambda Managed Instance Functions

You create an LMI function similarly to creating a standard Lambda function. You package your code, set your runtime, assign an execution role, and configure memory. The difference is specifying a CapacityProviderConfig to tell Lambda which Capacity Provider to use and how to size each execution environment. Specify CapacityProviderConfig during function creation with the Capacity Provider ARN and configure two execution environment settings. ExecutionEnvironmentMemoryGiBPerVCpu sets the memory-to-vCPU ratio (2:1, 4:1, or 8:1) based on your workload type and PerExecutionEnvironmentMaxConcurrency defines how many concurrent requests share each execution environment. This table shows how memory and vCPU allocation maps across supported execution environment ratio.

2:1 Ratio(Compute optimized) 4:1 Ratio(General purpose) 8:1 Ratio(Memory optimized)
Memory (GB) vCPU(s) Memory (GB) vCPU(s) Memory (GB) vCPU(s)
2 1 4 1 8 1
4 2 8 2 16 2
6 3 12 3 24 3
8 4 16 4 32 4
10 5 20 5
12 6 24 6
14 7 28 7
16 8 32 8
… …
32 16

Function Memory-to-CPU configuration

Set the function’s memory size (up to 32 GB for LMI) and ExecutionEnvironmentMemoryGiBPerVCpu ratio. The default ratio is 2:1. A 2:1 ratio map to compute optimized instances for CPU-intensive tasks like video encoding, 4:1 map to a general purpose for balanced workloads, and 8:1 maps to a memory optimized instances for large in-memory datasets or caching. You must set memory in multiples of the ratio. LMI requires a 2 GB minimum as execution environments need sufficient memory to handle multiple concurrent requests. LMI supports up to 32 GB memory per execution environment.

Multi-Concurrency settings

LMI supports multiple concurrent invocations sharing the same execution environment, reducing cost per invocation by maximizing vCPU utilization. This is particularly effective for I/O-bound workloads, where invocations waiting on database queries or API calls yield vCPU usage to other invocations during idle periods. Lambda defaults to max concurrency per execution environment based on your runtime: Node.js (64 per vCPU), Java, and .NET (32 per vCPU), Python (16 per vCPU). Use PerExecutionEnvironmentMaxConcurrency to set a lower limit based on your workload’s resource needs. Decrease it if you’re experiencing memory pressure or CPU contention. When environments reach their configured max concurrency, new invocations throttle until capacity frees up at the execution environment level. This table captures the maximum concurrency per vCPU for each supported programming language.

Language Default Max Concurrency
Node.js 64 per vCPU
Java 32 per vCPU
.NET 32 per vCPU
Python 16 per vCPU

This command creates a Lambda function and associates it with your Capacity Provider:

aws lambda create-function \
  --function-name my-lmi-function \
  --runtime python3.13 \
  --role arn:aws:iam::123456789012:role/LambdaExecutionRole \
  --handler app.lambda_handler \
  --zip-file fileb://function.zip \
  --memory-size 4096 \
  --capacity-provider-config '{
    "LambdaManagedInstancesCapacityProviderConfig": {
      "CapacityProviderArn": "arn:aws:lambda:us-east-1:123456789012:capacity-provider:my-lmi-capacity",
      "ExecutionEnvironmentMemoryGiBPerVCpu": 4.0,
      "PerExecutionEnvironmentMaxConcurrency": 10
    }
  }' \
  --region us-east-1

Publishing Lambda Managed Instance Functions

Important: publish a function version before invoking an LMI function. Publishing triggers Lambda to provision Amazon EC2 instances and initialize execution environments, so that the configured baseline capacity is ready before you start invoking. Expect a brief delay before your code goes live as Lambda provisions and launches Amazon EC2 instances. With LMI, execution environments pre-warm after publishing and remain invoke-ready, without cold starts for published versions. Standard Lambda environments initialize on first invoke (cold starts).

This command publishes a Lambda function version and provisions capacity:

aws lambda publish-version --function-name my-lmi-function \
--region us-east-1

After publishing, the function works with standard invocation methods including direct invokes, event source mappings, and service integrations with Amazon API Gateway, Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB Streams, and Amazon EventBridge.

Figure 3. LMI Invocation from event sources

Figure 3. LMI Invocation from event sources

Scaling LMI Functions

Lambda monitors CPU utilization at Capacity Provider level. When CPU utilization reaches the target threshold, Lambda automatically provisions additional EC2 instances, and creates more execution environments on those instances, up to the MaxVCpuCount limit you configured for your capacity provider. As demand decreases, Lambda consolidates workloads onto fewer EC2 instances. You can choose automatic scaling mode (Lambda adjusts thresholds based on your patterns) or manual mode (you set a target CPU percentage). Automatic mode works for variable traffic patterns or when getting started. Manual mode fits when you have predictable patterns and want precise control over scaling thresholds for cost optimization.

Min and max execution environments

Control scaling at the function level with min and max execution environments. The default minimum is 3 execution environments to maintain high availability across Availability Zones. Your total function concurrency equals the number of execution environments multiplied by PerExecutionEnvironmentMaxConcurrency. For example, with min set to 3 and PerExecutionEnvironmentMaxConcurrency of 10, you have provided capacity for 30 concurrent invocations. With max set to 20, you can scale up to 200 concurrent invocations with incoming traffic, based on CPU utilization or concurrency saturation per execution environment. Set max to cap total concurrency and prevent noisy neighbor issues when multiple functions share a Capacity Provider. LMI maintains a minimum number of execution environments with a minimum Amazon EC2 fleet, while standard Lambda scales to zero when idle. Set both min and max to 0 to deactivate a function without deleting it.

Figure 4. LMI Scaling

Figure 4. LMI Scaling

This command updates the minimum and maximum execution environments for your function:

aws lambda put-function-scaling-config \
  --function-name my-lmi-function \
  --qualifier $LATEST \
  --function-scaling-config MinExecutionEnvironments=5,MaxExecutionEnvironments=20 \
  --region us-east-1

We’ll cover scaling patterns and throughput optimization strategies in depth in a separate blog post.

Best Practices and Production Considerations

Thread Safety

Since LMI supports multiple invocations sharing execution environments, your code must be thread-safe. Code that isn’t thread-safe causes data corruption, security issues, or unpredictable behavior under concurrent load.

Thread safety essentials

Avoid mutating shared objects or global variables. Use thread-local storage for request-specific data. Initialize shared clients (AWS SDK, database connections) outside the function handler and verify that configurations remain immutable during invocations. Write to /tmp using request-specific file names to prevent concurrent writes.

Runtime-specific guidance

Java applications should use immutable objects, thread-safe collections, and proper synchronization. Node.js applications should use async context for request isolation. Python applications run separate processes per execution environment. So, focus on interprocess coordination and file locking for /tmp access.

Workload Optimization

I/O-bound workloads perform better with higher concurrency per environment. Use asynchronous patterns and non-blocking I/O to maximize efficiency. CPU-bound workloads get no benefit from concurrency greater than one per vCPU. Instead, configure more vCPUs per function for true parallelism for compute-heavy tasks like data transformation or image processing.

Testing

Validate your code under concurrent execution. Test with multiple simultaneous invocations to detect race conditions and shared state issues before production deployment. You can use LocalStack for local emulation of LMI. Learn more about LocalStack’s LMI support in their announcement blog.

Compatibility

Tools like Powertools for AWS work with LMI without code changes. However, if you’re reusing existing Lambda function code, layers, or packaged dependencies on LMI, test for thread safety and compatibility with the multi-concurrent execution model before production deployment.

Observability

LMI automatically publishes CloudWatch metrics at two levels: capacity provider (CPU, memory, network, and disk utilization across your Amazon EC2 fleet) and execution environment (concurrency, CPU, and memory per function). Monitor CPUUtilization to understand scaling headroom and right-size your MaxVCpuCount. Track ExecutionEnvironmentConcurrency against ExecutionEnvironmentConcurrencyLimit to catch throttling before it impacts users. Lambda publishes metrics at 5-minute intervals. Use CloudWatch alarms to stay ahead of capacity limits in production.

Conclusion

AWS Lambda Managed Instances combines serverless simplicity with compute flexibility, helping you run high-performance workloads with reduced operational complexity. You maintain the familiar programming model of Lambda while accessing the diverse instance types of Amazon EC2 and predictable pricing, making it well-suited for data processing pipelines, compute intensive operations and cost-sensitive steady-state applications.

Ready to get started with LMI? Deploy our Monte Carlo risk simulation example from GitHub to see LMI in action with a real compute-intensive workload. The sample includes complete infrastructure code and walks you through capacity provider configuration, function setup, and performance optimization.

We want to hear from you. Share your feedback, questions, and use cases on re:Post.

Best practices for Lambda durable functions using a fraud detection example

Post Syndicated from Debasis Rath original https://aws.amazon.com/blogs/compute/best-practices-for-lambda-durable-functions-using-a-fraud-detection-example/

AWS Lambda durable functions extend the Lambda programming model to build fault-tolerant multi-step applications and AI workflows using familiar programming languages. They preserve progress despite interruptions and execution can suspend for up to one year, for human approvals, scheduled delays, or other external events, without incurring compute charges for on-demand functions.

This post walks through a fraud detection system built with durable functions. It also highlights the best practices that you can apply to your own production workflows, from approval processes to data pipelines to AI agent orchestration. You will learn how to handle concurrent notifications, wait for customer responses, and recover from failures without losing progress. If you are new to durable functions, check out the Introduction to Durable Functions blog post first.

Fraud detection with human-in-the-loop

Consider a credit card fraud detection system, which uses an AI agent to analyze incoming transactions and assign risk scores. For ambiguous cases (medium-risk scores), the system needs human approval before authorizing a transaction. The workflow branches based on risk:

  • Low risk (score < 3): Authorize immediately
  • High risk (score ≥ 5): Send to the fraud department immediately
  • Medium risk (score 3–4): Suspend transaction, send SMS and email to cardholder, wait up to 24 hours for confirmation (wait time is customizable)
Figure 1. Agentic Fraud Detection with durable Lambda functions

Figure 1. Agentic Fraud Detection with durable Lambda functions

With human-in-the-loop workflows, response times can vary from minutes to hours. These delays introduce the need to durably preserve the state without consuming compute resources while waiting. With financial systems, we must also implement idempotency to guard against duplicate messages (invocations) and recover from failures without reprocessing completed work. To address these requirements, developers implement polling patterns with external state stores like Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3) to manage idempotency, pay for idle compute while waiting for callbacks, introduce external orchestration components, or build asynchronous message-driven systems to handle long-processing tasks.

Lambda durable functions provide a new alternative to address these challenges through durable execution, a pattern that uses checkpoints (saved state snapshots) to preserve progress and replays from saved state to recover from failures or resume after waiting. With checkpointing capabilities, you no longer need to pay Lambda compute charges while waiting, whether for callbacks, scheduled delays, or external events. Learn how to implement durable functions using the complete fraud detection implementation at this GitHub repository. You can deploy it to your AWS account and experiment with the code as you read. The repository includes deployment instructions, sample data, and helper functions for testing.

As we walk through the code, we’ll focus on best practices for designing workflows with durable execution and how to apply these patterns correctly in production workflows.

Design steps to be idempotent

Durable execution is designed to preserve progress through checkpoints and replay, but that reliability model means step logic can execute more than once. When steps retry, how do you prevent duplicate actions like charges to the credit card or repeated customer SMS or email notifications?

Durable functions use at-least-once execution by default, executing each step at least one time, potentially more if failures occur. When a step fails, it retries. There are two strategies to design idempotent steps that prevent duplicate side effects: using external API idempotency keys and using the at-most-once step semantics built into durable functions.

Strategy A: External API Idempotency Keys

// Strategy A: Use external API idempotency keys
await context.step(`authorize-${tx.id}`, async () => {
  return payment.charges.create({
    amount: tx.amount,
    currency: 'usd',
    idempotency_key: `tx-${tx.id}`, // Prevents duplicate charges
    description: `Transaction ${tx.id}`
  });
});

Notice the configuration:

  • idempotency_key in API call: If the step retries, the payment processor recognizes it’s a duplicate request and returns the original result
  • Defense in depth: Two layers of protection: Lambda checkpointing and external API idempotency

Each layer provides independent protection. If Lambda’s checkpoint fails, the external API prevents duplicate charges. For legacy systems without idempotency support, where it’s critical that an operation is not executed more than once, use at-most-once semantics:

Strategy B: Use At-Most-Once Semantics

For legacy systems without idempotency support, use at-most-once execution, a delivery feature that executes each step zero or one time, never more:

// Strategy B: At-most-once step semantics
await context.step("charge-legacy-system", async () => {
  return await legacyPaymentSystem.charge(tx.amount);
}, {
  semantics: StepSemantics.AtMostOncePerRetry,
  retryStrategy: createRetryStrategy({ maxAttempts: 0 })
});

This checkpoints before step execution, preventing the step from re-execution on retries. The tradeoff? If the step fails, you must decide whether to retry (risking duplicates) or fail the entire workflow.

Use idempotency for critical side effects like payment processing, database writes, external API calls, state transitions, and resource provisioning. Read more about idempotency here.

Prevent duplicate executions with DurableExecutionName

Idempotent steps prevent duplicate side effects within a single execution, but what about duplicate workflow executions running concurrently? For example, duplicate messages in the queue, users clicking “Submit” multiple times in the UI, or the same event arriving via multiple channels like webhook and API. Without protection, each invocation creates a separate durable execution, potentially running the fraud check multiple times, sending duplicate notifications, and creating confusion about which execution is authoritative. Durable functions provide DurableExecutionName to help ensure only one concurrent execution per unique name.

// Invoke fraud detection function with execution name
await lambda.invoke({
  FunctionName: 'fraud-detection',
  InvocationType: 'Event',
  DurableExecutionName: `tx-${transactionId}`,
  Payload: JSON.stringify({
    id: transactionId,
    amount: 6500,
    location: 'New York, NY',
    vendor: 'Amazon.com'
  })
});

Notice the configuration:

  • DurableExecutionName: tx-${transactionId}: Uses the transaction ID as a unique execution identifier
  • InvocationType: ‘Event’: Asynchronous invocation supports long-running workflows beyond 15 minutes
  • One execution per transaction: If three invocations arrive with the same transaction ID, only the first creates an execution. Subsequent requests with the same execution name and payload receive an idempotent response returning the existing execution’s ARN, rather than creating a new execution.

Lambda durable functions work with Lambda event sources, including event source mappings (ESM) such as Amazon Simple Queue Service (Amazon SQS), Amazon Kinesis, and DynamoDB Streams. ESMs invoke durable functions synchronously and inherit Lambda’s 15-minute invocation limit. Therefore, like direct Request/Response invocations, durable functions executions using event source mappings cannot exceed 15 minutes.

For workflows exceeding 15 minutes, use an intermediary Lambda function between the event source mapping and durable function:

// Intermediary function for SQS -> Durable function
export const handler = async (event) => {
  for (const record of event.Records) {
    const transaction = JSON.parse(record.body);
    await lambda.invoke({
      FunctionName: process.env.FRAUD_DETECTION_FUNCTION,
      InvocationType: 'Event',
      DurableExecutionName: `tx-${transaction.id}`,
      Payload: JSON.stringify(transaction)
    });
  }
};

This removes the 15-minute limit, allows executions up to one year, and enables custom execution name parameters for idempotency. Use Powertools for AWS Lambda to prevent duplicate invocations of the durable function when the event source mapping retries the intermediary function. Additionally, configure failure handling for your event source to capture failed invocations for future redrive or replay. For example, dead letter queues for SQS, or on-failure destinations for other event sources.

Match timeouts to invocation type

One important configuration detail ties these patterns together: matching your timeout settings to your invocation type. Lambda synchronous invocations (RequestResponse) have a hard 15-minute timeout limit. If you configure a durable execution to run for 24 hours but invoke it synchronously, the synchronous invocation fails immediately with an exception. Durable functions support workflows up to one year when invoked asynchronously.

// Lambda function configuration
{
  FunctionName: 'fraud-detection',
  Timeout: 300,
  MemorySize: 512,
  DurableConfig: {
    ExecutionTimeout: 90000
  }
}

And invoke asynchronously:

// Async invocation for long-running workflow
await lambda.invoke({
  FunctionName: 'fraud-detection',
  InvocationType: 'Event',
  DurableExecutionName: `tx-${transactionId}`,
  Payload: JSON.stringify(transaction)
});

Notice the configuration:

  • Timeout: 300: Lambda function timeout (5 minutes in this example, up to a maximum of 15 minutes). This defines the maximum duration for each active execution phase, including the initial invocation and any subsequent replays. Set this to cover the longest expected active processing time in your workflow.
  • ExecutionTimeout: { hours: 25 }: Durable execution timeout covers the workflow’s expected total duration including suspension periods. Set this slightly above the longest wait timeout to avoid edge cases.
  • InvocationType: ‘Event’: Asynchronous invocation removes the 15-minute limit and enables executions up to one year.

The Lambda function timeout applies to active execution phases (AI calls, notification sending). During suspension (waiting for callbacks), the function isn’t running, so this timeout doesn’t apply. Setting the durable execution timeout to a meaningful boundary prevents workflows from running longer than expected. Without an explicit timeout, executions can run up to the maximum lifetime of one year.

Synchronous (RequestResponse) Asynchronous (Event)
Total duration Under 15 minutes Up to 1 year
Caller needs result Yes No
Idempotency support Yes Yes
Waits with suspension Yes Yes

Execute Concurrent Operations with context.parallel()

In the fraud detection workflow, the system notifies the cardholder through multiple channels such as SMS and email. Preserving business logic when executing parallel workflows introduces code complexities such as managing execution state across branches, handling synchronization, and coordinating branch completion. Durable functions simplify parallel workflow implementation using context.parallel(), which executes branches concurrently while maintaining durable checkpoints for each branch and provides configurable options to handle partial completions. By checkpointing and managing the state internally, durable functions help make sure that the state is preserved even if there are retries or failures. Note that context.parallel() manages the internal execution state for each branch. If your branches interact with a shared external state (such as a database), you’re responsible for managing concurrent access to that external state.

// Human-in-the-loop: verify via email AND SMS (first response wins)
let verified = await context.parallel("human-verification", [
  (ctx) => ctx.waitForCallback("SendVerificationEmail",
    async (callbackId) => sendCustomerNotification(callbackId, 'email', tx)
  ),
  (ctx) => ctx.waitForCallback("SendVerificationSMS",
    async (callbackId) => sendCustomerNotification(callbackId, 'sms', tx)
  )
], {
  maxConcurrency: 2,
  completionConfig: {
    minSuccessful: 1 // Continue after 1 success
  }
});

Notice the configuration:

  • maxConcurrency: 2: Both notifications sent at the same time
  • minSuccessful: 1: We only need one channel to succeed, whichever responds first wins

Each parallel branch waits for its callback independently, and the durable execution checkpoints each branch as part of the execution state. Using the minSuccessful parameter, you control the minimum number of successful branch executions required for the parallel operation to complete. In this example, only one of the two branches needs to succeed. Verifications through SMS or email are both valid, and the workflow resumes as soon as either channel completes successfully. We call this the first-response-wins pattern. This pattern works well when you only need a single successful result from any parallel branch and want the remaining branches to stop blocking progress.

But what happens if neither channel responds? Without timeouts, this workflow could remain suspended for up to the configured execution lifetime.

Always configure callback timeouts

Let’s add timeout protection to the parallel verification from the previous section. context.waitForCallback() accepts a timeout option that bounds how long each branch waits before throwing an exception. By wrapping the parallel call in a try/catch, you can implement fallback logic when users don’t respond in time.

// Enhanced: parallel verification with timeout and error handling
let verified;
try {
  verified = await context.parallel("human-verification", [
    (ctx) => ctx.waitForCallback("SendVerificationEmail",
      async (callbackId) => sendCustomerNotification(callbackId, 'email', tx),
      { timeout: { days: 1 } }  // Wait up to 1 day for email response
    ),
    (ctx) => ctx.waitForCallback("SendVerificationSMS",
      async (callbackId) => sendCustomerNotification(callbackId, 'sms', tx),
      { timeout: { days: 1 } }  // Wait up to 1 day for SMS response
    )
  ], {
    maxConcurrency: 2,
    completionConfig: {
      minSuccessful: 1
    }
  });
} catch (error) {
  const isTimeout = error.message?.includes("timeout");
  if (isTimeout) {
    context.logger.warn("Customer verification timeout", { error, txId: tx.id });
    // Fallback: escalate to fraud department
    return await context.step("sendToFraudDepartment", async () =>
      sendToFraudDepartment(tx, true)
    );
  }
  throw error; // Re-throw non-timeout errors
}

Notice what changed from the previous section:

  • timeout: { days: 1 }: Each callback branch now has a maximum wait time of 1 day. If neither the email nor SMS callback arrives within that window, a timeout exception is thrown.
  • try/catch with timeout detection: The catch block distinguishes between timeout errors and other exceptions. When a timeout occurs, the workflow implements fallback logic by escalating the transaction to the fraud department, while non-timeout errors are re-thrown to be handled by the durable execution retry mechanism.

Without this error handling, the entire execution fails unhandled. The timeout also works with the minSuccessful configuration: if one branch times out but the other succeeds, the parallel operation still completes successfully since only one successful result is required.

For advanced use cases where the callback handler performs long-running work, you can also configure a heartbeatTimeout to detect stalled callbacks before the main timeout expires. See the Lambda Developer Guide for details.

Use callback timeouts for human approvals, external API callbacks, asynchronous processing, and third-party integrations.

Putting it all together: complete fraud detection implementation

Now let’s see how all the best practices work together in the complete fraud detection workflow:

import { withDurableExecution } from "@aws/durable-execution-sdk-js";
import { BedrockAgentCoreClient, InvokeAgentRuntimeCommand } from "@aws-sdk/client-bedrock-agentcore";

const agentRuntimeArn = process.env.AGENT_RUNTIME_ARN;
const agentRegion = process.env.AGENT_REGION || 'us-east-1';
const client = new BedrockAgentCoreClient({ region: agentRegion });

export const handler = withDurableExecution(async (event, context) => {
  const tx = {
    id: event.id,
    amount: event.amount,
    location: event.location,
    vendor: event.vendor
  };

  // AI fraud assessment with error handling
  tx.score = await context.step("fraudCheck", async () => {
    try {
      const payloadJson = JSON.stringify({ input: { amount: tx.amount } });
      const command = new InvokeAgentRuntimeCommand({
        agentRuntimeArn: agentRuntimeArn,
        qualifier: 'DEFAULT',
        payload: Buffer.from(payloadJson, 'utf-8'),
        contentType: 'application/json',
        accept: 'application/json'
      });
      const response = await client.send(command);
      const responseText = await response.response.transformToString();
      const result = JSON.parse(responseText);
      return result?.output?.risk_score ?? 5;  // Default to high-risk if score unavailable
    } catch (error) {
      context.logger.error("Fraud check failed", { error, txId: tx.id });
      return 5;
    }
  });

  // Route based on AI decision
  if (tx.score < 3) {
    // Best Practice: Idempotent authorization
    return await context.step(`authorize-${tx.id}`, async () =>
    authorizeTransaction(tx, { idempotency_key: `tx-${tx.id}` })
    );
  }

  if (tx.score >= 5) {
    return await context.step(`sendToFraudDepartment-${tx.id}`, async () =>
      sendToFraudDepartment(tx)
    );
  }

  // Medium risk: need human verification
  await context.step(`suspend-${tx.id}`, async () => suspendTransaction(tx));

  // Best Practice: Concurrent operations with timeout configuration
  let verified;
  try {
    verified = await context.parallel("human-verification", [
      (ctx) => ctx.waitForCallback("SendVerificationEmail",
        async (callbackId) => sendCustomerNotification(callbackId, 'email', tx),
        { timeout: { days: 1 } }
      ),
      (ctx) => ctx.waitForCallback("SendVerificationSMS",
        async (callbackId) => sendCustomerNotification(callbackId, 'sms', tx),
        { timeout: { days: 1 } }
      )
    ], {
      maxConcurrency: 2,
      completionConfig: {
        minSuccessful: 1
      }
    });
  } catch (error) {
    const isTimeout = error.message?.includes("timeout");
    context.logger.warn(
      isTimeout ? "Customer verification timeout" : "Customer verification failed",
      { error, txId: tx.id }
    );
    return await context.step(`timeout-escalate-${tx.id}`, async () =>
      sendToFraudDepartment(tx, true)
    );
  }

  // Idempotent final step with idempotency key
  return await context.step(`finalize-${tx.id}`, async () => {
    const action = !verified.hasFailure && verified.successCount > 0
      ? "authorize"
      : "escalate";
    if (action === "authorize") {
      return authorizeTransaction(tx, true, { idempotency_key: `finalize-${tx.id}` });
    }
    return sendToFraudDepartment(tx, true);
  });
});

Notice how the best practices work together: context.parallel() sends SMS and email concurrently, resuming when either channel responds. Both callbacks configure 1-day timeouts with try/catch handling that escalates on timeout. The DurableExecutionName: tx-${transactionId} parameter (specified at invocation time, shown in the following CLI example) provides execution-level deduplication, while idempotency keys in the authorization steps prevent duplicate charges at the application layer. Asynchronous invocation (InvocationType: 'Event') enables the 24-hour wait period.

Once deployed, invoke the function asynchronously with a sample transaction to see it in action:

transactionId="123456789"
aws lambda invoke \
  --function-name "fraud-detection:$LATEST" \
  --invocation-type Event \
  --durable-execution-name "tx-${transactionId}" \
  --cli-binary-format raw-in-base64-out \
  --payload "{\"id\": \"${transactionId} \", \"amount\": 6500, \"location\": \"New York, NY\", \"vendor\": \"Amazon.com\"}" \
  --region us-east-2 \
  response.json

Upon successful invocation, you can view the execution state in the Lambda console’s durable operations view. The execution shows a suspended state, waiting for customer response:

Figure 2: Suspended execution state

Figure 2: Suspended execution state

Notice the fraudCheck and suspendTransaction steps show as succeeded with checkpointed results. The human-verification parallel operation shows that both SMS and email branches started. The timeline shows the function in a suspended state. Simulate a customer response by sending a callback success through the console, AWS Command Line Interface (AWS CLI) or Lambda API:

aws durable-lambda send-durable-execution-callback-success \
  --callback-id <CALLBACK_ID_FROM_EMAIL_OR_SMS> \
  --result '{"status":"approved","channel":"email"}' \
  --cli-binary-format raw-in-base64-out
Figure 3: Completed execution with customer approval

Figure 3: Completed execution with customer approval

After receiving the customer’s approval, the durable execution resumes from its checkpoint, authorizes the transaction, and completes. The execution spanned hours but consumed only seconds of compute time.

Conclusion

With durable functions, Lambda extends beyond single-event processing to power core business processes and long-running workflows, while retaining the operational simplicity, reliability, and scale that define Lambda. You can build applications that run for days or months, survive failures, and resume where they left off, all within the familiar event-driven programming model.

Deploy the fraud detection workflow from our GitHub repository and experiment with human-in-the-loop patterns in your own account. For core concepts, see Introduction to AWS Lambda Durable Functions. For comprehensive documentation, see the Lambda Developer Guide. Browse Serverless Land for reference architectures and discover where durable execution fits in your designs.

Share your feedback, questions, and use cases in the SDK repositories or on re:Post.

AWS Weekly Roundup: NVIDIA Nemotron 3 Super on Amazon Bedrock, Nova Forge SDK, Amazon Corretto 26, and more (March 23, 2026)

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-nvidia-nemotron-3-super-on-amazon-bedrock-nova-forge-sdk-amazon-corretto-26-and-more-march-23-2026/

Hello! I’m Daniel Abib, and this is my first AWS Weekly Roundup. I’m a Senior Specialist Solutions Architect at AWS, focused on the generative AI and Amazon Bedrock. With over 28 years of experience in solution architecture, software development, and cloud architecture, I help Startups & Enterprises harness the power of generative AI with Amazon Bedrock. I’ve been at AWS for more than six and a half years, working closely with customers across Latin America, and I’m also passionate about Serverless technologies.

Outside of work and endurance sports, I’m a dedicated father to Cecília (7) and Rafael (4), who keep me busier—and happier— than any distributed system ever could. I’m based in São Paulo, you can find me on LinkedIn and X (@DCABib), where I share insights about generative AI, Amazon Bedrock, AWS serverless services, and the occasional Ironman throwback.

Now, let’s get into this week’s AWS news…

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Amazon Redshift increases performance for new queries in dashboards and ETL workloads by up to 7x — Amazon Redshift now delivers up to 7x faster performance for new queries in dashboards and ETL workloads. Queries you run for the first time — without cached results — now execute significantly faster, reducing wait times for interactive dashboards and accelerating your ETL pipelines. This is particularly impactful for workloads with high query variability where cache hits are less frequent.
  • NVIDIA Nemotron 3 Super now available on Amazon Bedrock — NVIDIA Nemotron 3 Super is now available in Amazon Bedrock, expanding the lineup of foundation models you can access through the unified Bedrock API. Nemotron 3 Super is a high-performance language model optimized for tasks such as text generation, complex reasoning, summarization, and code generation. You can now invoke Nemotron 3 Super alongside other foundation models in your existing Bedrock workflows, without managing any infrastructure.
  • Introducing Nova Forge SDK, a seamless way to customize Nova models for enterprise AI — Nova Forge SDK provides a streamlined way to fine-tune and customize Amazon Nova models for enterprise use cases. You can adapt Nova models to your domain-specific data and deploy them directly within Amazon Bedrock, reducing the complexity of building tailored AI solutions. The SDK handles the heavy lifting of model customization, letting you focus on your business logic rather than the underlying infrastructure.
  • Amazon Corretto 26 is now generally available — Amazon Corretto 26, the latest long-term support (LTS) release of the no-cost, production-ready distribution of OpenJDK, is now generally available. Corretto 26 includes the latest Java language features, performance improvements, and security patches, all backed by long-term support from AWS. You can use it across development and production environments on Amazon Linux, Windows, macOS, and Docker images.
  • AWS Lambda now supports Availability Zone metadata — AWS Lambda now provides Availability Zone metadata for your function invocations. You can now identify which Availability Zone your Lambda function is running in, enabling better observability, more informed architectural decisions, and simplified troubleshooting for latency-sensitive and multi-AZ workloads. This is particularly useful when correlating Lambda execution with other AZ-aware services in your architecture.
  • Amazon CloudWatch Logs now supports log ingestion using HTTP-based protocol — Amazon CloudWatch Logs now supports ingesting logs using an HTTP-based protocol, making it simpler to send logs from applications and services that use standard HTTP endpoints. You can now route logs to CloudWatch Logs without requiring custom agents or additional SDK integrations, lowering the barrier to centralized log management across your workloads.
  • Amazon EKS announces 99.99% Service Level Agreement and new 8XL scaling tier for Provisioned Control Plane clusters — Amazon EKS now offers a 99.99% Service Level Agreement (SLA) for clusters running on Provisioned Control Plane, up from the 99.95% SLA offered on standard control plane. EKS is also introducing the 8XL scaling tier, the largest available Provisioned Control Plane tier, which doubles the Kubernetes API server request processing capacity of the next lower 4XL tier — ideal for large-scale workloads like AI/ML training, high-performance computing (HPC), and large-scale data processing.

Other AWS news
Here are some additional posts and resources that you might find interesting:

  • Kiro for students — Kiro is now available for students, giving the next generation of builders access to AI-powered development tools at no cost. As Swami Sivasubramanian shared on LinkedIn, “Students are the future decision-makers shaping technology” — and Kiro gives them hands-on experience building with AI from day one. If you’re a student or know someone who is, this is a great opportunity to start building with AI-assisted development.
  • Strands Steering Hooks achieved 100% agent accuracy — The Strands Agents team published results showing that Steering Hooks can achieve 100% agent accuracy, outperforming both prompt engineering and rigid workflow approaches for controlling agent behavior. As Swami highlighted on LinkedIn, building reliable AI agents often means rethinking how we guide model behavior — and Steering Hooks offer a compelling new path to agent reliability.
  • Introducing Badges on AWS Builder Center — AWS Builder Center now features badges that recognize your contributions and achievements within the builder community. You can earn badges by sharing solutions, participating in challenges, and engaging with fellow builders. It’s a great way to showcase your expertise and track your growth.
  • Keep Building Together: The Power of Community — A thoughtful read on the power of community-driven learning and collaboration in the AWS ecosystem. Whether you’re just getting started with AWS or you’ve been building for years, the builder community is a place to connect, share knowledge, and grow together. I highly recommend checking it out.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Summits — Join AWS Summits in 2026, free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), Bengaluru (April 23–24), Singapore (May 6), Tel Aviv (May 6), and Stockholm (May 7).
  • AWS Community Days — Community-led conferences where content is planned, sourced, and delivered by community leaders, featuring technical discussions, workshops, and hands-on labs. Upcoming events include San Francisco (April 10) and Romania (April 23–24).
  • AWSome Women Summit LATAM — Taking place on March 28 in Mexico City, this event celebrates and empowers women in cloud technology across Latin America. A fantastic initiative for the LATAM tech community.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse the AWS Events and Webinars for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

AWS Weekly Roundup: Amazon Connect Health, Bedrock AgentCore Policy, GameDay Europe, and more (March 9, 2026)

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-connect-health-bedrock-agentcore-policy-gameday-europe-and-more-march-9-2026/

Fiti AWS Student Community Kenya!

Last week was an incredible whirlwind: a round of meetups, hands-on workshops, and career discussions across Kenya that culminated with the AWS Student Community Day at Meru University of Science and Technology, with keynotes from my colleagues Veliswa and Tiffany, and sessions on everything from GitOps to cloud-native engineering, and a whole lot of AI agent building.

JAWS Days 2026 is the largest AWS Community Day in the world, with over 1,500 attendees on March 7th. This event started with a keynote speech on building an AI-driven development team by Jeff Barr, and included over 100 technical and community experience sessions, lightning talks, and workshops such as Game Days, Builders Card Challenges, and networking parties.

Now, let’s get into this week’s AWS news…

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Introducing Amazon Connect Health, Agentic AI Built for Healthcare — Amazon Connect Health is now generally available with five purpose-built AI agents for healthcare: patient verification, appointment management, patient insights, ambient documentation, and medical coding. All features are HIPAA-eligible and deployable within existing clinical workflows in days.
  • Policy in Amazon Bedrock AgentCore is now generally available — You can now use centralized, fine-grained controls for agent-tool interactions that operate outside your agent code. Security and compliance teams can define tool access and input validation rules using natural language that automatically converts to Cedar, the AWS open-source policy language.
  • Introducing OpenClaw on Amazon Lightsail to run your autonomous private AI agents — You can deploy a private AI assistant on your own cloud infrastructure with built-in security controls, sandboxed agent sessions, one-click HTTPS, and device pairing authentication. Amazon Bedrock serves as the default model provider, and you can connect to Slack, Telegram, WhatsApp, and Discord.
  • AWS announces pricing for VPC Encryption Controls — Starting March 1, 2026, VPC Encryption Controls transitions from free preview to a paid feature. You can audit and enforce encryption-in-transit of all traffic flows within and across VPCs in a region, with monitor mode to detect unencrypted traffic and enforce mode to prevent it.
  • Database Savings Plans now supports Amazon OpenSearch Service and Amazon Neptune Analytics — You can save up to 35% on eligible serverless and provisioned instance usage with a one-year commitment. Savings Plans automatically apply regardless of engine, instance family, size, or AWS Region.
  • AWS Elastic Beanstalk now offers AI-powered environment analysis — When your environment health is degraded, Elastic Beanstalk can now collect recent events, instance health, and logs and send them to Amazon Bedrock for analysis, providing step-by-step troubleshooting recommendations tailored to your environment’s current state.
  • AWS simplifies IAM role creation and setup in service workflows — You can now create and configure IAM roles directly within service workflows through a new in-console panel, without switching to the IAM console. The feature supports Amazon EC2, Lambda, EKS, ECS, Glue, CloudFormation, and more.
  • Accelerate Lambda durable functions development with new Kiro power — You can now build resilient, long-running multi-step applications and AI workflows faster with AI agent-assisted development in Kiro. The power dynamically loads guidance on replay models, step and wait operations, concurrent execution patterns, error handling, and deployment best practices.
  • Amazon GameLift Servers launches DDoS Protection — You can now protect session-based multiplayer games against DDoS attacks with a co-located relay network that authenticates client traffic using access tokens and enforces per-player traffic limits, at no additional cost to GameLift Servers customers.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

From AWS community
Here are my personal favorite posts from AWS community and my colleagues:

  • I Built a Portable AI Memory Layer with MCP, AWS Bedrock, and a Chrome Extension — Learn how to build a persistent memory layer for AI agents using MCP and Amazon Bedrock, packaged as a Chrome extension that carries context across sessions and applications.
  • When the Model Is the Machine — Mike Chambers built an experimental app where an AI agent generates a complete, interactive web application at runtime from a single prompt — no codebase, no framework, no persistent state. A thought-provoking exploration of what happens when the model becomes the runtime.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Community GameDay Europe — Think you know AWS? Prove it at the AWS Community GameDay Europe on March 17, a gamified learning event where teams compete to solve real-world technical challenges using AWS services.
  • AWS at NVIDIA GTC 2026 — Join us at our AWS sessions, booths, demos, and ancillary events in NVIDIA GTC 2026 on March 16 – 19, 2026 in San Jose. You can receive 20% off event passes through AWS and request a 1:1 meeting at GTC.
  • AWS Summits — Join AWS Summits in 2026: free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), and Bengaluru (April 23–24).
  • AWS Community Days — Community-led conferences where content is planned, sourced, and delivered by community leaders. Upcoming events include Slovakia (March 11), Pune (March 21), and the AWSome Women Summit LATAM in Mexico City (March 28)

Browse here for upcoming AWS led in-person and virtual events, startup events, and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— seb

Enabling high availability of Amazon EC2 instances on AWS Outposts servers (Part 3)

Post Syndicated from Brianna Rosentrater original https://aws.amazon.com/blogs/compute/enabling-high-availability-of-amazon-ec2-instances-on-aws-outposts-servers-part-3/

This post is part 3 of the three-part series ‘Enabling high availability of Amazon EC2 instances on AWS Outposts servers’. We provide you with code samples and considerations for implementing custom logic to automate Amazon Elastic Compute Cloud (EC2) relaunch on Outposts servers. This post focuses on guidance for using Outposts servers with third party storage for boot and data volumes, whereas part 1 and part 2 focus on automating EC2 relaunch between standalone servers. Outposts servers support integration with Dell PowerStore, HPE Alletra Storage MP B10000 systems, NetApp on-premises enterprise storage arrays, and Pure Storage FlashArray.

Outposts servers provide compute and networking services that are designed for low-latency, local data processing needs for on-premises locations such as retail stores, branch offices, healthcare provider locations, or environments that are space-constrained. Outposts servers use EC2 instance store storage to provide non-durable block-level storage to the instances running stateless workloads. For applications that require persistent storage, you can create a three-tier architecture by connecting your Outposts servers to a third-party storage appliance. In this post, you will learn how to implement custom logic to provide high availability (HA) for your applications running on Outposts servers using two or more servers for N+1 fault tolerance. The code provided is meant to help you get started, and can be modified further for your unique workload needs.

Overview

In the following sections we will show how custom logic can be used to automate EC2 instance relaunch between two or more Outposts servers using boot and data volumes on third party storage. If your EC2 instance fails while using this solution, an Amazon CloudWatch alarm monitoring the EC2 StatusCheckFailed_Instance metric of your source EC2 instance will be triggered, and you will receive an Amazon Simple Notification Service (Amazon SNS) notification. An AWS Lambda function will then relaunch your EC2 instance onto the destination Outposts server that you’ve set up for resiliency. This is done using a launch template created during setup, and the script will connect your relaunched instance to the existing boot and data volumes on your third party storage appliance. This storage device provides shared storage for your Outposts servers. If a single server fails, new instances can connect to existing volumes on the array. This allows for a zero data loss Recovery Point Objective (RPO) and a Recovery Time Objective (RTO) equaling the time it takes to launch your EC2 instance. Take advantage of the features on your storage appliance for configuring data durability and resiliency to hardware failures, and make sure that you are regularly backing up your SAN volumes.

Figure 1 – Solution Architecture for automated EC2 Relaunch

Prerequisites

The following prerequisites are required to complete the walkthrough:

  • Two Outposts servers that can be set up as an active-active or active-passive resilient pair.
  • For workloads with a low threshold for downtime, ensure that your secondary Outpost server that’s used for recovery has a unique service link connection.
  • Outposts servers must be colocated within the same Layer 2 (L2) network.
  • Network latency between the Outposts servers must not exceed 5ms round trip time (RTT).
  • A storage appliance that supports the iSCSI protocol. Credentials to manage the storage appliance initiator/target mappings. See Simplifying the use of third-party block storage with AWS Outposts for more information.
  • If you’re setting this up from an Outposts consumer account, you must configure Amazon CloudWatch cross-account observability between the consumer account and the Outposts owning account to view Outposts metrics in your consumer account.
  • Create launch templates for the EC2 instances that you want to protect, the launch wizard will help you create these.
  • Credentials with permissions for AWS CloudFormation, Amazon EC2, and (optional) AWS Secrets Manager if authentication is required. IAM Permission Examples.md is provided in the repository.
  • A Windows or Linux host that can access the storage appliance and your AWS account (management computer).
  • AWS Outposts iPXE Amazon Machine Image (AMI) from the AWS Marketplace.
  • Python 3.8 or later (recommended) is used to run the init.py script that dynamically creates a CloudFormation stack in the account specified as an input parameter.
  • AWS SDK for Python (Boto3) version 1.26.0 or later recommended.
  • Operating system with iSCSI boot support (Windows Server 2022 and Red Hat Enterprise Linux 9 AMIs are provided).
  • Internet access to AWS service endpoints for the private subnet hosting the recovery Lambda function.
  • Download the repository sample-outposts-third-party-storage-integration.

Walkthrough

The first step is to deploy an EC2 instance configured to boot from a volume on the third-party storage that is prepared with an OS boot image. This step uses the launch wizard portion of the solution.

  1. Download and extract the OutpostServer_Recovery_3Pstorage repository to the management computer that has the AWS SDK for Python (Boto3) and Python installed.
  2. Run launch_wizard from the sample-outposts-third-party-storage-integration directory. You can run interactively or provide arguments for region, subnet, iPXE AMI, storage vendor, storage management ip, and credentials.

Figure 2 – Running launch wizard

  1. When prompted for a feature name, enter sanboot.
  2. For Guest OS type, enter in Linux or Windows.
  3. When prompted “Do you want to continue with this unverified AMI?”, select Y.
  4. The launch wizard will provide a list of instance types available on the Outpost server associated with the subnet you specified. Enter the instance type that you want to use.
  5. The launch wizard will now prompt you for optional EC2 Key Pair, Security Group, and Instance Profile settings for the EC2 instance that you are launching.
  6. Next, the launch wizard prompts you to specify an instance name. Note that specifying an instance name is required to set up automated instance recovery because the instance name is used as part of the recovery process.

Figure 3 – Taking user input for variable values

  1. The launch wizard prompts for root volume size. This is the root volume that the iPXE AMI boots from. The default is a 1GB volume on the Outpost server instance storage.
  2. Next, the launch wizard prompts you to select which third party storage controller you want to use based on the management ip that you specified. In this example, we are using NetApp, so I select a NetApp Storage Virtual Machine (SVM) named outpost_iscsi.
  3. If the connection to the storage array is successful and the protocol is available (iSCSI or NVMe over TCP) you are provided additional storage options for initiator group and logical unit number (LUN).
  4. In this example, we are using NetApp with iSCSI, so I can select an existing initiator group or create a new one.
  5. You can specify an existing initiator qualified name (IQN), or the launch wizard can generate a new one. IMPORTANT: Make sure that IQNs are unique to each instance because duplicates can cause data corruption.
  6. Next the launch wizard prompts which LUN’s you want to connect to this instance. For this example, I am going to use a Windows Server 2022 boot volume that I already created on the NetApp storage array.
  7. You are now asked which storage array target interface you want to use for connecting to these LUNs.
  8. The launch wizard provides the capability to specify guest OS scripts to customize the OS after sanboot. Combining this capability with storage array cloning provides a streamlined process for deploying new instances.
  9. The launch wizard now displays the EC2 user data template that it generated for use with the iPXE AMI and asks if you want to proceed with launching the instance.
  10. After the EC2 instance is launched, select yes to proceed with automated instance recovery setup.

Figure 4 – Running launch template creation script

Generating EC2 launch templates for recovery and failback

In the second step, we are generating EC2 launch templates for the EC2 instance launched in step 1. Launch templates can be generated for the primary and secondary Outpost servers. The launch template for the secondary Outpost server can be used for automated or manual recovery of the EC2 instance. Failback to the primary Outpost server is manual using the primary launch template.

  1. Select the instance that you want automated recovery for and select the subnet that you launched the instance in. This subnet represents the primary Outpost server that the instance is running on.

Figure 5 – Selecting subnets for EC2 instance relaunch

  1. When prompted to create a second launch template for Outpost server recovery, select yes, and then select to use the same instance (for recovery on different Outpost server).
  2. When you get a list of available subnets, select the subnet that’s associated with your secondary Outpost server. This is the server that the EC2 instance will be launched on in the event of the EC2 StatusCheckFailed_Instance metric triggers the CloudWatch alarm.
  3. You will see both launch templates created successfully.

Deploying automated EC2 instance recovery

The third step creates a CloudFormation template for monitoring, notifications, and automated recovery of the EC2 instance deployed in step 1. The CloudFormation template automatically captures the instance and secondary launch template information necessary for automatic recovery.

  1. Select Y to set up automated recovery. This will create a CloudFormation stack.
  2. Provide a name and description for the CloudFormation stack.
  3. Select whether you want automated recovery or notification only. This provides flexibility to choose manual or automatic recovery based on whether you want to verify the primary Outpost server is down before initiating recovery.
  4. In the AWS CloudFormation console, monitor the CloudFormation stack creation process.

Figure 6 – CloudFormation stack creation in progress

  1. After the CloudFormation Stack is complete, you have successfully deployed an EC2 instance using third party storage for boot and data volumes on a primary Outpost server. You also created instance recovery capabilities by using the Amazon Outpost server automated recovery solution for third party storage.
  2. You can verify whether the EC2 StatusCheckFailed_Instance is healthy under the Alarms section in the Amazon CloudWatch console.

Considerations

The logic discussed in this post relies on the secondary destination Outposts server having a connected service link. For more information about how to create a highly available service link connection for your Outpost servers, see the Networking section of AWS Outposts High Availability Design and Architecture Considerations whitepaper.

Clean up

Confirm whether it is safe to terminate the Amazon EC2 instance that you launched with this walkthrough. The operating system and data volumes are on the third party storage, so EC2 instance termination only removes the iPXE AMI from the Outposts server instance storage. To clean up, complete the following steps.

  1. Terminate the Amazon EC2 instance. Then, verify that the Instance state is Terminated to ensure that the instance is not using Outposts server resources.
  2. Delete the Amazon EC2 Launch Templates associated with the Amazon EC2 instance that you terminated. The names of the launch templates that were automatically generated will start with ‘lt-‘, followed by the instance name and the instance id. If you generated a recovery launch template, it will have a ‘-recovery’ suffix in the name.
  3. Delete the AWS CloudFormation Stack. The Stack name will start with ‘autorestart-‘ followed by the Amazon EC2 instance name.
  4. Clean up your initiators, initiator group, and LUNs on the third party storage array.

Conclusion

With the use of custom logic through AWS tools such as CloudFormation, CloudWatch, Amazon SNS, and AWS Lambda, you can architect for HA for stateful workloads on Outposts server. By implementing the custom logic in this post, you can automatically relaunch EC2 instances running on a source Outposts server to a secondary destination Outposts server if an instance fails, and connect to existing volumes on a shared storage appliance for recovery. This also reduces the downtime of your applications in the event of a hardware or service link failure. The code provided in this post can be further expanded upon to meet the unique needs of your workload.

While the use of infrastructure-as-code (IaC) can improve your application’s availability and be used to standardize deployments across multiple Outposts servers, it’s crucial to do regular failure drills to test the custom logic in place. This is to make sure that you understand your application’s expected behavior on relaunch in the event of a failure. To learn more about Outposts servers, visit the Outposts servers User Guide. Reach out to your AWS account team, or fill out this form to learn more about Outposts servers.

Automate AWS Lambda Runtime Upgrades with AWS Transform custom

Post Syndicated from Venugopalan Vasudevan original https://aws.amazon.com/blogs/devops/automate-aws-lambda-runtime-upgrades-with-aws-transform-custom/

Introduction

Organizations carry a growing burden of technical debt — aging codebases, outdated runtimes, and legacy frameworks that slow innovation, increase security risk, and inflate maintenance costs. Addressing this debt requires tackling a wide range of code transformation challenges: version upgrades, runtime migrations, framework transitions, and language translations, all of which must be repeated across multiple codebases. Today, most organizations perform these tasks manually, consuming 20–30% of enterprise software development effort. Even where automation exists, it’s typically narrow in scope and requires significant upfront investment — leaving most organizations unable to scale transformations effectively and, as a result, unable to meaningfully reduce the technical debt that continues to compound over time.

AWS Transform custom addresses this gap — an intelligent AI agent that learns organization-specific code transformations, executes them consistently at scale, and improves from developer feedback, without requiring specialized automation expertise. This blog explores how AWS Transform custom tackles one of the most pressing transformation challenges today.

Managing Lambda Runtime Lifecycles

AWS Lambda follows a runtime deprecation policy that aligns with the end of community long-term support for programming languages, with all current deprecation schedules available in the AWS Lambda runtime documentation. As upstream language maintainers deprecate runtime versions – including Python 3.8, Node.js 14, and Java 8  , organizations face the critical challenge of upgrading hundreds or thousands of Lambda functions before these runtimes reach end-of-life.

When a Lambda runtime reaches end of life, functions lose access to security patches and technical support, leaving applications potentially exposed to known vulnerabilities and compliance risks. Performance degrades as optimizations in newer runtimes go unrealized, and technical debt compounds — the longer you wait, the harder and more expensive the migration becomes.

For organizations managing hundreds or thousands of Lambda functions across multiple runtimes and languages, the effort is amplified by the scale of coordination required, the manual burden of testing and validation, and the reality that upgrade expertise is often siloed within a handful of engineers. It’s a recurring, high-stakes cycle that pulls teams away from building features.

This is exactly the type of repeatable, organization-wide transformation that AWS Transform custom code transformations can automate. The following sections explore how you can use AWS Transform custom to address Python runtime upgrades and demonstrate the automated approach with a practical example.

Sample application

This demonstration uses SAM Python CRUD Sample — an open-source serverless application built with AWS SAM that implements a full CRUD API. The application consists of five Lambda functions that create, read, update, list, and delete activity records. The following walkthrough shows how AWS Transform custom automates Python runtime upgrades by migrating these Lambda functions from a deprecated runtime (Python 3.8) to a modern runtime version (Python 3.13).

Prerequisites

Before beginning the transformation process, verify the following requirements:

  • AWS Transform CLI installed and configured in your development environment
  • Authentication with AWS credentials configured locally and proper IAM permissions to call AWS Transform
  • Git installed for cloning sample repositories
  • uv package manager for Python environments

For detailed setup instructions, see the Getting Started with AWS Transform custom guide.

Hands-On Example: Lambda Runtime Upgrade

Step 1: Prepare the Sample Project

Clone a sample Python Lambda repository to your local environment

git clone https://github.com/aws-samples/sam-python-crud-sample.git
cd sam-python-crud-sample

Verify the Initial Setup and make sure all the Tests Pass:

uv venv --python 3.8 # uv will automatically download Python 3.8 if not already installed
source .venv/bin/activate
uv pip install -r requirements.txt
uv pip install -r requirements_dev.txt
uv pip install "moto[dynamodb]<3"
python -m pytest tests/ -v -o "addopts="

Step 2: Start the Transformation

AWS Transform custom supports both interactive and non-interactive execution (non-interactive mode for CI/CD and batch execution is covered at the end). Launch the CLI in interactive mode with -t to trust all tools, which lets you use natural language to invoke and define transformations:

atx -t

Note : The -t flag trusts all tool executions without prompting for confirmation. This is convenient

for walkthroughs but means the agent can run shell commands automatically. Review the Trust settings for details on controlling tool permissions.

From here, you can use natural language to list and invoke transformations.

>list all the transformations available

AWS Managed Transformations list

This lists all available transformations — both AWS-managed and custom. AWS provides built-in transformations for common tasks like language upgrades (Java, Python, Node.js), SDK migrations, and Graviton migration. You can also create your own custom transformations using natural language, docs, and code samples.

Invoke the transformation by specifying the transformation name AWS/python-version-upgrade and project path. The agent will prompt you for additional inputs like target Python version and codebase path during the flow.

> Run AWS/python-version-upgrade on my project

Python version upgrade transformation start

Step 3: Transformation Planning

Before starting the planning process, you can provide any additional feedback if any. You can say proceed if you don’t have specific preferences.

Python version upgrade transformation pre-planning

This will start the planning process, where the agent will analyze all the source files, context, additional guidance and generate a step-by-step comprehensive plan detailing:

  • Runtime version updates (Python 3.8 → 3.13)
  • Dependency compatibility checks
  • Code pattern updates for Python 3.13 compatibility
  • AWS Lambda configuration changes
  • Infrastructure as Code changes

The transformation plan is designed to help maintain functionality while leveraging Python 3.13’s improvements.

Python version upgrade transformation plan summaryPython version upgrade transformation plan summary

You can review the plan and provide feedback, or tell the agent to go ahead and execute it.

Step 4: Transformation Execution and Validation

After reviewing the plan, AWS Transform custom executes the transformation automatically, updating:

  • Lambda runtime configuration
  • Python version-specific syntax
  • Dependency versions for Python 3.13 compatibility
  • Any deprecated function calls
  • Any Infrastructure as code templates as well

At the end of each step, the agent commits the incremental changes to a local git branch. If build or test errors occur, the agent attempts to self-debug and resolve issues. As a user, you can stop the transformation and provide feedback if necessary. Once all the steps are complete, the agent will produce a summary of changes.

Python version upgrade transformation execution completion

Next, the agent runs a full validation — comparing the executed changes against the plan for any deviations, verifying all exit criteria are met, and running build commands and unit tests to confirm everything passes.

Once the validation is complete, the agent will ask for feedback. You can provide feedback on the execution results and ask the agent to modify/add/remove changes if needed.

Python version upgrade transformation validation completion

Step 5: Verify Changes

Once the validation is complete, the agent summarizes the changes:

Python version upgrade transformation summary of changes

You can quit the atx session by issuing /quit in the terminal.

All the changes are committed to a local staging branch. You can also view this by executing following commands

git status
git branch
git diff main <atx-result-staging-...>

With these steps, you can upgrade your Lambda functions which are already running on deprecated runtimes or nearing EOL with reduced manual effort.

Non-Interactive mode

You can also run this transformation in a non-interactive mode with the following command supplying all the information, so that agent can run without asking for any user inputs. This mode is designed for headless execution, CI/CD pipeline integration and bulk execution where no human intervention is available or desired.

atx custom def exec -p . -n AWS/python-version-upgrade --configuration "validationCommands=pytest,additionalPlanContext=The target Python version to upgrade to is Python 3.13" -x -t

Parameter breakdown:

  • -n AWS/python-version-upgrade: Name of the AWS managed Python migration transformation
  • -p .: Path to the current directory containing your Lambda function
  • -t : Trust all tools without prompting
  • -x : non-interactive headless mode
  • --configuration : Validation commands to be used after the transformation and additional instructions to the agent . This example, configures the agent to use “pytest” as the validation command after transformation is complete and specifies the target version of python3.13 as additionalPlanContext. This helps agent with additional context during planning of the changes.

You can also specify these parameters in a config.json and execute like below.

atx custom def exec --configuration 'file://config.json'

config.json file contains all the information about the project repository path, transformation name, build and validation commands to use. Save the below snippet to config.json in the current directory.

{
  "codeRepositoryPath": ".",
  "transformationName": "python-version-upgrade",
  "validationCommands": "pytest",
  "additionalPlanContext": "The target Python version to upgrade to is Python 3.13"
}

How to scale this to multiple Lambda function upgrades?

Now that you have successfully used AWS Transform custom to upgrade a single Lambda function, you can scale this to hundreds or thousands of functions across your organization. Central engineering teams can create campaigns through the AWS Transform web application to define the transformation, specify target repositories, and track progress across the organization. For execution at scale, choose the model that fits your environment — both run in your environment, with access to your existing development resources, build systems, and tool chains. You don’t need to move your code anywhere — AWS Transform custom meets you where you are.

  1. Batch script execution — Ideal for teams that want to run transformations directly on developer machines, EC2 instances or existing CI/CD infrastructure. Wrap the AWS Transform custom CLI in a batch processing script that iterates across multiple repositories using a CSV or JSON input file. The script supports both serial and parallel execution modes with configurable job limits, retry mechanisms, and comprehensive logging. Refer to this GitHub repo for the sample batch launcher script and execution instructions.
  2. Containerized execution on AWS — Best suited for enterprise-scale rollouts where you need managed infrastructure, job orchestration, and centralized monitoring. Run transformations using containers deployed on AWS Batch with AWS Fargate. This solution provides a REST API for job submission, automatic IAM credential management, and full Amazon CloudWatch monitoring — all deployable with a single AWS CDK command. To get started, refer to this GitHub repo and blog.

Cleanup

If you followed along with the hands-on example, remove the cloned repository and virtual environment to free up local resources:

deactivate
cd ..
rm -rf ./sam-python-crud-sample

If you created any AWS resources during testing, delete them to avoid ongoing charges.

Conclusion

Keeping Lambda runtimes current is a recurring operational burden that only grows with scale. What starts as a simple version bump quickly compounds into dependency updates, syntax changes, infrastructure modifications, and extensive testing — multiplied across every function in your fleet.

AWS Transform custom turns this into a repeatable, automated workflow. As we demonstrated, upgrading a multi-function Python 3.8 application to Python 3.13 required just a single CLI invocation — the agent can handle planning, code changes, dependency updates, infrastructure configuration, and validation end to end. And with non-interactive mode and the scaled execution options, you can extend this to hundreds of repositories without manual intervention.

To get started:

About the authors

Venu-author

Venugopalan Vasudevan

Venugopalan Vasudevan (Venu) is a Senior Specialist Solutions Architect at AWS, where he leads Agentic AI initiatives focused on AWS Transform. He helps customers adopt and scale AI-powered developer and modernization solutions to accelerate innovation and business outcomes.

gokul-author

Gokul Sarangaraju

Gokul Sarangaraju is a Senior Solutions Architect at AWS, specializing in code modernization using agentic AI and AWS services. He helps customers adopt AWS technologies, optimize costs and usage, and build scalable, cost-effective data analytics solutions.

6,000 AWS accounts, three people, one platform: Lessons learned

Post Syndicated from Ben Freiberg original https://aws.amazon.com/blogs/architecture/6000-aws-accounts-three-people-one-platform-lessons-learned/

This post is cowritten by Julius Blank from ProGlove.

As software-as-a-service (SaaS) platforms grow, balancing speed of innovation with strong security and tenant data isolation becomes critical. While the same AWS Identity and Access Management (IAM) mechanisms secure both shared and dedicated environments, establishing a hard security boundary is often easier in an account-per-tenant model because the account itself becomes the isolation boundary. In shared-account deployments, you instead rely on resource-level boundaries such as tenant-scoped IAM policies and data partitioning. This multi-tenancy increases architectural and operational complexity and can introduce security challenges if safeguard mechanisms are not properly designed and enforced. By adopting an account-per-tenant model on Amazon Web Services (AWS), you can achieve clearer security boundaries, streamlined ownership of services, and more transparent cost attribution, but this comes at the expense of increased investment in platform automation.

At ProGlove, we build smart wearable barcode scanning solutions that connect frontline workers to digital workflows. Our scanners integrate with Insight, our AWS based SaaS platform, to provide real-time process visibility. This helps customers in manufacturing, logistics, and retail improve their productivity, reduce errors, and enhance ergonomics on the shop floor.

This post describes why we chose a account-per-tenant approach for our serverless SaaS architecture and how it changes the operational model. It covers the challenges you need to anticipate around automation, observability and cost. We will also discuss how the approach can affect other operational models in different environments like an enterprise context.

Why multi-account?

Many SaaS providers begin their journey with a straightforward, dedicated deployment model, often with one AWS account per tenant. This approach makes initial implementation straightforward and limits the scope of issues, but as the platform scales, operational overhead and inefficiencies from idle or underutilized resources increase. These inefficiencies can be mitigated with serverless architectures that scale automatically to demand. Over time, providers often look to shared or multi-tenant models to consolidate operations and improve cost efficiency. However, this shift introduces new challenges as the number of tenants and services grows:

  • Blast radius – An accidental misconfiguration or vulnerability could expose multiple tenants.
  • Quota limits – Tenants in a single AWS account share the same quotas.
  • Operational complexity – Shared infrastructure makes it difficult to reason about ownership of resources.
  • Customization limits – Making changes for one tenant risks impacting others.
  • Cost visibility – Attributing resource usage to individual tenants is challenging.

Choosing between a dedicated or shared model is ultimately a trade-off. Dedicated deployments are more straightforward to build but require investment in SaaS operations and orchestration to manage at scale, whereas shared models reduce operational overhead but increase architectural and management complexity.

AWS recommends a multi-account strategy to organizing your AWS environment. At scale, the AWS account boundary is the easiest way to implement isolation. Accounts are fully isolated containers for compute, storage, networking and more, with no shared scope unless you explicitly configure it.

Working backwards from our use case, we decided to take this model to its logical extreme: every tenant gets their own AWS account. The services they consume are deployed directly into that account. In that account, we deploy the full set of microservices that the tenant requires. These services run exclusively with that tenant’s data and configuration. At our current scale, ProGlove manages approximately:

That translates to over 120,000 deployed service instances and roughly 1,000,000 Lambda functions in production. The following diagram shows an overview of the main services used in our platform.

AWS multi-account architecture diagram showing hierarchical organization with Root, Audit, Monitoring, Deployment, and Tenant accounts containing various AWS services

Benefits of the account-per-tenant model

This model brings several benefits that directly support security, agility, and operational clarity, including a strong isolation model, simplified mental model, customization per tenant, and transparent cost attribution. Tenant data is not co-located. Each account has its own storage, compute, and permissions. If a security issue, runaway process, or misconfiguration occurs, the impact is limited to that tenant’s account while other tenants remain unaffected. For developers, they don’t need to think multi-tenancy as a deployed service instance always belongs to exactly one tenant. This reduces cognitive load and simplifies debugging. Developers can easily be provided with isolated, production-like tenant accounts to eliminate the gap between development and production environments. You can modify, test, and migrate individual accounts independently. This helps to create tailored deployments, such as activating premium features for certain tenants, without impacting the overall system.

AWS Cost Explorer and linked accounts make it straightforward to report and charge back costs on a per-tenant basis. For SaaS providers with consumption-based pricing models, this becomes a strong advantage.

When conducting an AWS Well-Architected Framework review together with AWS, we found that many items from the operational excellence as well as the security pillar didn’t even apply to our setup anymore. This made completing those review sections quick and straightforward.

Challenges and trade-offs

The account-per-tenant model, like most architectural choices, involves trade-offs. Although the model provides strong isolation, it introduces challenges in platform operations. The approach shifts complexity away from application development to platform development.

Provisioning, configuring, and managing thousands of accounts isn’t feasible manually. Automation of account creation, baseline setup, IAM roles, guardrails, and service enablement is mandatory. We rely on AWS Organizations, its service control policies (SCPs), and AWS CloudFormation StackSets, as well as custom tooling to handle this.

Some of the involved workflows lend themselves well to automation, whereas others can be implemented more effectively using traditional scripting and manual operations, as long as the overhead introduced is low enough. For example, account creation is a fully automated process using AWS Step Functions, but the retirement and closure of accounts are performed manually through regularly run scripts.

AWS account lifecycle management diagram showing automated provisioning with Step Functions and CloudFormation, plus manual retirement process with scripts

Some AWS services are billed per provisioned resource and independent of utilization as opposed to fully scaling to zero when not used. Prominent examples are Amazon Elastic Compute Cloud (Amazon EC2) or Amazon Relational Database Service (Amazon RDS), where resources need to be provisioned to use the service. Even the smallest EC2 instance type is charged at around USD $3, which adds up to USD $3,000 when deployed into 1,000 accounts. By contrast, serverless offerings such as AWS Lambda or Amazon DynamoDB automatically scale based on actual usage, minimizing idle resource costs. Although the per‑invocation or per‑request pricing for serverless services can seem higher, these models often offset the operational overhead and resource wastage associated with always‑on infrastructure. In any case, costs should be carefully modeled, measured, and optimized.

Monitoring infrastructure across accounts and Regions at scale is significantly harder than monitoring a handful of accounts. Observability tooling should be centralized, but without reintroducing the very risks that accounts are meant to isolate. It’s important to point out that Amazon CloudWatch offers greatly improved cross-account observability features today than when we started, for example, the Observability Access Manager.

Developers, operations teams, and platform services and tools need to operate across accounts on a daily basis. This requires a robust identity model with IAM roles and cross-account trust policies. If not designed carefully, this can become a source of complexity and security risk. Also, make sure to follow the best practice of avoiding long-lived credentials because these introduce a major security threat and monitoring effort if deployed into many accounts. AWS service limits are enforced per account. In a shared-account model, you monitor a single set of quotas. In an account-per-tenant setup, quota management becomes distributed and harder to predict. Proactive quota requests and monitoring are essential. For example, AWS Lambda employs a quota for the number of concurrent executions that functions in a single account share. In case a tenant is under heavier load, it’s likely for the corresponding account to experience throttling errors of Lambda functions, which is why it’s essential to provide a single pane of glass view to keep track of the quota usage and adapt as necessary. Although multi-account strategies are common at the enterprise level, adopting them at the SaaS tenant level is less common. Patterns, tooling, and reference architectures are still evolving, which means building custom solutions becomes necessary. Make sure to research available resources and consult AWS so you don’t reinvent the wheel.

Scaling observability across tenants

Observability can become a challenge in this architecture. If each tenant account emits its own logs, metrics, and traces, operational visibility becomes fragmented. For enhanced cross-account capabilities, we used a third-party observability solution. As an example, we forward telemetry (logs and metrics) to a central application where we can configure multi-alerts that are defined one time and applied to tenant accounts individually. This not only reduces cost but also simplifies the operational experience. Engineers interact with a single view, while underlying telemetry still originates from isolated accounts.

It’s vital to use tags whenever possible to correlate telemetry data as well as to use a consistent tagging and naming convention. Depending on the scale of operations, consider using AWS Organizations tag policies to enforce a consistent scheme. As an example, we include fields for the source AWS account ID in most metrics and logs to make sure we can easily drill down into the data for one particular tenant.

Key takeaways:

  • Don’t replicate per-account alarms blindly. Use streaming and aggregation.
  • Use tags for consistent context across thousands of instances.
  • Stay current with AWS feature releases with the AWS News Blog: metric streams, Amazon EventBridge integrations, Amazon CloudWatch Observability Access Manager, and other offerings can streamline your observability stack.
  • Follow the What’s New with AWS feed.

CI/CD and deployment at scale

Deploying microservices into one AWS account is straightforward. Deploying the same service into thousands of accounts requires a different approach. Our application code is stored in a monorepo, which helps us to enforce the same version of libraries or Lambda layers among others. The following diagram illustrates how we update many tenant accounts using AWS CodePipeline combined with AWS CloudFormation StackSets to deploy the applications. Each pipeline execution updates many target accounts in parallel, with only a single StackSet update operation in a central account.

AWS CloudFormation StackSet architecture showing centralized deployment from Infrastructure Account to multiple Tenant Accounts via CodePipeline

While this provides the necessary scale, it also introduces new failure modes:

  • Partial rollouts – If one account fails to deploy, rollback or retry strategies need to be defined and tested.
  • Pipeline duration – Large-scale updates can take significant time to propagate.
  • Tooling maturity – StackSets are powerful but still evolving, and operational edge cases are possible.

In practice, this requires investing in platform engineering. A dedicated team builds and maintains internal tools that abstract deployment complexity away from service developers. Developers remain focused on business logic, and the platform team takes care of consistency and reliability across accounts.

Cost management

Cost modeling changes significantly with this architecture. In a shared account, many costs are pooled, making per-tenant attribution difficult. In a account-per-tenant model, costs are naturally segmented by account .On the positive side, tenant-specific cost reporting is trivial. SaaS providers can align billing directly with AWS usage and even get monthly reporting per tenant automatically through AWS billing.

Costs that scale per account needs to be carefully considered. At scale, even small charges per resource become meaningful. For example, collecting metrics from thousands of accounts requires careful planning and the chosen approach has great influence on costs. At this scale, it isn’t feasible to use standard observability tooling out of the box because the volume of collected data can make per‑account costs economically unsustainable. Instead, focus on understanding which metrics you need to monitor and select an observability approach that allows you to implement that. As a recommendation, evaluate cost multipliers early. Services that scale linearly with the number of accounts should be avoided where possible. Make sure to verify your assumptions with actual measurements.

Operational considerations

To succeed with this model, you need to be prepared to invest in platform capabilities:

  • Account management – Automate everything from creation to decommissioning.
  • Baseline guardrails – Enforce compliance and security controls using SCPs and a strict IAM management.
  • Developer training – Make sure teams understand the scope and boundaries of their services.
  • CI/CD investment – Pipelines need to scale to thousands of accounts without blocking innovation.
  • Observability discipline – Monitoring needs to be consistent, centralized, and cost-effective.

Conclusion

In this post, we described how ProGlove implemented a large-scale account-per-tenant model on AWS and how that model shifts complexity from service code to platform operations. This is a trade-off that requires more platform automation, scalable CI/CD pipelines, and disciplined observability practices. The benefits are strong tenant and workload isolation, transparent costs, and severely reduced blast radius. These benefits are key for platform providers operating at scale with a strictly limited operations team size. Managing thousands of AWS accounts with three people might sound impossible. But with the right architectural choices, every new workload adds only marginal operational load while the platform absorbs the exponential scale. The team size stays constant, and efficiency grows with every account added. If security, compliance, and clarity are top priorities, this approach can serve as a strong foundation for your platform. Working backwards from these requirements can help you achieve the same balance: scaling your tenant base drastically, without scaling your operations team at the same rate.

Read more on Best practices for a multi-account environment, Managing stacks across accounts and Regions with StackSets, and the SaaS Lens for the AWS Well-Architected Framework.

About the authors

Optimizing Compute-Intensive Serverless Workloads with Multi-threaded Rust on AWS Lambda

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/compute/optimizing-compute-intensive-serverless-workloads-with-multi-threaded-rust-on-aws-lambda/

Customers use AWS Lambda to build Serverless applications for a wide variety of use cases, from simple API backends to complex data processing pipelines. Lambda’s flexibility makes it an excellent choice for many workloads, and with support for up to 10,240 MB of memory, you can now tackle compute-intensive tasks that were previously challenging in a Serverless environment. When you configure a Lambda function’s memory size, you allocate RAM and Lambda automatically provides proportional CPU power. When you configure 10,240 MB, your Lambda function has access to up to 6 vCPUs.

However, there’s an important consideration that many developers discover: simply allocating more memory may not automatically make your function faster. If your code runs sequentially, it will only use one vCPU regardless of how many are available. The remaining vCPUs sit idle while you’re still paying for the full memory allocation.

To help benefit from Lambda’s multi-core capabilities, your code should explicitly implement concurrent processing through multi-threading or parallel execution. Without this, you’re paying for compute power you’re not using.

Rust provides excellent support for this pattern. The AWS Lambda Rust Runtime provides developers with a language that combines exceptional performance with built-in concurrency primitives. In this post, we show you how to implement multi-threading in Rust to achieve 4-6x performance improvements for CPU-intensive workloads.

Our Test Workload: Why Bcrypt Password Hashing?

For this analysis, we use bcrypt password hashing as our CPU-intensive workload to evaluate multi-core scaling behavior. This choice is deliberate for several reasons:

  1. Real-world relevance: Bcrypt is commonly used in authentication systems, making our benchmarks practically relevant rather than synthetic.
  2. Predictable CPU work: Bcrypt with cost factor 10 provides approximately 100ms of pure CPU work per operation on typical hardware, creating a consistent and measurable baseline.
  3. Embarrassingly parallel: Each hash operation is completely independent, making it an ideal candidate for parallel processing without shared state or lock contention.
  4. CPU-bound: Bcrypt is deterministic and CPU-bound (not memory or I/O bound), isolating the performance characteristics we want to measure.

Throughout this post, we process batches of passwords and measure how multi-threading improves throughput as we scale from 1 to 6 vCPUs.

Understanding Lambda’s vCPU Allocation

AWS Lambda allocates CPU resources proportionally to the configured memory. According to AWS Lambda function memory documentation, at 1,769 MB a function has the equivalent of one vCPU.

vCPU Allocation by Memory:

Memory (MB)

 Approximate vCPUs
128 – 1,769 ~1
1,770 – 3,538 ~2
3,539 – 5,307 ~3
5,308 – 7,076 ~4
7,077 – 8,845 ~5
8,846 – 10,240

~6

Note: The num_cpus crate returns the number of logical CPUs visible to the Lambda environment, which may differ from the allocated vCPU share. At lower memory configurations, you may see 2 CPUs reported even though only 1 vCPU worth of compute time is allocated.

Solution Overview

The solution consists of a Rust Lambda function that:

  1. Receives a request specifying the number of items to process
  2. Detects available vCPUs and configures a thread pool accordingly
  3. Processes items in parallel using the Rayon library (a data parallelism library that allows you to convert sequential iterators into parallel ones with a .par_iter() call)
  4. Returns performance metrics including duration and throughput

Architecture Diagram: Lambda receives request, initializes Rayon thread pool based on WORKER_COUNT environment variable, processes bcrypt hashes in parallel across multiple vCPUs, and returns results.

Creating a Multi-threaded Rust Lambda Function

Create a new Lambda project using Cargo Lambda:

cargo lambda new rust-multithread-demo
cd rust-multithread-demo

Dependencies

Update Cargo.toml with the necessary dependencies:

[package]
name = "rust-multithread-lambda"
version = "0.1.0"
edition = "2021"

[dependencies]
lambda_runtime = "1.0.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
bcrypt = "0.15"
rayon = "1.7"
num_cpus = "1.16"

[profile.release]
opt-level = 3
lto = true
codegen-units = 1
strip = true

The optimization flags in [profile.release] reduce binary size and improve performance:

  • opt-level = 3: Maximum optimization
  • lto = true: Link-time optimization for smaller binaries
  • strip = true: Remove debug symbols

Implementing the Lambda Entry Point

First, let’s look at how we initialize the thread pool during cold start:

src/main.rs:

use lambda_runtime::{run, service_fn, Error, LambdaEvent};
mod handler;
use handler::{function_handler, get_worker_count, init_thread_pool, ProcessRequest};

#[tokio::main]
async fn main() -> Result<(), Error> {
    // Initialize Rayon thread pool at cold start (once per container lifecycle)
    init_thread_pool(get_worker_count());

    run(service_fn(|event: LambdaEvent<ProcessRequest>| async move {
        function_handler(event.payload).await
    }))
    .await
}

Why initialize in main() and not in the handler?

  1. Deterministic Configuration: The thread pool is configured once per container, before any requests arrive. This prevents race conditions if multiple requests try to initialize concurrently.
  2. Container Reuse: Lambda containers can serve multiple requests. Initializing in main() ensures the configuration is set during the cold start and persists for all subsequent warm invocations.
  3. Performance: Thread pool setup happens during cold start (already counted as initialization time), not during request processing.

Implementing the Request Handler

src/handler.rs:

use serde::{Deserialize, Serialize};
use std::env;
use std::sync::Once;
use std::time::Instant;
use std::collections::HashSet;
use std::sync::Mutex;
use rayon::prelude::*;

static INIT: Once = Once::new();

#[derive(Deserialize)]
pub struct ProcessRequest {
    count: usize,
    mode: String,
}

#[derive(Serialize)]
pub struct ProcessResponse {
    processed: usize,
    duration_ms: u128,
    mode: String,
    workers: usize,
    detected_cpus: usize,
    avg_ms_per_item: f64,
    memory_used_kb: u64,
    threads_used: usize, // Actual threads that processed items (proves multi-threading)
}

// CPU-intensive bcrypt hashing with cost factor 10
fn hash_password(password: &str) -> Result<String, bcrypt::BcryptError> {
    bcrypt::hash(password, 10)
}

// Process items one at a time (baseline for comparison)
fn process_sequential(items: Vec<String>) -> Result<(Vec<String>, usize), Box<dyn std::error::Error + Send + Sync>> {
    let results: Result<Vec<String>, _> = items
        .iter()
        .map(|item| hash_password(item))
        .collect();
    results
        .map(|r| (r, 1))
        .map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)
}

// Process items in parallel using Rayon's work-stealing scheduler
// Thread pool size is configured once at cold start via init_thread_pool()
fn process_parallel(items: Vec<String>) -> Result<(Vec<String>, usize), Box<dyn std::error::Error + Send + Sync>> {
    let thread_ids: Mutex<HashSet<std::thread::ThreadId>> = Mutex::new(HashSet::new());

    let results: Result<Vec<String>, _> = items
        .par_iter()
        .map(|item| {
            thread_ids.lock().unwrap().insert(std::thread::current().id());
            hash_password(item)
        })
        .collect();

    let threads_used = thread_ids.lock().unwrap().len();
    results
        .map(|r| (r, threads_used))
        .map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)
}

// Get worker count from env var or detect CPUs, clamped to 1-6
pub fn get_worker_count() -> usize {
    if let Ok(count_str) = env::var("WORKER_COUNT") {
        if let Ok(count) = count_str.parse::<usize>() {
            return count.clamp(1, 6);
        }
    }
    num_cpus::get().clamp(1, 6)
}

// Initialize Rayon global thread pool (only once per Lambda container)
pub fn init_thread_pool(workers: usize) {
    INIT.call_once(|| {
        let _ = rayon::ThreadPoolBuilder::new()
            .num_threads(workers)
            .build_global();
    });
}

// Read RSS memory from /proc/self/statm (Linux only)
fn get_memory_usage_kb() -> u64 {
    std::fs::read_to_string("/proc/self/statm")
        .ok()
        .and_then(|s| s.split_whitespace().nth(1)?.parse::<u64>().ok())
        .map(|pages| pages * 4)
        .unwrap_or(0)
}

// Main Lambda handler - processes items sequentially or in parallel
pub async fn function_handler(request: ProcessRequest) -> Result<ProcessResponse, Box<dyn std::error::Error + Send + Sync>> {
    if request.count == 0 { return Err("count must be greater than 0".into()); }
    if request.count > 1000 { return Err("count exceeds maximum of 1000 items".into()); }

    let items: Vec<String> = (0..request.count)
        .map(|i| format!("password_{:06}", i))
        .collect();

    let workers = get_worker_count();
    let mode = match request.mode.as_str() {
        "sequential" => "sequential",
        "parallel"   => "parallel",
        _            => if workers > 1 { "parallel" } else { "sequential" },
    };

    let start = Instant::now();
    let (results, threads_used) = match mode {
        "sequential" => process_sequential(items)?,
        _            => process_parallel(items)?,
    };
    let duration_ms = start.elapsed().as_millis();

    Ok(ProcessResponse {
        processed: results.len(),
        duration_ms,
        mode: mode.to_string(),
        workers: if mode == "parallel" { workers } else { 1 },
        detected_cpus: num_cpus::get(),
        avg_ms_per_item: duration_ms as f64 / request.count as f64,
        memory_used_kb: get_memory_usage_kb(),
        threads_used,
    })
}

Key Implementation Details

Thread Pool Initialization at Cold Start: The code initializes the thread pool in main() before the Lambda runtime starts, not during request processing. This approach is designed to eliminate race conditions and provide deterministic behavior across all invocations.

Important Note: Lambda initializes the thread pool once per container. The thread pool configuration retains its original value even if you change the WORKER_COUNT environment variable between invocations within the same container. For production deployments, keep WORKER_COUNT consistent for the function’s lifecycle.

Input Validation: The handler validates that count is between 1 and 1000 to prevent resource exhaustion.

Thread Tracking: The threads_used field proves multi-threading is working by tracking unique thread IDs during parallel processing. This provides empirical validation that work is distributed across multiple threads.

Memory Tracking: The memory_used_kb field reports RSS memory usage by reading /proc/self/statm, providing visibility into actual memory consumption.

Mode Selection: The function supports three modes:

  • sequential: Single-threaded processing
  • parallel: Multi-threaded processing using Rayon
  • auto: Automatically selects based on available workers

Building and Deploying

With the implementation complete, let’s compile the function for Lambda’s environment and deploy it to AWS.

# Build for ARM64 (Graviton2) - recommended for cost efficiency
cargo lambda build --release --arm64

# Or build for x86_64
cargo lambda build --release --x86-64

The build process produces a binary of approximately 1.7 MB (uncompressed) or 0.8 MB (zipped).

Deploy to AWS

Use Cargo Lambda to deploy the function with your desired memory configuration and worker count.

# Deploy with 6144 MB memory (4 vCPUs) and 4 workers
cargo lambda deploy rust-multithread-lambda \
    --memory 6144 \
    --timeout 30 \
    --env-var WORKER_COUNT=4

Note: To test different configurations, repeat the build and deploy commands with different --memory values and WORKER_COUNT settings for each configuration you want to benchmark. For comprehensive testing across architectures, build with --arm64, deploy all memory configurations, then rebuild with --x86-64 and deploy again.

Required IAM Permissions

The Lambda execution role needs the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

Test the Function

After deployment, verify the function works correctly by invoking it with a test payload.

aws lambda invoke \
    --function-name rust-multithread-lambda \
    --payload '{"count":20,"mode":"parallel"}' \
    --cli-binary-format raw-in-base64-out \
    response.json

Performance Benchmarks

We tested multiple configurations on ARM64 (Graviton2) to measure the impact of multi-threading.

Test workload: Processing 20 bcrypt password hashes (cost factor 10)

Note: Benchmark results may vary between runs due to factors such as Lambda placement, underlying hardware differences, and AWS infrastructure conditions. The numbers presented here are representative of typical performance observed across multiple test runs.

Performance Results: ARM64 (Graviton2)

Memory vCPUs Workers Avg (ms) P50 (ms) P95 (ms) P99 (ms) Min Max Speedup
1536 MB ~1 1 1,885 1,882 1,898 1,898 1,877 1,907 1.00x
2048 MB ~2 2 1,334 1,331 1,341 1,341 1,324 1,356 1.41x
4096 MB ~3 3 685 683 699 699 669 704 2.75x
6144 MB ~4 4 463 464 467 467 453 469 4.07x
8192 MB ~5 5 338 343 345 345 325 346 5.57x
10240 MB ~6 6 280 278 292 292 271 293 6.73x

Performance Results: x86_64

Memory vCPUs Workers Avg (ms) P50 (ms) P95 (ms) P99 (ms) Min Max Speedup
1536 MB ~1 1 1,671 1,675 1,681 1,681 1,659 1,684 1.00x
2048 MB ~2 2 1,253 1,249 1,265 1,265 1,241 1,294 1.33x
4096 MB ~3 3 892 891 899 899 888 900 1.87x
6144 MB ~4 4 429 425 443 443 417 449 3.89x
8192 MB ~5 5 330 323 349 349 317 358 5.06x
10240 MB ~6 6 292 292 298 298 291 298 5.72x

Architecture Comparison

Memory Workers ARM64 Avg x86_64 Avg Diff % Faster Arch
1536 MB 1 1,885 ms 1,671 ms -12.8% x86_64
2048 MB 2 1,334 ms 1,253 ms -6.4% x86_64
4096 MB 3 685 ms 892 ms +23.2% ARM64
6144 MB 4 463 ms 429 ms -7.9% x86_64
8192 MB 5 338 ms 330 ms -2.4% x86_64
10240 MB 6 280 ms 292 ms +4.1% ARM64

Key Observations

Cold Start Performance: Rust’s cold start initialization times are consistently between 19-28 ms across all memory configurations and architectures. ARM64 (Graviton2) shows slightly faster cold starts (19-23 ms) compared to x86_64 (26-29 ms). Both are significantly faster than interpreted runtimes because the binary is pre-compiled.

Near-Linear Scaling: Both architectures achieve impressive speedups:

  • ARM64: 6.73x speedup with 6 workers (exceeds theoretical 6x!)
  • x86_64: 5.72x speedup with 6 workers

Latency Consistency: The P95 and P99 metrics show excellent consistency:

  • ARM64 at 6 vCPUs: P50=278ms, P95=292ms, P99=292ms (low variance)
  • x86_64 at 6 vCPUs: P50=292ms, P95=298ms, P99=298ms

Both architectures show consistent latency at maximum parallelization.

Cost Analysis

Let’s analyze the cost implications of different configurations for processing 20 bcrypt hashes.

Cost Comparison: ARM64 vs x86_64 (us-east-1, as of January 2026):

Config Memory Workers ARM64 Duration ARM64 Cost/1M x86_64 Duration x86_64 Cost/1M Cheaper Arch
1 vCPU 1536 MB 1 1,885 ms $38.60 1,671 ms $42.78 ARM64
2 vCPU 2048 MB 2 1,334 ms $36.46 1,253 ms $42.77 ARM64 *
3 vCPU 4096 MB 3 685 ms $37.47 892 ms $60.80 ARM64
4 vCPU 6144 MB 4 463 ms $37.97 429 ms $44.00 ARM64
5 vCPU 8192 MB 5 338 ms $36.94 330 ms $45.10 ARM64
6 vCPU 10240 MB 6 280 ms $38.27 292 ms $49.87 ARM64
*Cheaper Arch

Cost Formulas:

  • ARM64: (Memory in GB) × (Duration in seconds) × $0.0000133334
  • x86_64: (Memory in GB) × (Duration in seconds) × $0.0000166667 (25% higher rate)

Key Insight: The 2 vCPU ARM64 configuration provides the lowest cost at $36.46 per million invocations while achieving 1.41x speedup. All ARM64 configurations remain cost-competitive ($36-$39 range) despite significant performance differences, demonstrating how increased throughput can offset higher memory costs.

Choosing the Right Configuration:

Priority Recommended Config Rationale
Lowest Cost ARM64, 2048 MB, 2 workers $36.46/1M invocations, 1.41x speedup
Balanced ARM64, 4096 MB, 3 workers $37.47/1M invocations, 2.75x speedup
Low Latency ARM64, 10240 MB, 6 workers 280ms avg, 6.73x speedup

When to Use Multi-threaded Rust on Lambda

Recommended Use Cases

  • Batch data processing: Transform, validate, or enrich large datasets
  • Cryptographic operations: Hashing, encryption, digital signatures
  • Image/video processing: Resize, transcode, analyze media files
  • Scientific computing: Simulations, data analysis, machine learning inference
  • High-volume workloads: Functions invoked >100,000 times per day benefit from optimization

When to Consider Alternatives

  • I/O-bound operations: Use async Rust instead of multi-threading for database queries or API calls
  • Simple transformations: Functions completing in <100ms rarely benefit from parallelization
  • Low-volume workloads: Development overhead may not be justified for <10,000 invocations per day
  • Rapid prototyping: Python or Node.js may be more appropriate when iteration speed is critical

Cleanup

To delete the resources created in this post:

# Delete the Lambda function
aws lambda delete-function --function-name rust-multithread-lambda

# Delete the CloudWatch log group
aws logs delete-log-group --log-group-name /aws/lambda/rust-multithread-lambda

Note: If you deployed multiple configurations for testing, you’ll need to delete each function individually by repeating the delete command with each function name, or use the SAM template for bulk cleanup:

aws cloudformation delete-stack --stack-name rust-multithread-benchmark

Conclusion

When you allocate more memory to your Lambda function, AWS provides proportionally more vCPUs—up to 6 vCPUs at 10,240 MB. However, sequential code only uses one vCPU, leaving the additional compute power idle while you pay for the full allocation. Multi-threaded Rust with Rayon enables you to harness all available vCPUs for CPU-intensive workloads, transforming unused capacity into real performance gains.

Our benchmarks demonstrate this clearly:

  • Near-linear scaling: ARM64 achieved 6.73x speedup with 6 workers—you get proportional returns on your vCPU investment
  • Fast cold starts: 19-28 ms initialization across all configurations, eliminating the cold start concerns often associated with compiled languages
  • Consistent latency: ARM64 at 6 vCPUs shows only 1ms variance between P50 and P99, critical for predictable response times
  • Cost efficiency: ARM64 is 15-20% cheaper than x86_64 with better scaling at maximum parallelization

The key takeaway: If your Lambda function performs CPU-intensive work and you’re allocating more than 1,769 MB of memory, you likely have multiple vCPUs available. Without multi-threading, those vCPUs sit idle. Rayon’s parallel iterators allow you to switch from sequential to parallel processing by changing .iter() to .par_iter() in your code.

Recommended starting point: ARM64 with 4096 MB (3 workers) offers an excellent balance of cost and performance for most workloads. Scale up to 6 vCPUs for latency-critical applications, or down to 2 vCPUs for maximum cost savings.

Additional Resources

The complete sample code, SAM template, and test scripts from this post are available at Github Repository.

Building fault-tolerant applications with AWS Lambda durable functions

Post Syndicated from Rahul Pisal original https://aws.amazon.com/blogs/compute/building-fault-tolerant-long-running-application-with-aws-lambda-durable-functions/

Business applications often coordinate multiple steps that need to run reliably or wait for extended periods, such as customer onboarding, payment processing, or orchestrating large language model inference. These critical processes require completion despite temporary disruptions or system failures. Developers currently spend significant time implementing mechanisms to track progress, handle failures, and manage resources when waiting for external events, shifting focus from business logic to undifferentiated tasks.

At re:Invent 2025, Amazon Web Services (AWS) launched AWS Lambda durable functions, a new capability extending Lambda’s event-driven programming model with built-in capabilities to build fault-tolerant multi-step applications and AI workflows using familiar programming languages. At its core, durable functions are regular Lambda functions, so your development and operational processes for Lambda continue to apply. However, when you create a Lambda function you can now enable durable execution, so that you can checkpoint progress, automatically recover from failures, and suspend execution for up to one year when waiting on long-running tasks, such as human-in-the-loop processes.

How Lambda durable functions work

When working with standard Lambda functions, your code runs from start to finish in a single invocation. If a failure occurs at any point during the execution, the entire function must be retried by the invoking event source. Any state that needs to be preserved between executions must be explicitly saved and retrieved. This is typically done by using external storage services such as Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3). Furthermore, you must typically guard against duplicate (concurrent) invocations of the same event and have a strategy to safely deploy updates while continuing to process events.

In contrast, with Lambda durable functions, developers use durable operations such as “Steps” and “Waits” in the event handler to checkpoint progress, handle failures, and suspend execution during wait periods without incurring compute charges for on-demand functions. These durable operations and any optional state returned from them are automatically persisted by Lambda in a fully-managed durable execution backend. If failures occur during the execution, or if your function resumes its execution after being paused, Lambda invokes your function again, restoring (replaying) the previous state by executing the event handler from the start, but skipping over completed durable operations. To streamline this checkpoint/replay mechanism for developers, you can use the Lambda durable execution SDK to wrap or annotate your event handler, which enhances the existing Lambda context with several new methods like context.step() and context.wait(). Furthermore, you can use methods such as context.waitForCallback() to wait on external jobs or asynchronous processes, such as “human-in-the-loop” scenarios. The execution is paused until a SendDurableExecutionCallbackSuccess or SendDurableExecutionCallbackFailure response is sent to the Lambda API.

Getting started

Use the AWS Serverless Application Model (AWS SAM) to create a new durable function with sam init with an AWS Quick Start Template. Lambda durable functions are also supported by the AWS Cloud Development Kit (AWS CDK), AWS Command Line Interface (AWS CLI), AWS CloudFormation and other infrastructure as code (IaC) frameworks such as Terraform.

Consider the following function, which performs user onboarding. First, it creates a user profile based on some data, then it sends out an email for verification and waits until the user either confirms the email address, or a 24-hour timeout is reached. Finally, it sends out a confirmation.

import {
  DurableContext,
  withDurableExecution,
} from '@aws/durable-execution-sdk-js';
export const handler = withDurableExecution(
  async (event: OnboardingEvent, context: DurableContext) => {
    try {    
      // Create user profile
      const profile = await context.step("create-profile", async () =>
        createUserProfile(event.email, event.name)
      );
      // Wait for email verification via callback
      const verification = await context.waitForCallback(
        "wait-for-email-verification",
        async (callbackId) => {
          // Send email to user and pass callbackId
          await sendVerificationEmail(profile, callbackId);
        },
        {
          timeout: { hours: 24 } 
        }
      );
      // Send confirmation and welcome email
      const result = await context.step("complete-onboarding", async () => {
        if (!verification || !verification.verified) 
     return { ...profile, status: 'failed' };
        await sendWelcomeEmail(profile.email, profile.name);
        return { ...profile, status: 'active' };
      });
      return result;
    } catch (error) {
      // omitted 
    }
  }
);

Durable functions have built-in and fully customizable error handling for steps. For example, if the profile was successfully created and verified, but a temporary error occurred when sending out the confirmation, then the step is retried. The retry skips over any previously completed checkpoints, such as the profile creation and callback. Only the code within the send confirmation step is run again.

Next, you update the AWS SAM template to include your durable function. You create a Lambda durable function by including the DurableConfig setting for your function. Note that you currently cannot add a durable configuration to a function that was originally created without it. The ExecutionTimeout defines after which time the durable execution times out to protect against runaway or deadlock application bugs. This setting is separate from the invocation timeout, which defines for how long a single invocation can run. The maximum invocation timeout for a single function invocations remains unchanged at 15 minutes. With Lambda durable functions, you will typically see multiple invocations per durable execution, such as when using the wait capabilities in the SDK or automatic retries. You can set the ExecutionTimeout for up to one year when using asynchronous invocations.

The RetentionPeriodInDays defines how long the execution data of a durable execution is available to you after executions complete.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
 
Resources:
  UserOnboardingFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: UserOnboardingFunction
      CodeUri: ./src
      Handler: index.handler
      Runtime: nodejs24.x
      Architectures:
        - x86_64
      MemorySize: 256
      Timeout: 60		   // Timeout for an individual invocation
      DurableConfig:		   // This makes the function a durable function
        ExecutionTimeout: 90000 // 25h timeout for the durable execution overall
        RetentionPeriodInDays: 7 
UserOnboardingFunctionRole:
    Type: AWS::IAM::Role
    // omitted for brevity

You must include the necessary permissions for your function. For example, the AWSLambdaBasicDurableExecutionRole managed policy only allows the minimal AWS Identity and Access Management (IAM) actions to create/retrieve checkpoints and logs to increase security. Therefore, it does not include permissions to invoke other (durable) functions or manage callbacks. Refer to the documentation for more details.

Testing locally

Before deploying your function, you can test it locally using AWS SAM local invoke.

AWS SAM locally invokes your function and runs the event handler until it reaches the context.waitForCallback(). To complete callbacks, AWS SAM offers new commands to interact with your durable functions. In this example, you send a Success response to complete the callback. You can also include relevant data in the response. You can send the response directly using the on-screen guide or using another AWS SAM CLI command from another process.

sam local callback succeed <your-callback-id> --result '<your data>'

To inspect an execution, you can use AWS SAM to retrieve the durable execution history of your function, which includes details about steps, callbacks, and wait durations, as shown in the following example code.

sam local execution history <execution-arn>

Depending on your use case, you can instead send a Failure response to a callback and handle those errors in your code. For example, by performing compensation logic in a subsequent step:

sam local callback fail <your-callback-id> --result '<your data>'

Now that you have verified that your function works as intended, deploy it to AWS using sam deploy command.

Best practices and considerations

Invoking a Lambda durable function requires a qualified Amazon Resource Name (ARN), such as an alias or version. We recommend that you don’t use the $LATEST qualifier except for rapid prototyping or local testing. Using explicit versions ensures that replays always happen with the same code with which the execution was started. This is to ensure deterministic execution and prevent inconsistencies when updating your function code during executions.

We recommend bundling the durable execution SDK with your function code using your preferred package manager. The SDKs are fast-moving, so you can update dependencies as new features become available.

There are other durable operations in the Lambda durable functions SDK that you can use to build your application:

  • waitForCondition(): Pauses the execution of your function until a condition is met. For example, the status of a job polled with an API. For this to work, you provide the waitStrategy and a check function to poll the status.
  • parallel(): Runs multiple durable operations in parallel within the same function, with configurable options such as the maximum number of concurrent branches and desired failure behavior. This streamlines managing durability and checkpointing for simultaneous asynchronous actions.
  • map(): Creates a durable operation and checkpoint for each item of an array, based on the provided mapping function. The items are processed concurrently.
  • invoke(): Invokes another Lambda function and waits for its result. The SDK creates a checkpoint, invokes the target function, and resumes your function when the invocation completes. This enables function composition and workflow decomposition.

Refer to the developer guide for more details.

Lambda compute charges apply to all invocations, including any replays. When using wait operations, the function suspends execution and, for on-demand functions, doesn’t incur duration charges until execution resumes. You’re also charged for durable operations, data written, and data retention. To learn more about Lambda durable functions pricing, refer to the Lambda pricing page.

For the latest Region availability, visit the AWS Capabilities by Region page.

Conclusion

AWS Lambda durable functions extends the Lambda programming model to streamline building fault-tolerant, long-running applications using familiar programming patterns. You can use Lambda durable functions to write multi-step workflows in your preferred programming language, using built-in methods that automatically handle progress checkpointing and error recovery. This streamlines your architectures so that you can focus on your business logic, and optimize cost by charging only for active compute time.

You can build durable functions for Python or Node.js based Lambda functions using the Lambda API, AWS Management Console, AWS CLI, AWS CloudFormation, AWS SAM, AWS SDK, and AWS CDK.

To get started, visit the Lambda Developer Guide or watch the re:Invent breakout session.

Mastering millisecond latency and millions of events: The event-driven architecture behind the Amazon Key Suite

Post Syndicated from Ali Ufuk Yucel original https://aws.amazon.com/blogs/architecture/mastering-millisecond-latency-and-millions-of-events-the-event-driven-architecture-behind-the-amazon-key-suite/

Background

Amazon Key empowers customers to securely manage access to their homes and businesses through innovative solutions. Through a suite of consumer and business products, the Amazon Key team is transforming how customers receive deliveries and manage access to their spaces. Our In-Garage Delivery service offers a secure and convenient solution for receiving Amazon packages and groceries directly inside customers’ garages. For property managers and building owners, Amazon Key provides comprehensive access management solutions that enable safe and efficient delivery operations in apartment buildings and gated communities, enhancing both security and convenience for residents.

In this post, we explore how the Amazon Key team used Amazon EventBridge to modernize their architecture, transforming a tightly coupled monolithic system into a resilient, event-driven solution. We explore the technical challenges we faced, our implementation approach, and the architectural patterns that helped us achieve improved reliability and scalability. The post covers our solutions for managing event schemas at scale, handling multiple service integrations efficiently, and building an extensible architecture that accommodates future growth.

Opportunities

Service Coupling and System Fragility

Our legacy architecture faced significant challenges stemming from its tightly coupled design, where service interactions created a complex web of dependencies impacting system stability and scalability. Making service modifications was particularly challenging, as adding or removing services required careful consideration of numerous interdependencies. An incident highlighted this vulnerability when an issue in Service-A triggered a cascade of failures across many upstream services, with increased timeouts leading to retry attempts and ultimately resulting in service deadlocks. System fragility was further demonstrated when problems with a single device vendor, despite being responsible only for specific delivery operations, caused widespread degradation across multiple system services.

Loose Event Schemas

Our old event management infrastructure lacked explicit schema definitions and employed a loosely-typed data architecture, leading to several critical issues. Events were difficult to maintain as use cases expanded, and the absence of formal schema documentation impacted transparency and team collaboration. The design made it almost impossible to implement backward-incompatible changes, such as removing unused fields or events for performance optimization. Without a repository for schema management, team-to-team collaboration for schema modifications (adding fields, removing fields, deprecating fields, or marking fields as required) became challenging. The system also lacked organized validation logic, making it difficult for publishers to identify invalid events before they entered the system. Additionally, the loosely typed schemas lost important semantic context, such as inheritance and composition relationships between different event schemas.

Inconsistent Event Routing and Management

The event routing logic was manually managed and lacked the sophistication needed for growing use cases. The system only supported basic validation of events, primarily checking for required fields, with limited capability for extending validation rules or implementing more complex routing logic. Features that were commonly available in off-the-shelf solutions, such as parallel publishing to multiple subscribers, required significant custom development and ongoing maintenance effort. The implementation only supported a limited number of subscribers to the event pipeline, with no sustainable pathway for adding more consumers. While attempts were made to reduce coupling through SNS/SQS pairs between services, these solutions were implemented on an ad-hoc basis, lacking standardization and creating additional maintenance overhead. This approach led to redundant work and failed to abstract away common functionality, resulting in an inefficient and hard-to-maintain system.These challenges collectively highlighted the need for a more robust and flexible architectural approach that could better serve the system’s evolving needs while improving reliability, maintainability, and scalability.

Design

Given our requirements and the architectural challenges we faced, we implemented a single-bus, multi-account pattern to optimize our system architecture. In this design, each service team maintains complete ownership and autonomy over their application stack, enabling independent development and deployment cycles. Meanwhile, our DevOps team manages a centralized infrastructure stack that encompasses event bus rules, target configurations, and service integrations. This separation of concerns provides several key benefits:

  1. Clear ownership boundaries: Service teams can focus on their core business logic while leveraging a standardized event infrastructure.
  2. Centralized governance: The DevOps team facilitates consistent event routing patterns, security controls, and monitoring across service integrations.
  3. Simplified operations: A single event bus reduces operational complexity while maintaining logical separation through well-defined routing rules.
  4. Enhanced security: The multi-account structure provides natural isolation boundaries while still enabling controlled cross-account event flows.
  5. Streamlined compliance: Centralized management of data exchange patterns makes it easier to implement and maintain compliance requirements.

While EventBridge provided the foundation, we developed additional components to meet our specific requirements.  Our team built three key components: a schema repository serving as the single source of truth for event definitions, a client library that handles schema validation and provides developer-friendly abstractions, and an infrastructure library offering reusable components for subscriber integration.

Event Schema Repository

Amazon EventBridge’s schema discovery and documentation capabilities provide powerful solutions for managing event-driven architectures. The service automatically captures event structures in the schema registry, maintaining versions as events evolve over time. While EventBridge provides developers with tools to implement validation using external solutions or custom application code, it currently does not include native schema validation capabilities. For our organization’s large-scale event-driven architecture, schema validation was a critical requirement. We evaluated two implementation approaches: a centralized validation service or client-side validation at the publisher/subscriber level. The centralized approach would have required managing additional infrastructure, scaling considerations, and introduced latency through extra network hops. After analyzing these factors alongside our requirements for schema governance and team autonomy, we implemented a custom schema repository with client-side validation.

This architecture prioritizes developer experience through immediate validation feedback while maintaining our standards for schema versioning and release management. The repository serves as the foundation for our event-driven architecture, providing essential capabilities for data governance and quality control. By acting as the single source of truth for event definitions, it enables standardized validation across clients, enforces data quality checks, establishes clear ownership boundaries, and maintains comprehensive audit trails for schema changes. Publishers and subscribers leverage these schemas to maintain data consistency and compatibility as their services evolve. The repository has become instrumental in facilitating efficient cross-team collaboration through self-service schema discovery, documentation, and automated validation during development. It maintains a comprehensive registry of event publishers and their corresponding subscribers, providing clear visibility into event flow patterns and dependencies across the system. Teams can quickly manage schema evolution with clear deprecation policies and migration paths, while the system helps detect breaking changes early in the development cycle. This collaborative approach has significantly improved team velocity and reduced integration issues between services.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "$id": "/resource/event/schema/EventV1.json",
    "title": "EventV1",
    "description": "Schema for a simple event.",
    "type": "object",
    "properties": {
        "id": {
            "description": "Id of the event.",
            "type": "string"
        },
        "type": {
            "description": "Type of the event.",
            "$ref": "EventType.json"
        },
        "time": {
            "description": "Time at which the event occurred. It uses ISO 8601 Date Time Format. Reference: https://www.iso.org/iso-8601-date-and-time-format.html",
            "type": "string",
            "format": "date-time"
        },
        "publisher": {
            "description": "Publisher of the event.",
            "$ref": "../core/Publisher.json"
        }
    },
    "required": [
        "id",
        "type",
        "time",
        "publisher"
    ]
}

Client Library

The client library serves as a crucial component for both publishers and subscribers, streamlining their integration with the central event bus. At its core, the library leverages our Event Schema Repository, generating code bindings at build time to provide developers with type-safe and intuitive interfaces for event creation and handling. This approach significantly enhances developer productivity by offering straightforward and convenient methods to construct events and interact with the bus, reducing the likelihood of errors and improving code readability.

A key feature of the client library is its built-in validation mechanism. By utilizing the schemas from our local repository, the library performs thorough validation of events before they are published. This proactive approach catches potential issues early in the development cycle, making sure that only well-formed events conforming to the agreed-upon schemas make it to the event bus. Once validated, the library handles the serialization process and manages the actual publishing of events to the bus, abstracting and simplifying data transformation and transport.

For subscribers, the client library offers equally valuable functionality. It seamlessly handles the deserialization of incoming events, presenting them to the subscribing services in a readily usable format. This feature saves development time and reduces the risk of parsing errors, allowing teams to focus on business logic rather than data handling intricacies. By providing these comprehensive capabilities, our client library has become an indispensable tool in our event-driven network, promoting consistency, reliability, and efficiency across our microservices architecture.

Subscriber Constructs Library

We developed a subscriber constructs library using AWS Cloud Development Kit (CDK) to simplify and standardize the integration process with our central event bus. This library abstracts the setup and management of underlying infrastructure required for event consumption, enabling teams to focus on their core business logic rather than infrastructure configuration details.

The library automates the creation of essential components required for reliable event processing. It provisions a dedicated event bus within the subscriber’s account, establishes the necessary IAM roles and permissions for secure cross-account communication with the central event bus, and configures standardized monitoring and alerting for event processing. This automation not only reduces the potential for configuration errors but also facilitates consistent implementation of our architectural patterns across different teams.

/**
 * Subscriber implementation to provision necessary AWS infrastructure.
 *
 */
const subscription = new Subscription(scope, id, {
    name: "DeliveryService", // Name of your application
    application: {
       region: Region.US_EAST_1, // Region of your Application
    },
});

Conclusion

Amazon Key team’s journey to modernize their architecture and build a resilient, event-driven solution exemplifies the powerful benefits of leveraging AWS EventBridge and adopting a well-designed event-driven architecture. By addressing the challenges of service coupling, loose event schemas, and inconsistent event routing, the team was able to transform their system into a more reliable, scalable, and maintainable resource. The key architectural patterns and components they implemented have had a significant impact on their ability to deliver innovative solutions to their customers.

Reliability and Scale:

  • Built a decoupled event system processing 2000 events/second with 99.99% success rate
  • Achieved consistent 80ms p90 latency from ingestion to target invocation across 14M subscriber calls
  • Avoided the need for new infrastructure for event exchange through standardized event routing
  • Enabled migration of existing complex interdependencies to event-driven architecture

Developer Experience:

  • Reduced service integration time for new use cases from five days to one day (80% improvement)
  • New event onboarding on the Custom Event Schema repository now takes four hours, down from 48 hours
  • Publisher/subscriber integration completed in eight hours, previously took 40 hours
  • Standardized client library addressed 90% of common integration errors

Security and Governance :

  • Single control plane manages 100% of event bus infrastructure
  • Automated security compliance checks catch 100% of unauthorized data exchange patterns
  • Real-time monitoring dashboard tracks every event flow and schema change
  • Schema repository provides complete audit trail for system modifications

The solutions developed by the Amazon Key team provide a blueprint for other organizations looking to modernize their architectures and leverage the power of event-driven design patterns. By adopting similar architectural patterns and components, such as the schema repository and client libraries, other organizations can be empowered to achieve similar benefits.

About the authors

AWS Weekly Roundup: Amazon Bedrock agent workflows, Amazon SageMaker private connectivity, and more (February 2, 2026)

Post Syndicated from Betty Zheng (郑予彬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-bedrock-agent-workflows-amazon-sagemaker-private-connectivity-and-more-february-2-2026/

Over the past week, we passed Laba festival, a traditional marker in the Chinese calendar that signals the final stretch leading up to the Lunar New Year. For many in China, it’s a moment associated with reflection and preparation, wrapping up what the year has carried, and turning attention toward what lies ahead.

Looking forward, next week also brings Lichun, the beginning of spring and the first of the 24 solar terms. In Chinese tradition, spring is often seen as the season when growth begins and new cycles take shape. There’s a common saying that “a year’s plans begin in spring,” capturing the idea that this is a time to set one’s direction and start fresh.

Last week’s launches
Here are the launches that got my attention this week:

  • Amazon Bedrock enhances support for agent workflows with server-side tools and extended prompt caching – Amazon Bedrock introduced two updates that improve how developers build and operate AI agents. The Responses API now supports server-side tool use, so agents can perform actions such as web search, code execution, and database updates within AWS security boundaries. Bedrock also adds a 1-hour time-to-live (TTL) option for prompt caching, which helps improve performance and reduce the cost for long-running, multi-turn agent workflows. Server-side tools are available with OpenAI GPT OSS 20B and 120B models, and the 1-hour prompt caching TTL is generally available for select Claude models by Anthropic in Amazon Bedrock.
  • Amazon SageMaker Unified Studio adds private VPC connectivity with AWS PrivateLink – Amazon SageMaker Unified Studio now supports AWS PrivateLink, providing private connectivity between your VPC and SageMaker Unified Studio without routing customer data over the public internet. With SageMaker service endpoints onboarded into a VPC, data traffic remains within the AWS network and is governed by IAM policies, supporting stricter security and compliance requirements.
  • Amazon S3 adds support for changing object encryption without data movement – Amazon S3 now supports changing the server-side encryption type of existing encrypted objects without moving or re-uploading data. Using the UpdateObjectEncryption API, you can switch from SSE-S3 to SSE-KMS, rotate customer -managed AWS Key Management Service (AWS KMS) keys, or standardize encryption across buckets at scale with S3 Batch Operations while preserving object properties and lifecycle eligibility.
  • Amazon Keyspaces introduces table pre-warming for predictable high-throughput workloads – Amazon Keyspaces (for Apache Cassandra) now supports table pre-warming, which helps you proactively set warm throughput levels so tables can handle high read and write traffic instantly without cold-start delays. Pre-warming helps reduce throttling during sudden traffic spikes, such as product launches or sales events, and works with both on-demand and provisioned capacity modes, including multi-Region tables. The feature supports consistent, low-latency performance while giving you more control over throughput readiness.
  • Amazon DynamoDB MRSC global tables integrate with AWS Fault Injection Service – Amazon DynamoDB multi-Region strong consistency (MRSC) global tables now integrate with AWS Fault Injection Service. With this integration, you can simulate Regional failures, test replication behavior, and validate application resiliency for strongly consistent, multi-Region workloads.

Additional updates
Here are some additional projects, blog posts, and news items that I found interesting:

  • Building zero-trust access across multi-account AWS environments with AWS Verified Access – This post walks through how to implement AWS Verified Access in a centralized, shared-services architecture. It shows how to integrate with AWS IAM Identity Center and AWS Resource Access Manager (AWS RAM) to apply zero trust access controls at the application layer and reduce operational overhead across multi-account AWS environments.
  • Amazon EventBridge increases event payload size to 1 MB – Amazon EventBridge now supports event payloads up to 1 MB, an increase from the previous 256 KB limit. This update helps event-driven architectures carry richer context in a single event, including complex JSON structures, telemetry data, and machine learning (ML) or generative AI outputs, without splitting payloads or relying on external storage.
  • AWS MCP Server adds deployment agent SOPs (preview) – AWS introduced deployment standard operating procedures (SOPs) that AI agents can deploy web applications to AWS from a single natural language prompt in MCP -compatible integrated development environments (IDEs) and command line interfaces (CLIs) such as Kiro, Cursor, and Claude Code. The agent generates AWS Cloud Development Kit (AWS CDK) infrastructure, deploys AWS CloudFormation stacks, and sets up continuous integration and continuous delivery (CI/CD) workflows following AWS best practices. The preview supports frameworks including React, Vue.js, Angular, and Next.js.
  • AWS Network Firewall adds generation AI traffic visibility with web category filtering – AWS Network Firewall now provides visibility into generative AI application traffic through predefined web categories. You can use these categories directly in firewall rules to govern access to generative AI tools and other web services. When combined with TLS inspection, category-based filtering can be applied at the full URL level.
  • AWS Lambda adds enhanced observability for Kafka event source mappings – AWS Lambda introduced enhanced observability for Kafka event source mappings, providing Amazon CloudWatch Logs and metrics to monitor event polling configuration, scaling behavior, and event processing state. The update improves visibility into Kafka-based Lambda workloads, helping teams diagnose configuration issues, permission errors, and function failures more efficiently. The capability supports both Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Apache Kafka event sources.
  • AWS CloudFormation 2025 year in review – This year-in-review post highlights CloudFormation updates delivered throughout 2025, with a focus on early validation, safer deployments, and improved developer workflows. It covers enhancements such as improved troubleshooting, drift-aware change sets, stack refactoring, StackSets updates, and new -IDE and AI -assisted tooling, including the CloudFormation language server and the Infrastructure as Code (IaC) MCP server.

Upcoming AWS events
Check your calendars so that you can sign up for this upcoming event:

AWS Community Day Romania (April 23–24, 2026) – This community-led AWS event brings together developers, architects, entrepreneurs, and students for more than 10 professional sessions delivered by AWS Heroes, Solutions Architects, and industry experts. Attendees can expect expert-led technical talks, insights from speakers with global conference experience, and opportunities to connect during dedicated networking breaks, all hosted at a premium venue designed to support collaboration and community engagement.

If you’re looking for more ways to stay connected beyond this event, join the AWS Builder Center to learn, build, and connect with builders in the AWS community.

Check back next Monday for another Weekly Roundup.

–betty

Serverless ICYMI Q4 2025

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/serverless-icymi-q4-2025/

Stay current with the latest serverless innovations that can transform your applications. In this 31st quarterly recap, discover the most impactful AWS serverless launches, features, and resources from Q4 2025 that you might have missed.

In case you missed our last ICYMI, check out what happened in Q3 2025.

2025 Q4 calendar

2025 Q4 calendar

Serverless at re:Invent 2025

This post covers the biggest serverless announcements from re:Invent 2025, highlighting key feature updates that can improve your applications, and shares valuable resources to keep you informed.

AWS re:Invent 2025 had more than 60,000 in-person attendees and more than 2 million online viewers for the keynotes. The event featured 3,500 sessions from 3,000 speakers, which included information on 530 AWS service and feature announcements.

Keynote Igniting the serverless movement

Keynote Igniting the serverless movement

The serverless content consisted of two tracks: Containers and Serverless (CNS) and Application Integration (API). These tracks included 150 unique sessions watched in-person by more than 16,000 attendees. There were developer-focused experiences including a Road to re:Invent Hackathon, AWS Builder Loft, and Builders Arena. Serverlesspresso, the coffee shop powered by serverless technology, operated in two locations during the event: the Expo Hall and the certification lounge.

Serverless and developer community photo

Serverless and developer community photo

Find a curated list of serverless videos on Serverless Land YouTube.

AWS Lambda durable functions

Managing state across multi-step serverless workflows has traditionally required complex external orchestration tools. AWS Lambda durable functions expand how developers can use Lambda. You can now build reliable multi-step applications and AI workflows directly within Lambda.

AWS Lambda durable functions code

AWS Lambda durable functions code

Durable functions automatically checkpoint progress by saving the current state and completed steps at key points during execution. This allows them to suspend execution for up to one year during long-running tasks and recover from failures by resuming from the last checkpoint rather than restarting from the beginning, all without requiring additional infrastructure management.

Developers can now build in Python or TypeScript, wrap calls in steps with automatic retries and checkpointing. You can use waits to suspend execution for minutes, hours, or even up to a year without paying for idle compute. Durable functions use a replay mechanism to maintain state and handle failures gracefully. The replay mechanism works by re-executing your function code from checkpoints when recovering from failures, ensuring state consistency without data loss. This also means you don’t need complex external orchestration tools for many use cases. This can be helpful for AI workflows and multi-step applications where you need reliable state management without managing external infrastructure.

For more information, read the launch blog post and watch the re:Invent Breakout Session video: Deep Dive on AWS Lambda durable functions (CNS380)

AWS Lambda Managed Instances

Lambda now offers Lambda Managed Instances, a new compute option that combines Amazon EC2 flexibility with fully managed infrastructure. AWS automatically handles instance provisioning, scaling, and maintenance while allowing access to the full range of EC2 capabilities, including Graviton4, network-optimized instances, and other specialized compute options.

AWS Lambda Managed Instances configuration

AWS Lambda Managed Instances configuration

Your functions run on dedicated EC2 capacity from your account, in your own Amazon Virtual Private Cloud (Amazon VPC). AWS still manages the operational overhead, including OS patching, load balancing, and auto-scaling. This gives you access to specialized hardware options while maintaining the serverless operational model. You can further improve costs by using EC2 pricing models, including Compute Savings Plans and Reserved Instances for Lambda workloads. Each instance can handle multiple concurrent requests, making this particularly valuable for high-volume, steady-state workloads where predictable pricing and specific hardware requirements matter.

For more information, read the launch blog post and watch the re:Invent Breakout Session video: Lambda Managed Instances: EC2 Power with Serverless Simplicity (CNS382).

Other Lambda announcements

Multi-tenant SaaS applications face challenges like data leakage between tenants and noisy neighbor effects where one tenant’s workload impacts others. They also struggle with implementing custom isolation mechanisms. Tenant isolation mode addresses these by processing function invocations in separate execution environments for each tenant. This manages tenant-level compute environment isolation automatically.

AWS Lambda tenant isolation

AWS Lambda tenant isolation

Lambda adds Provisioned Mode for Amazon SQS event-source mappings, providing predictable performance and reduced cold starts for high-throughput SQS processing workloads.

You can now send up to 1 MB of data in asynchronous Lambda invocations, increased from 256 KB, helping you build more complex data processing scenarios.

Lambda functions now support IPv6 networking, so you don’t need NAT Gateways when accessing the internet or other AWS services from VPC-connected functions.

Lambda internet connectivity through a NAT Gateway (IPv4) and Lambda internet connectivity through an egress-only internet gateway (IPv6).

Lambda internet connectivity through a NAT Gateway (IPv4) and Lambda internet connectivity through an egress-only internet gateway (IPv6).

Lambda Rust support is now generally available, moving from experimental status. This is backed by AWS Support and the Lambda availability SLA.

Lambda has expanded its runtime support by adding Python 3.14, Node.js 24, and Java 25 as both managed runtimes and container base images, providing access to the latest language features and ensuring long-term support.

Amazon ECS

Amazon Elastic Container Service (Amazon ECS) Express Mode streamlines the deployment and management of containerized applications by automating the infrastructure setup that traditionally slows down developers.

Amazon ECS Express Mode deployment

Amazon ECS Express Mode deployment

This means you can focus on building applications while deploying with confidence using AWS best practices. Express Mode lets you deploy production-ready containerized web applications and APIs with a single command. This automatically handles domains, networking, load balancing, AWS Identity and Access Management (IAM) roles, and auto-scaling through simplified APIs. When your applications evolve and require advanced features, you can seamlessly configure and access the full capabilities of the resources, including Amazon ECS. Learn more from the launch blog post.

Amazon ECS announced a public preview of a fully managed MCP server, enabling AI-powered experiences for development and operations. The Model Context Protocol (MCP) server provides enterprise-grade capabilities like automatic updates and patching, centralized security through AWS IAM integration, comprehensive audit logging via AWS CloudTrail, and the proven scalability, reliability, and support of AWS.

Amazon Elastic Container Registry (ECR) managed container image signing enhances your security posture and eliminates the operational overhead of setting up signing. Container image signing allows you to verify that images are from trusted sources. ECR automatically signs images as they are pushed using the identity of the entity pushing the image. Signing operations are logged through CloudTrail for full auditability.

Amazon API Gateway

Amazon API Gateway allows you to improve the responsiveness of your REST APIs by progressively streaming response payloads back to the client. With this new capability, you can use streamed responses to enhance user experience when building LLM-driven applications (such as AI agents and chatbots), improve time-to-first-byte (TTFB) performance for web and mobile applications, stream large files, and perform long-running operations while reporting incremental progress using protocols such as server-sent events (SSE).

Amazon API Gateway streaming

API Gateway introduces private integration with Application Load Balancers (ALBs). You can use this to expose your VPC-based applications securely through REST APIs without exposing your ALBs to the public internet.

You can also now configure enhanced TLS security policies on API endpoints and custom domain names, providing you with greater control over the security posture of your APIs.

Amazon EventBridge

Amazon EventBridge introduced an enhanced visual rule builder that helps developers discover and subscribe to events from custom applications and over 200 AWS services. The console-based interface integrates the EventBridge schema registry with a comprehensive event catalog and intuitive drag-and-drop canvas that simplifies building event-driven applications. Developers can browse and search through events with readily available sample payloads and schemas without having to hunt through individual service documentation. The schema-aware visual builder guides developers through creating event filter patterns and rules, reducing syntax errors and accelerating development time.

EventBridge also allows targeting SQS fair queues.

AWS Step Functions

AWS Step Functions allows for enhanced local testing through the TestState API, providing programmatic access to comprehensive testing capabilities without deploying to AWS. This helps you build automated test suites that validate your workflow definitions locally on your development machines. Test error handling patterns, data transformations, and mock service integrations using your preferred testing frameworks.

There is also a new metrics dashboard, giving you visibility into your workflow operations at both the account and state machine levels.

Other announcements

Savings Plans flexible pricing model extends to AWS managed database services with the launch of Database Savings Plans. This helps reduce database costs by up to 35% when committing to a consistent amount of usage ($/hour) over a 1-year term. Savings automatically apply each hour to eligible usage across supported database services, and additional usage beyond the commitment is billed at on-demand rates.

Amazon DynamoDB now supports multi-attribute composite keys in global secondary indexes. You no longer need to concatenate values into synthetic keys manually, which sometimes results in the need to backfill data before adding new indexes. Instead, you can create primary keys using up to eight existing attributes, making it easier to model diverse access patterns and adapt to new query requirements.

Amazon Bedrock introduced AgentCore with quality evaluations and policy controls for deploying trusted AI agents at scale.

Bedrock also added 18 fully managed open weight models, expanding AI model options for developers.

The Strands Agents SDK is an open source framework that takes a model-driven approach to building and running AI agents in just a few lines of code. TypeScript support is now available in preview so you can choose between Python and TypeScript for building Strands Agents.

Amazon S3 Vectors became generally available. S3 Vectors delivers purpose-built, cost-optimized vector storage for AI agents, inference, Retrieval Augmented Generation (RAG), and semantic search at billion-vector scale.

Serverless blog posts

October

November

Serverless Office Hours

Join our livestream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. Episodes are available on-demand at serverlessland.com/office-hours.

October

November

December

Still looking for more?

The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Serverless Developer Advocacy team to see the latest news, follow conversations, and interact with the team.

And finally, visit Serverless Land for all your serverless needs.

More room to build: serverless services now support payloads up to 1 MB

Post Syndicated from Anton Aleksandrov original https://aws.amazon.com/blogs/compute/more-room-to-build-serverless-services-now-support-payloads-up-to-1-mb/

To support cloud applications that increasingly depend on rich contextual data, AWS has raised the maximum payload size from 256 KB to 1 MB for asynchronous AWS Lambda function invocations, Amazon Simple Queue Service (Amazon SQS), and Amazon EventBridge. Developers can use this enhancement to build and maintain context-rich event-driven systems and reduce the need for complex workarounds such as data chunking or external large object storage.

Overview

Modern cloud applications rely on context-rich, structured data to drive intelligent behavior. Large language model (LLM) prompts, telemetry signals, personalization data, machine learning (ML) outputs, and user interaction logs are no longer simple strings. Instead, they’re typically complex, nested JSON or YAML objects carrying meaningful context. Previously, developers working with serverless services such as Amazon SQS, Lambda (asynchronous invocations and Amazon SQS event-source mapping), or EventBridge had to carefully manage their data to fit within the 256 KB payload size limit. This commonly meant chunking larger payloads, externalizing payloads to object stores such as Amazon S3, or using data compression. These workarounds added complexity and latency, creating edge cases that were difficult to monitor and debug.

With the recent launches, you can now transmit payloads up to 1 MB, significantly reducing the need for complex data chunking and architectural workarounds. This increased capacity streamlines design patterns, reduces operational overhead, and makes event-driven systems more intuitive to build and maintain. Developers can now include richer data in single payloads—from detailed LLM prompts and full system states to comprehensive context and complete transaction histories.

The new 1 MB payload size limit applies to asynchronous Lambda function invocations, whether you trigger them using either SQS event-source mapping, AWS Command Line Interface (AWS CLI), AWS SDKs, Lambda Invoke API, or AWS services such as EventBridge. The increased limit also extends to all messages and events flowing through Amazon SQS queues and EventBridge Event Buses.

Getting started

There’s nothing you need to do to get started. This enhancement is automatically applied to all new and existing Lambda functions, SQS queues, and EventBridge Event Buses.

If you were previously chunking data at 256KB (or lower) threshold, then you might need to make changes to your service configurations or business logic code to start using the new limit. For example, if you’ve explicitly set Amazon SQS MaximumMessageSize attribute, then you might need to adjust it to a new desired value. Larger payloads might also result in higher costs, as described in the following section.

Real-world example: rich event context in agentic event-driven architectures

Event-driven architectures allow services to operate independently without centralized coordination. In these systems, comprehensive event context is essential. With the increased 1 MB payload limit, events can now carry more comprehensive data—from user profiles and order details to historical interactions. This enables services such as inventory, shipping, and notifications to act autonomously.

Consider the following example. In hospitality and quick-service industries, customer satisfaction depends on timely, thoughtful service recovery. When a guest submits negative feedback through a survey, review, or complaint form, service teams must gather context, interpret the issue, and craft a response. Traditionally, this meant manually piecing together visit logs, loyalty data, and prior complaints. Now, this can be fully automated using an AI agent powered by AWS serverless services and Amazon Bedrock, as shown in the following figure.

Figure 1: Customer feedback processing pipeline

The workflow:

  1. Receive: A new review is submitted through the Review application and emitted as an event to EventBridge Event Bus.
  2. Detect: Event Bus delivers the event to downstream Feedback analysis agent. The agent running in a Lambda function recognizes the review as low-rating or complaint.
  3. Enrich: The agent collects the guest’s visit metadata, booking details, loyalty activity, and complaint history using attached MCP tools into a single structured JSON payload (up to 1 MB).
  4. Queue: The payload is sent to an SQS queue for further asynchronous processing by downstream components.
  5. Generate: A separate Lambda function polls messages from Amazon SQS and invokes an Amazon Bedrock model to analyze the full complaint context, draft a personalized response, suggest a gesture (such as a refund or credit), and classify issue severity.
  6. Deliver: The message is logged and sent to the customer, and to the service team for further analysis.

This use case demonstrates the importance of having a rich context: current and previous visits details, loyalty tier, prior interactions, and feedback history. Previously, teams had to offload pieces of context to Amazon S3 and reference them externally, adding latency and architectural complexity. The new 1 MB payload size means that all this information can be transported together, improving the serverless agentic workflow efficiency and streamlining maintenance.

Best practices when using large payloads

The following sections outline best practices that you should apply when using larger payloads.

Performance considerations

Monitor Lambda function memory usage carefully when working with larger payloads, because parsing and processing complex JSON objects can increase memory usage and execution duration. Test your systems thoroughly under load, especially for high-throughput applications, by benchmarking with realistic payload sizes and traffic patterns. Although the payload limit has increased to 1 MB, the Lambda 15-minute timeout and memory limits remain unchanged. When applicable, you can use compression to process even larger datasets efficiently, but remember to account for the added CPU overhead of compression and decompression in your performance calculations. Read the Monitoring best practices for event delivery with Amazon EventBridge post for more best practices to tune your event-driven architectures performances.

Operational guidelines

Configure dead-letter-queues (DLQ) to make sure that failed messages are retained for inspection and troubleshooting. This becomes especially important with larger payloads, because debugging complex data structures necessitates access to the complete message context. Implement robust error handling and retries to manage transient failures, particularly when processing rich payload content that may contain nested structures or complex relationships.

To further optimize throughput, you can batch similar smaller events together into a single payload. However, avoid mixing unrelated events and maintain clear boundaries between different business domains and processes.

Always make sure that your downstream dependencies are capable of handling larger payloads.

When to use external storage

Even with the increased 1 MB payload limit, there are scenarios where patterns such as claim check remain a sound architectural choice. These patterns involve storing a full payload in an external system, such as Amazon S3, and passing a lightweight reference through your event stream. This approach continues to provide value when payloads exceed the new limit, when data needs to be reused by multiple consumers, or when strict governance, traceability, and security requirements are involved. For example, audit logs, image metadata, or large ML inference inputs may still surpass the 1 MB boundary, even when compressed. Instead of risking truncation or fragmentation, a claim check enables consistent, scalable access to the complete data set.

You can use open source libraries such as the Kafka sink connector for EventBridge and Amazon SQS Extended Client Library (available for Python and Java) that abstract complexities of storing large objects in external storage.

Cost management

Although larger payloads enable richer context in your applications, logging full payloads can increase storage and processing costs. Services such as CloudWatch Logs charge based on data volume, thus implementing selective logging, payload truncation, or sampling becomes crucial for high-volume events. Consider logging only essential fields or implementing smart sampling strategies based on business importance.

For full payload archival and retention, evaluate cost-effective storage solutions such as Amazon S3 with appropriate lifecycle policies. This can include moving older logs to cheaper storage tiers or implementing automated cleanup procedures for non-critical data. Balance your retention needs with cost optimization by defining clear policies for what data needs to be kept and for how long.

Review the pricing pages for AWS Lambda, Amazon EventBridge, and Amazon SQS to learn about the costs of delivering and processing events and messages.

Conclusion

The increase in maximum payload size from 256 KB to 1 MB enables developers to build more efficient distributed architectures. You can use this enhancement to transport richer context in event and message payloads, reducing the need for complex workarounds that previously added architectural complexity and operational overhead. This added room to transmit rich context means that you can streamline your workflows, improve observability, and reduce architectural complexity whether using choreography or orchestration patterns.

Go to the developer guides for AWS Lambda, Amazon EventBridge, and Amazon SQS, to learn more about how to take advantage of this update.

To learn more about serverless architectures, visit Serverless Land.

Create a customizable cross-company log lake, Part II: Build and add Amazon Bedrock

Post Syndicated from Colin Carson original https://aws.amazon.com/blogs/big-data/create-a-customizable-cross-company-log-lake-part-ii-build-and-add-amazon-bedrock/

In Part I, we introduced the business background behind Log Lake. In this post, we describe how to build it, and how to add model invocation logs from Amazon Bedrock.

The original use case of Log Lake was to join AWS CloudTrail logs (with StartSession API calls) with Amazon CloudWatch logs (with session keystrokes from within Session Manager, a capability of AWS Systems Manager), to help a manager review an employee’s use of elevated permissions to determine if the use was appropriate. Because there might be only one event of elevated privileges in millions or billions of rows of log data, finding the right row to review was like looking for a needle in a haystack.

Log Lake is not just for Session Manager, but also general purpose CloudTrail and CloudWatch logs. After adding CloudWatch and CloudTrail logs to raw tables at scale, you can set up AWS Glue jobs to process the many tiny JSON files of raw tables into bigger binary files for “readready” tables. Then, these readready tables could be queried with different filters to answer questions for many use cases, such as legal or regulatory reviews for compliance, deep forensic investigations for security, or auditing. Log Lake is an answer to the question “Are there logs, and if so, how do I get them?”

Solution overview

Log Lake is a data lake for compliance-related use cases, uses CloudTrail and CloudWatch as data sources, has separate tables for writing (original in raw JSON file format) and reading (read-optimized readready in transformed Apache ORC file format), and gives you control over the components so you can customize it for yourself.

The following diagram shows the system architecture.

The workflow consists of the following steps:

  1. An employee uses Session Manager to access Amazon Elastic Compute Cloud (Amazon EC2). Sessions might include sessionContext.sourceIdentity if a principal provided it while assuming a role (requires sts:SetSourceIdentity in the role trust policy). Our AWS Glue jobs filtered on this field to reduce cost and improve performance.
  2. Logging in to an EC2 instance using Session Manager and performing actions during a session triggers two kinds of logs: CloudTrail records API activity (StartSession) and CloudWatch records session data from within the service (sessionData). Sample CloudTrail and CloudWatch log files are in the GitHub repository, generated from a real Systems Manager session. We recommend you upload these files in your first deployment, but alternatively, you can generate your own data files.
  3. An Amazon Data Firehose subscription copies logs to Amazon Simple Storage Service (Amazon S3) using a CloudWatch subscription filter. CloudWatch combines multiple log events into one Firehose record when it is sent using subscription filters. This is why Log Lake uses regex serde to process CloudWatch rather than JSON serde. When using Firehose subscription filters, Firehose compresses data with GZIP level 6 compression.
  4. Optionally, replication rules copy files to consolidated S3 buckets.
  5. The AddAPart AWS Lambda function associates many tiny JSON files with raw Hive tables in the Data Catalog using the AWS Glue API, triggered by S3 event notifications.
  6. The AWS Glue job reads raw tables and writes to bigger binary ORC files, a columnar file format suitable for analytics. Amazon Athena needs JSON documents on separate lines for processing. In our benchmarking using CloudWatch and CloudTrail workloads, ORC ZLIB had the fastest (lowest) query duration, and was half the file size of Parquet Snappy (1246 MB ORC ZLIB vs 2.4GB Parquet Snappy). Also, ORC is used by AWS CloudTrail Lake. To test file formats, logs from CloudTrail Systems Manager (eventsource='ssm.amazonaws.com') were copied to generate a total population of JSON files over 500 GB. First, a JSON table was created. Then two additional tables were created using Athena CTAS: one for ORC ZLIB, and one for Parquet Snappy. Tests compared three subsequent query durations for three different workloads across ORC vs. Parquet.
  7. The AddAPart Lambda function associates ORC files with Hive readready tables. AddAPart for readready is created using the same stack as for raw, but different parameters (bucket, table, and so on). Hive table format was used for raw because incoming files were JSON, and readready used Hive (not Iceberg) for consistency and append only operations.
  8. Users can query readready tables using the Athena API.

Log Lake uses multiple services together:

  • CloudTrail logs for StartSession API activity (required for auditing, compliance, legal purposes)
  • CloudWatch logs to extend and add keystrokes from Session Manager, so what happened within a session can be reviewed for appropriate use
  • Lambda and Amazon Simple Queue Service (Amazon SQS) for asynchronous invocation of S3 event notifications, for serverless event-driven processing to associate data files with metadata tables
  • The Data Catalog as a metastore to register table metadata, either standalone or as part of a data mesh architecture
  • AWS Glue Spark jobs to transform data from original raw format to read-optimized tables
  • Athena for one-time queries

The architecture of Log Lake includes the following design choices:

  • Separate tables for writing (raw) and reading (readready).
  • Asynchronous invocation using Lambda and Amazon SQS to add partitions for files (AddAPart).
  • AWS Glue jobs with Spark SQL and views (“many view”).
  • AWS services designed to do one thing well, such as Amazon S3 for storage and Amazon SQS for message queueing. This gives data engineers control over components for cost or customization.

Separate tables for reading (readready) and writing (raw)

The concept of raw and readready tables represents two distinct approaches to data storage and processing, each serving different purposes in a data architecture:

  • Raw tables – Source-aligned and write-optimized. They are backed by many tiny files (KB in size) in original format. For CloudWatch and CloudTrail, this means JSON file format.
  • ReadReady tables – Source-aligned and read-optimized. They are backed by bigger binary files, usually larger than 10 MB, in columnar file format.

Part I contains our comparison of performance, cost, and convenience of both table layers.

Add partition Lambda functions (AddAPart)

Log Lake uses an event-based, asynchronous invocation approach to add partitions to raw tables. We call this approach “AddAPart with LoLLs” (Lots of Little Lambdas). It is optimized for adding new incoming files in text format to existing Hive tables as fast as possible, with the following assumptions:

  • Incoming raw files are in JSON or CSV and must be stored and queried in original format (can’t be changed to Iceberg-compatible formats such as Parquet or ORC). Append only, not merge or update.
  • Partition management must be automatic.
  • File-based, no dependency on a job (files can be landed by different pipelines in different ways, and handled consistently by the same AddAPart function).
  • No dependency on Athena partition projection (Data Catalog only).

The AddAPart function consists of five steps:

  1. An S3 event notification triggers the AddAPart producer Lambda function.
  2. The AddAPart producer sends messages to a first-in-first-out (FIFO) SQS queue.
  3. Amazon SQS helps prevent duplicate messages using MessageDeduplicationId.
  4. The AddAPart consumer processes a message and translates it to a partition placer profile.
  5. The AddAPart consumer uses the AWS Glue API to create a partition if none exists.

The following are some ways we have used AddAPart:

  • Minimizing the time it takes to associate new data (JSON files) with new partitions (table in Hive).
  • Reducing the cost of partition adding (duplicate S3 prefixes are ignored).
  • Altering file names (Data Firehose postprocessing Lambda functions are an alternative).
  • Customization, such as ignoring files with certain regex patterns in the S3 prefix or file name. If you want to exclude a data source or do an emergency power off, you can do it from within AddAPart without modifying other resources.

“Many view” AWS Glue jobs

Both Log Lake jobs are what we call “many view” AWS Glue jobs, which use createOrReplaceTempView from Spark, using code like the following:

from pyspark.sql import DataFrame, SparkSession
# code
def create_view_from_sqlstatement(
    logger: logging.Logger, spark: SparkSession, sqlstatement: str, name_of_view: str
) -> None:
    """
    Create a view from a SQL statement.
    """
    result_as_df = spark.sql(sqlstatement)
    result_as_df.createOrReplaceTempView(name_of_view)
    logger.info(f"created view {name_of_view} from sqlstatement...")
# code
name_of_view = "some_step_as_view"
sql_statement = some_statement_for_step
create_view_from_sqlstatement(
logger,
spark,
sql_statement,
name_of_view,
)
sql_statement_for_job="select * from some_step_as_view"
returned_df = spark.sql(sql_statement_for_job)

We have used this approach to address the following antipatterns:

  • Trying to do everything in one step – Trying to do all operations and relational algebra in a single Spark SQL statement can become too complex to troubleshoot, understand, or maintain. For us, when we see a single statement with at least 200 lines and 2 subqueries, we prefer to break it down into smaller statements.
  • Code that is not standardized (inconsistent APIs and approaches) that is harder to maintain, support, and enhance – We have seen the freedom of Spark to mix API approaches (Spark SQL API, RDD API, DataFrame API) result in inconsistency and complexity in large code bases with many contributors.
  • Mixing business logic with Spark environment (such as session settings) – Business logic should be separate and portable.

AWS Glue jobs with custom bounded execution and tables that support workload partitioning

You can tell AWS Glue jobs to look at a maximum of n days or n rows with custom bounds, which we implement using Spark Data Frames as follows:

name_of_view = "mybounds_as_view"

sql_statement = f"""
SELECT (current_timestamp() - INTERVAL {days_begin} DAY) floor_as_time
,cast((current_timestamp() - INTERVAL {days_begin} DAY) AS date) floor_as_date
,cast((current_timestamp() + INTERVAL  {days_end} DAY) AS date) ceiling_as_date
"""

Also, jobs can prune data using table partitions (and use partition indexes). This helps you prepare routine mechanisms up front that are ready to run and recover from missing data by running relative backfill jobs until data is up to date.

Prerequisites

Complete the following prerequisite steps to implement this solution:

  1. Download the repository: 
    git clone https://github.com/aws-samples/sample-log-lake-for-compliance.git

  2. Create or identify an S3 bucket to use during the walkthrough. This will be used for storing the AWS Glue job scripts, Lambda Python files, and AWS CloudFormation stacks.
  3. Copy all files under log_lake to the S3 bucket.
  4. If the S3 bucket is encrypted using an AWS Key Management Service (AWS KMS) key, note the Amazon Resource Name (ARN) of the key.

Build Log Lake

To build Log Lake, follow the deployment steps in the how_to_deploy.md file in the repo.

After deployment is complete, you can upload demo data files and run the AWS Glue jobs to demo how to answer the question, “Who did what in session manager?” For this, switch over to the how_to_demo.md file and follow the steps.

When you are done, you should see the following tables in the Data Catalog:

  • from_cloudtrail_readready – Contains processed CloudTrail session data
  • from_cloudwatch_readready – Contains processed CloudWatch session logs

You can view them on the AWS Glue console or query them directly in Athena. The following is a sample query from the repository that shows how to join both tables to get API activity from CloudTrail and join it to session data (keystrokes) from CloudWatch:

SELECT t.eventsource 
,t.eventname 
,t.eventtime 
,w.logaccountid 
,w.loggroup 
,w.subscriptionfilters 
,w.eventtime as eventtime_from_cloudwatch
,w."session" as session_from_cloudwatch
 FROM loglakeblog.from_cloudwatch_readready w 
 inner join loglakeblog.from_cloudtrail_readready t
on  w.f_sessionid=t.f_sessionid

Add Amazon Bedrock model invocation logs

Adding Bedrock model invocation logs to Log Lake is important to enable human review of agent actions with elevated permissions. Some examples of the need for human oversight are tool use, computer use, agentic misalignment, and high impact AI in federal agencies. If you have not considered this use case and are using LLMs, we urge you to review Amazon Bedrock logs and consider either a managed product or a self-built data lake like Log Lake.

In this post, we use “agentic” and “agent” to refer to a large language model (LLM) using tools with some autonomy to iterate toward a goal.

To generate model invocation logs for this post, we created a custom Lambda function to ask Anthropic’s Claude 4.5 to list files in a bucket using a tool. We used this as a plausible future scenario where a human might need to review an agent’s actions and logs to decide if an agent’s tool use was appropriate.

The following diagram shows the components involved.

For logging inputs and outputs of LLMs running on Bedrock, refer to Monitor model invocation using CloudWatch Logs and Amazon S3. For simplicity, we avoided CloudWatch logs and set up logging directly to Amazon S3.

For logging API activity for Amazon Bedrock, refer to Monitor Amazon Bedrock API calls using CloudTrail.

We have included examples of the CloudTrail and CloudWatch files from Amazon Bedrock model invocation logs in the repository.

Before you create the model invocation logs, make sure you have created the from_cloudtrail_readready table from the previous steps.

Follow the steps in the GItHub repo to add Amazon Bedrock model invocation logs to Log Lake. When done, you should have the tablereplace_me_with_your_database.from_bedrock_readready.

You can query this table using Athena and join it to from_cloudtrail_readready, using SQL like the following example from the repo:

SELECT 
t.useridentity_arn 
,t.eventtime 
,t.eventsource 
,t.eventname 
,b.request_time 
,b.modelid 
,regexp_extract(b.input_messages, '^(.*)({"input":{.*"type":"tool_use"})(.*)$', 2) as input_message_with_tool_use
,b.input_messages
,b.input_inputtokencount
,b.output_outputbodyjson_content
,b.output_outputtokencount
FROM loglakeblog.from_cloudtrail_readready t
left outer join loglakeblog.from_bedrock_readready b 
on t.requestid = b.requestid 
where t.logcalendarday>20240601

Use an agent to review an agent

The predefined query we used in our demo is what we used when we knew the needle in the haystack (tool_use in input messages), but this approach wouldn’t work for new, unknown patterns that require running SQL queries in multiple steps to understand complex data.

Our solution includes a method for an agent in Amazon Bedrock to review an agent in Amazon Bedrock. In this post’s repository, we include a Log Lake Looker Lambda function, which uses an LLM (Anthropic’s Claude) to talk to a database (the Log Lake AWS Glue database).

This pattern is not new. It has been described in 2024 in the paper DB-GPT: Empowering Database Interactions with Private Large Language Models as “a paradigm shift in database interactions, offering a more natural, efficient, and secure way to engage with data repositories.” This is an extension of an older idea from 1998: an interface to data was described in the Distributed Computing Manifesto as “the client is no longer dependent on the underlying data structure or even where the data is located.”

Using an agent to query Log Lake has multiple benefits:

  • An engineered agent can deliver consistent, reliable, high-quality answers during stressful situations, such as a time-sensitive incident response or high-visibility investigation
  • Users don’t have to write their own queries and can reduce their cognitive load (“What was that long column name?”)
  • It can reduce onboarding and training time (the agent implements the training and specialized knowledge of the data structures)

You can ask Log Lake Looker an open-ended question and get an answer without writing a query. Log Lake Looker performs the following actions for you:

  1. Create a valid SQL query from a natural language user prompt. Log Lake Looker is optimized for the from_bedrock_readready table using a system prompt, like the Anthropic SQL sorcerer example.
  2. Run the query in Athena using a custom tool.
  3. Review tool results (rows) and replies with a simple summary.
    When using input and output that can be verbose, like query results, you might need to manage tokens in your context window. For example, if the sum of input and output tokens exceeds the model’s context window, newer Claude models return a validation error, such as the following error we saw during testing: 

    Unexpected ClientError: err=ValidationException('An error occurred (ValidationException) when calling the InvokeModel operation: Input is too long for requested model.') type(err)= error_code='ValidationException' error_message='Input is too long for requested model.'

  4. Compact context by removing tool results. This improves time to answer performance, quality of answer, and reduces proliferation of potentially sensitive data to model invocation logs.
  5. Either run a follow-up query or suggest next steps for the human user.

Log Lake Looker looks at small samples from from_bedrock_readready using more than one try. This means the model reflects on its output and can create a follow-up query based on query results. To learn more about this, we recommend reading about reflection and iterative refinement. We have seen useful responses from agents using iterative approaches, especially when context is managed (for example, a specific system prompt using one table only or a limit on conversational turns) and tool results are compacted.

We’ve seen the agent answer simple questions like “can you query my table and tell me what you find?” in less than 60 seconds more than 50% of the time, without optimizing for any specific question. The following are snippets of CloudWatch logs to show you what’s possible, using Anthropic’s Claude Sonnet 4.5:

2025-12-02 06:05:47 lambda_function lambda_handler INFO Event received: {
"prompt_from_user": "Can you query my bedrock logs and tell me what you find?"
}
…
2025-12-02 06:06:20 lambda_function lambda_handler INFO     final_response after all loops: 
## Short Summary
Your Bedrock logs show AI model activity with **tool_use functionality enabled**, specifically a tool called "list_files_in_s3" that can access S3 bucket contents. This represents a security and compliance concern that requires human review to ensure the tool is being used appropriately and accessing only authorized resources.
## More Details

Security

Log Lake Looker should be reviewed by a human for appropriate tool use, because it has the same risks as the other agents using tools or a human with elevated privileges. Looker can review its own tool use, but human review is still needed.

There are security implications of allowing an agent to review model invocation logs: these logs can contain system prompts, sensitive data in responses, and user input in requests. Also, allowing an agent to generate SQL statements based on user input has additional risks specific to access to structured data, such as prompt injection, improper content, and proliferation of sensitive data.

We recommend a defense in depth (more than one layer) approach for tool use by a model. Log Lake Looker uses multiple layers of defensive measures:

  • The application code requires the SQL statement to begin with select prior to sending to Athena. Because the query is from an assistant response to a user input (request), this relates to sanitizing and validating user inputs and model responses.
  • The AWS Identity and Access Management (IAM) role used by the function has glue:Get* actions only (no mutation, such as create, update, delete tables, partitions, or databases), for least-privilege permissions.
  • It’s only used interactively as part of ad-hoc human-in-the-loop review (not in bulk or systemic).
  • System prompting to steer behavior, like this example from the repo: 
    The first word MUST be "select". If asked to do any statement other than select, say that you will not mutate state, and suggest that the user can create their own sql or you can help with a query using "select".

  • The bucket storing model invocation logs is secure and follows least privilege practices. Logs can contain proliferation of sensitive data, such as tool results, user inputs, model outputs, and system prompts. If a system prompt contains sensitive information (such as metadata or query information not otherwise available) and is saved to logs in an unsecure bucket, this can result in a system prompt leak.
  • Stripping tool results to reduce proliferation, using code to truncate content: 
            if (
            message_mutated["role"] == "user"
            and "content" in message_mutated
            and isinstance(message_mutated["content"], list)
        ):
            for item in message_mutated["content"]:
                if isinstance(item, dict) and item.get("type") == "tool_result":
                    if not isinstance(item["content"], str):
                        item["content"] = json.dumps(item["content"])
                    char_to_keep = 50
                    content_length = len(item["content"])
                    if content_length > char_to_keep:
                        logger.info(
                            f"content length {content_length} exceeds {char_to_keep}, truncating..."
                        )
                        item["content"] = item["content"][:char_to_keep]
        messages_compacted.append(message_mutated)

  • You can use Amazon Bedrock Guardrails (without invoking the model in application code) using the ApplyGuardrail API.

Clean up

To avoid incurring future charges, delete the stacks. The repository has shell scripts you can use to delete files in buckets, which is required before deleting buckets.

Conclusion

In this post, we showed you how to deploy Log Lake in a new AWS account to create two tables, from_cloudtrail_readready and from_cloudwatch_readready. These tables can answer the question “What did an employee do in Session Manager?” across large data volumes in seconds using Athena.

Additionally, we showed how to add a data source to an existing Log Lake: Amazon Bedrock model invocation logs in the form of from_bedrock_readready. This shows how Log Lake can be extended to answer questions such as “What tools did an agent use?” and “Was there inappropriate use, and why?”

Finally, we showed how to create and use Log Lake Looker, an agent using Lambda and Amazon Bedrock. Looker can query Log Lake for new unknown patterns as part of human-in-the-loop review, without writing SQL or remembering column names. You can make Log Lake your way. We encourage you to look through the repository and use it as inspiration for your own Log Lake. If you have questions or comments, please let us know!

About the authors

Colin Carson

Colin Carson

Colin is a Data Engineer at AWS ProServe. He has designed and built data infrastructure for multiple teams at Amazon, including Internal Audit, Risk & Compliance, HR Hiring Science, and Security.

Sean O’Sullivan

Sean O’Sullivan

Sean is a Cloud Infrastructure Architect at AWS ProServe. He partners with Global Financial Services customers to drive digital transformation projects, helping them architect, automate, and engineer solutions in AWS.

.NET 10 runtime now available in AWS Lambda

Post Syndicated from Henrique Graca original https://aws.amazon.com/blogs/compute/net-10-runtime-now-available-in-aws-lambda/

AWS Lambda now supports .NET 10 as both a managed runtime and base container image. .NET is a popular language for building serverless applications. Developers can now use the new features and enhancements in .NET when creating serverless applications on Lambda. This includes support for file-based apps to streamline your projects by implementing functions using just a single file.

.NET 10 delivers runtime and compiler optimizations including enhancements to the JIT compiler and improvements to Native AOT that reduce executable size and startup time. For details of the .NET 10 features, you can go to the .NET 10 overview.

You can develop Lambda functions in .NET 10 using the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS Toolkit for Visual Studio, AWS Extensions for .NET CLI (Amazon.Lambda.Tools), AWS Serverless Application Model (AWS SAM), AWS Cloud Development Kit (AWS CDK), and other infrastructure as code (IaC) tools.

You can also use .NET 10 with Powertools for AWS Lambda (.NET), a developer toolkit that helps you implement serverless best practices. Use cases include observability, batch processing, AWS Systems Manager Parameter Store integration, idempotency, and more.

This post demonstrates what’s new in the .NET 10 Lambda runtime and how you can use the new .NET 10 runtime in your serverless applications.

File-based C# applications

.NET 10 introduces file-based apps, which are programs contained in a single .cs file, without a .csproj file or a complex folder structure. File-based apps are an ideal way to streamline the development and management of .NET Lambda functions. They are fully supported by the Lambda .NET 10 runtime and associated developer tooling.

Creating C# file-based apps

The fastest way to get started creating a C# file-based Lambda function is to use the Amazon.Lambda.Templates package. Version 8.0.1 of the package adds the lambda.FileBased template as well as updating the rest of the templates in the package to .NET 10.

Install the package by running the following command:

dotnet new install Amazon.Lambda.Templates

Create a new C# file-based Lambda function by running the following command:

dotnet new lambda.FileBased -n MyLambdaFunction

This creates a file in the current directory called MyLambdaFunction.cs with all of the required startup code necessary for a Lambda function. The following is the starting content of the file:

// C# file-based Lambda functions can be deployed to Lambda using the 
// .NET Tool Amazon.Lambda.Tools version 6.0.0 or later. 
// 
// Command to install Amazon.Lambda.Tools 
//   dotnet tool install -g Amazon.Lambda.Tools 
// 
// Command to deploy function 
//    dotnet lambda deploy-function <lambda-function-name> MyLambdaFunction.cs 
// 
// Command to package function 
//    dotnet lambda package MyLambdaFunction.zip MyLambdaFunction.cs 
 
 
#:package [email protected] 
#:package [email protected] 
#:package [email protected] 
 
// Explicitly setting TargetFramework here is done to avoid 
// having to specify it when packaging the function with Amazon.Lambda.Tools 
#:property TargetFramework=net10.0 
 
// By default File-based C# apps publish as Native AOT. When packaging Lambda function 
// unless the host machine is Amazon Linux a container build will be required. 
// Amazon.Lambda.Tools will automatically initate a container build if docker is installed. 
// Native AOT also requires the code and dependencies be Native AOT compatible. 
// 
// To disable Native AOT uncomment the following line to add the .NET build directive 
// that disables Native AOT. 
//#:property PublishAot=false 
 
using Amazon.Lambda.Core; 
using Amazon.Lambda.RuntimeSupport; 
using Amazon.Lambda.Serialization.SystemTextJson; 
using System.Text.Json.Serialization; 
 
// The function handler that will be called for each Lambda event 
var handler = (string input, ILambdaContext context) => 
{ 
    return input.ToUpper(); 
}; 
 
// Build the Lambda runtime client passing in the handler to call for each 
// event and the JSON serializer to use for translating Lambda JSON documents 
// to .NET types. 
await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaSerializerContext>()) 
        .Build() 
        .RunAsync(); 
 
// Since Native AOT is used by default with C# file-based Lambda functions the source generator 
// based Lambda serializer is used. Ensure the input type and return type used by the function 
// handler are registered on the JsonSerializerContext using the JsonSerializable attribute. 
[JsonSerializable(typeof(string))] 
public partial class LambdaSerializerContext : JsonSerializerContext 
{ 
}

File-based functions use executable assembly handlers, in which the compiler generates the Main() method containing your function code. Therefore, your code must include the Amazon.Lambda.RuntimeSupport NuGet package and implement the LambdaBootstrapBuilder.Create method to bootstrap the runtime client.

File-based applications also use .NET Native AOT by default. You can disable Native AOT by adding #:property PublishAot=false to the top of the file. For more information on using Native AOT in Lambda, go to Compile .NET Lambda function code to a native runtime format in the Lambda documentation.

Deploying C# file-based apps

To deploy your function using the dotnet CLI with the Amazon.Lambda.Tools extension, pass the .cs filename as an added argument. Native AOT is enabled by default, thus the build must match the target architecture. If you’re building on the same architecture as the target Lambda function and on Amazon Linux 2023, then the build runs natively. Otherwise, Amazon.Lambda.Tools uses a Docker container to build the function.

For example, to deploy for x86_64 (default architecture):


dotnet lambda deploy-function ToUpper ToUpper.cs --function-runtime 
dotnet10 --function-role <role-arn>

Alternatively, to deploy for arm64:


dotnet lambda deploy-function ToUpper ToUpper.cs --function-runtime 
dotnet10 --function-role <role-arn> --function-architecture arm64

Debugging C# file-based apps

Visual Studio Code with the C# Dev Kit support debugging C# file-based applications.

1. Install the test tool.
dotnet tool install -g Amazon.Lambda.TestTool

2. Start the emulator.
dotnet lambda-test-tool start --lambda-emulator-port 5050

3. Configure .vscode/launch.json to attach to the process.

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "LambdaDebugFile",
      "type": "coreclr",
      "request": "launch",
      "program": "${fileDirname}/artifacts/Debug/${fileBasenameNoExtension}.dll",
      "cwd": "${workspaceFolder}",
      "console": "internalConsole",
      "stopAtEntry": false,
      "env": {
        "AWS_LAMBDA_RUNTIME_API": "localhost:5050/${fileBasenameNoExtension}"
      },
      "preLaunchTask": "build-active-file"
    }
  ]
}

4. Configure .vscode/tasks.json to build the active file.

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "build-active-file",
      "command": "dotnet",
      "type": "process",
      "args": [
        "build",
        "${file}",
        "--output",
        "./artifacts/Debug"
      ],
      "problemMatcher": "$msCompile"
    }
  ]
}

The configuration uses ${file}, which allows the build task to target whichever C# file is currently active in your editor. This enables seamless debugging across multiple single-file functions.

Lambda Managed Instances

The Lambda runtime for .NET 10 includes support for Lambda Managed Instances, so that you can run .NET 10 Lambda functions on Amazon Elastic Compute Cloud (Amazon EC2) instances while maintaining serverless operational clarity. Therefore, you can use current-generation EC2 instances, including Graviton4, network-optimized instances, and other specialized compute options, without managing instance lifecycles, operating system patching, or scaling policies. Lambda Managed Instances provides access to Amazon EC2 commitment-based pricing models, such as Compute Savings Plans and Reserved Instances, which can provide up to a 72% discount over Amazon EC2 On-Demand pricing. This offers significant cost savings for steady-state workloads while maintaining the familiar Lambda programming model. For more information, go to Lambda Managed Instances.

With Lambda Managed Instances, each function execution environment can process multiple function invokes at the same time. In .NET, Lambda uses .NET Tasks for asynchronous processing of multiple concurrent requests. You should apply the same concurrency safety practices when using Lambda Managed Instances that you would in any other multi-concurrent environment. For example, any mutable state—including shared collections, database connections, and static objects—must be thread safe. For more information, go to .NET runtime for Lambda Managed Instances.

Performance considerations

At launch, new Lambda runtimes receive less usage than existing established runtimes. This can result in longer cold start times due to reduced cache residency within internal Lambda sub-systems. Cold start times typically improve in the weeks following launch as usage increases. Therefore, AWS recommends not drawing conclusions from side-by-side performance comparisons with other Lambda runtimes until the performance has stabilized.

Performance is highly dependent on workload, thus you should conduct your own testing instead of relying on generic test benchmarks. There are a range of features available to reduce the impact of cold starts for Lambda functions that use .NET 10, including SnapStart, provisioned concurrency, Native AOT, and Lambda Managed Instances.

In addition, .NET 10 Lambda workloads might experience slightly lower performance until some added runtime enhancements are released. Go to dotnet/runtime#120288 for details.

Migrating from .NET 8 to .NET 10

To use .NET 10 and the new file-based features, you must update your tools.

1. Install or update the .NET 10 SDK.

2. If you are using AWS SAM, then install or update to the latest version.

3. If you are using Visual Studio, then install or update the AWS Toolkit for Visual Studio to version 1.83.0.0 or later.

4. If you use the .NET Lambda Global Tools extension (Amazon.Lambda.Tools) for the .NET CLI, then update to version 6.0.0 or later to support file-based C#.

To upgrade a function to .NET 10, check your code and dependencies for compatibility with .NET 10, run tests, and update as necessary. Generative AI can help: consider using AWS Transform custom or coding assistants such as Kiro to help with upgrades.

Upgrading using the dotnet CLI

For projects using the dotnet CLI with the Amazon.Lambda.tools extension:

1. Open the .csproj project file.

2. Update the TargetFramework to net10.0.

3. Update NuGet packages Amazon.Lambda.* to the latest versions.

4. If using aws-lambda-tools-defaults.json, then set function-runtime to dotnet10.

5. Run dotnet lambda deploy-function to deploy.

Upgrading using the AWS Toolkit for Visual Studio

To upgrade a function to .NET 10:

1. Open the .csproj project file and update the TargetFramework to net10.0.

2. Update NuGet packages to the latest versions.

3. Right-click the project in Solution Explorer and choose Publish to AWS Lambda.

Upgrading container image functions

Along with the preceding changes, the .NET 8 and .NET 10 runtimes are built on the provided.al2023 runtime, which is based on the Amazon Linux 2023 minimal container image. The Amazon Linux 2023 minimal image uses microdnf as a package manager, symlinked as dnf. This replaces the yum package manager used in .NET 6 and earlier Amazon Linux 2-based images. If you deploy your Lambda functions as container images, then you must update your Dockerfiles to use dnf instead of yum when upgrading to the .NET 10 base image from .NET 6 or earlier base images.

Learn more about the provided.al2023 runtime in the post Introducing the Amazon Linux 2023 runtime for AWS Lambda.

Using the .NET 10 runtime in Lambda

The following sections demonstrate how to use the .NET 10 runtime in Lambda.

The console

On the Create Function page of the Lambda console, choose .NET 10 in the Runtime dropdown menu, as shown in the following figure.

Figure 1: Creating a .NET 10 function in the Lambda console

To update an existing Lambda function to .NET 10, navigate to the function in the Lambda console. Choose Edit in the Runtime settings panel, then choose .NET 10 from the Runtime dropdown menu, as shown in the following figure.

Figure 2: Editing runtime settings to choose .NET 10

Lambda container image

Change the .NET base image version by modifying the FROM statement in your Dockerfile:

FROM public.ecr.aws/lambda/dotnet:10
# Copy function code
COPY artifacts/publish/ ${LAMBDA_TASK_ROOT}

AWS SAM

In AWS SAM, set the Runtime attribute to dotnet10:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Simple Lambda Function
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: My .NET Lambda Function
      CodeUri: ./src/MyFunction/
      Handler: MyFunction::MyFunction.Function::FunctionHandler
      Runtime: dotnet10

For file-based functions, set CodeUri to the C# file path relative to the AWS SAM template. AWS CloudFormation commands such as deploy-serverless and package-ci package the file-based Lambda function as a .NET executable. The Handler field must be set to the .NET assembly name, which for file-based C# applications is the filename minus the .cs extension:

ToUpperFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: ToUpperFunction
    Runtime: dotnet10
    CodeUri: ./ToUpperFunction.cs
    MemorySize: 512
    Timeout: 30

AWS SAM supports generating new serverless .NET 10 applications using the sam init command. Refer to the AWS SAM documentation for details.

Conclusion

AWS Lambda now supports .NET 10 as a managed language runtime to help developers build more efficient, powerful, and scalable serverless applications. .NET 10 language additions include C# 14 features, runtime optimizations, and improved Native AOT support. This release also introduces file-based C# applications for streamlined single-file Lambda functions, and it includes support for Lambda Managed Instances for specialized compute requirements and at-scale cost efficiency.

You can build and deploy functions using .NET 10 using the AWS Management Console, AWS CLI, AWS SDK, AWS SAM, AWS CDK, or your choice of IaC tool. You can also use the .NET 10 container base image if you prefer to build and deploy your functions using container images.

Try the .NET 10 runtime in Lambda today and experience the benefits of this updated language version.

To find more .NET examples, use the Serverless Patterns Collection. For more serverless learning resources, visit Serverless Land.

Architecting conversational observability for cloud applications

Post Syndicated from Anton Aleksandrov original https://aws.amazon.com/blogs/architecture/architecting-conversational-observability-for-cloud-applications/

Modern cloud applications are commonly built as a collection of loosely coupled microservices running on services like Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), or AWS Lambda. This architecture gives engineering teams flexibility and scalability, but its inherently distributed nature also makes troubleshooting more difficult. When something breaks, engineers often find themselves digging through logs, events, and metrics scattered across different observability layers. With Kubernetes, for example, without a deep understanding of the service, troubleshooting can turn into a time-consuming effort to manually correlate information from different sources.

In this post, we walk through building a generative AI–powered troubleshooting assistant for Kubernetes. The goal is to give engineers a faster, self-service way to diagnose and resolve cluster issues, cut down Mean Time to Recovery (MTTR), and reduce the cycles experts spend finding the root cause of issues in complex distributed systems.

Overview

One of the challenges of architecting a modern cloud application is keeping observability intact across many moving pieces. Anyone who has ever tailed logs in one terminal while running kubectl describe and curl commands in another, knows how tedious this can get. Distributed systems are powerful, but they’re also complex. Kubernetes, for example, offers strong orchestration capabilities, yet troubleshooting inside a cluster often means navigating multiple layers of abstractions such as pods, nodes, networking, logs, and events. On top of that, the system generates a large volume of telemetry, including kubelet logs, application logs, cluster events, and metrics. Making sense of these layers requires both expertise on the system and application knowledge.

This skill gap shows up in the numbers. According to the 2024 Observability Pulse Report, 48% of organizations say that lack of team knowledge is their biggest challenge to observability in cloud-native environments. MTTR has also been going up for three years straight, with most teams (82%) saying it can take more than an hour to resolve production issues.

When something goes wrong and your applications start to fail for an unknown reason, engineers often must start stitching together signals from multiple sources to find the root cause. That can be tedious for specialists, and it gets worse when the issue is intermittent or spans across services. Often, multiple teams need to get involved – application engineers may not know Kubernetes well, while platform teams may not have deep insight into the applications. This can result in longer troubleshooting cycles, potentially degraded user experience, and pulling engineers away from planned work that drives business goals.

Figure 1. A multitude of telemetry sources in Kubernetes clusters

This is where generative artificial intelligence (AI) can help. Users can build an AI assistant that combines large language model (LLM)-driven analysis and guidance with existing telemetry data. This assistant enables engineers to troubleshoot issues faster, in a self-service way, without requiring every team to become Kubernetes experts. In the following sections, we show how to build such an assistant for Amazon EKS, however, keep in mind that a similar approach can be extended to other compute services like Amazon ECS or AWS Lambda.

Solution architecture

Architecting this AI-powered troubleshooting assistant consists of three primary parts:

  • Deployment approach selection: The solution supports two architectures – a traditional Retrieval-Augmented Generation (RAG)-based chatbot and a modern Strands-based agentic system that uses the Strands Agents SDK with EKS MCP Server integration for direct EKS API access.
  • Telemetry collection and storage: Collecting telemetry from various sources and storing it as vector embeddings in Amazon OpenSearch (RAG approach) or as 1024-dimensional embeddings in Amazon S3 Vectors (Strands approach).
  • Interactive troubleshooting interface: Building either a web-based chatbot that retrieves relevant telemetry and injects it into LLM prompts, or a Slack integrated multi-agent system that uses MCP tools for real-time Kubernetes diagnostics.

For this architecture walkthrough, we focus on the RAG-based approach. The first step is setting up a pipeline that can reliably collect, process, and store telemetry data. This pipeline aggregates telemetry from the relevant data sources, such as application logs, kubelet logs, and Kubernetes events. In Kubernetes environments, this can be done with a telemetry processor and forwarder, such as Fluent Bit, which streams telemetry into Amazon Kinesis Data Streams. On the receiving end, we use a Lambda function to normalize collected data, Amazon Bedrock to generate vector embeddings, and OpenSearch Serverless to store this embedded representation for efficient retrieval. Because these services are serverless, we can avoid the overhead of managing infrastructure and can focus on the troubleshooting workflow itself.

Figure 2. Collecting telemetry from sources, generating embeddings, and saving in OpenSearch

Pro tip: for better performance and cost-efficiency, your Lambda functions should use batching when ingesting data from Kinesis, generating embeddings, and storing them in OpenSearch.

Once telemetry is collected, converted to embeddings, and stored in OpenSearch, the next step is building a chatbot that uses RAG. Using RAG means that when a user asks a question, the chatbot looks up semantically similar telemetry in OpenSearch, adds it to the prompt, and sends it to the LLM. Instead of generic answers, the model now has relevant telemetry and cluster-specific details it can use to generate useful next steps, such as precise kubectl commands for the troubleshooting assistant, as illustrated in the following diagram.

Figure 3. Chatbot is using user queries augmented with telemetry context to send kubectl commands to the troubleshooting assistant. 

One powerful aspect of this design is its iterative nature. The chatbot hands instructions to a troubleshooting assistant running in the cluster, which executes a set of allowlisted, read-only kubectl commands. The output comes back to the LLM, which can decide whether it needs to investigate further (by asking the troubleshooting assistant to run more kubectl commands), or present a clear resolution path to the engineer. This cycle gradually builds a richer picture of the issue by combining historical telemetry with real-time cluster state to speed up root cause analysis.

Figure 4. Iterative troubleshooting process.

Here’s the end-to-end troubleshooting flow illustrated in the preceding diagram:

  1. An engineer enters a query into the chatbot interface, for example “My pod is stuck in pending state. Investigate.”
  2. The chatbot sends the query to Bedrock, which converts it into vector embeddings.
  3. Using those embeddings, the chatbot retrieves semantically matching telemetry that was previously stored in OpenSearch.
  4. The chatbot generates an augmented prompt, which contains both the original query and semantically relevant telemetry, and passes it to the LLM. The LLM responds with a list of kubectl commands to run for further diagnostics.
  5. The chatbot forwards those commands to the troubleshooting assistant running in the EKS cluster. The agent executes them with a service account that has read-only permissions, following the principle of least privilege, and sends the output back.
  6. Based on the output, the chatbot asks LLM to decide whether to continue investigation (by asking the agent to run more commands), or whether it has enough context to produce an answer.
  7. Once enough information has been gathered (investigation concluded), the chatbot composes a final prompt, including the query, telemetry, and investigation results, and asks the LLM for a final resolution, which it then returns to the engineer.

Example implementation

Use the example repo to deploy the solution in your AWS account. Follow the instructions in README.md for provisioning and testing the sample project using Terraform. Resources provisioned by the example project incur costs in your AWS account. Make sure to clean up the project as described in the README.md to avoid unexpected costs.

The repository provides two deployment architectures controlled by the deployment_type Terraform variable:

  1. RAG-based deployment (default): See the ./terraform/modules directory for the “ingestion-pipeline” module that creates a Kinesis Data Stream and Lambda function to generate embeddings using "amazon.titan-embed-test-v2:0" and store them in OpenSearch. The “agentic-chatbot” module handles the Gradio web interface and kubectl command execution.
  2. Strands agentic deployment: this approach uses the Strands Agents SDK to create a multi-agent system with three specialized agents:
    1. Agent Orchestrator: Coordinates troubleshooting workflows
    2. Memory Agent: Manages conversation context and historical insights
    3. K8s Specialist: Handles Kubernetes diagnostics

The agentic system stores knowledge as 1024-dimensional embeddings in Amazon S3 Vectors, providing cost-optimized vector storage for AI agents. EKS MCP Server integration enabled direct EKS API access through standardized MCP tools located in ./apps/agentic-troubleshooting/src/tools/. Engineers interact via Slack bot integration, where the Strands agents can execute kubectl commands through the MCP protocol while maintaining Pod Identity security for AWS service access.

The following screenshot shows an example chatbot response to a query about a pod being stuck in pending state. The assistant generated and ran multiple kubectl commands to build the output and came up with recommendations for issue remediation.

Figure 5. EKS cluster troubleshooting, example output

See AWS re:Invent 2025 – Streamline Amazon EKS operations with Agentic AI and KubeCon – From Logs To Insights: Real-time Conversational Troubleshooting for Kubernetes with GenAI sessions for a deeper dive into solution implementation.

Security considerations

When implementing AI agents for Kubernetes environments, security must be a primary consideration throughout the architecture. The solution requires secure communication channels between the chatbot and EKS clusters, with interactions authenticated through AWS Identity and Access Management (AWS IAM) roles.

Permissions-wise, command execution security is critical. Implementing strict allowlists that only allow read-only kubectl operations to help prevent unauthorized cluster modifications while maintaining diagnostic capabilities. The troubleshooting assistant should also operate with minimal Kubernetes RBAC permissions, limited to viewing pods, services, events, and logs within specific namespaces.

Data protection measures must include sanitizing application logs before embedding generation to help prevent sensitive information exposure, encrypting the telemetry data in transit through Kinesis and at rest in OpenSearch using AWS Key Management Service (AWS KMS).

Follow the AWS Well-Architected Framework Security Pillar principles, deploy components within Amazon Virtual Private Cloud (Amazon VPC) using private subnets and VPC endpoints to minimize network exposure, implement comprehensive logging of troubleshooting activities for audit purposes, and validate user inputs to protect against prompt injection attacks that could manipulate the AI assistant’s behavior.

Conclusion

In this post, we walked through how to architect a generative AI-powered troubleshooting assistant that gives engineers a way to solve Kubernetes issues in a self-service way, without always needing service experts to step in. By combining telemetry analysis with AI-driven context, engineers can get to the root causes faster and keep MTTR low. Assistant’s ability to pull from multiple telemetry sources, run safe diagnostic commands, and provide actionable recommendations helps to make the troubleshooting process more efficient and less disruptive to ongoing work.

As distributed systems continue to grow in scale and complexity, solutions like the one described in this post become essential. Putting AI on top of your observability data helps to practically handle these challenges today, while also setting you up for more autonomous, resilient operations in the future.

Build multi-step applications and AI workflows with AWS Lambda durable functions

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/build-multi-step-applications-and-ai-workflows-with-aws-lambda-durable-functions/

Modern applications increasingly require complex and long-running coordination between services, such as multi-step payment processing, AI agent orchestration, or approval processes awaiting human decisions. Building these traditionally required significant effort to implement state management, handle failures, and integrate multiple infrastructure services.

Starting today, you can use AWS Lambda durable functions to build reliable multi-step applications directly within the familiar AWS Lambda experience. Durable functions are regular Lambda functions with the same event handler and integrations you already know. You write sequential code in your preferred programming language, and durable functions track progress, automatically retry on failures, and suspend execution for up to one year at defined points, without paying for idle compute during waits.

AWS Lambda durable functions use a checkpoint and replay mechanism, known as durable execution, to deliver these capabilities. After enabling a function for durable execution, you add the new open source durable execution SDK to your function code. You then use SDK primitives like “steps” to add automatic checkpointing and retries to your business logic and “waits” to efficiently suspend execution without compute charges. When execution terminates unexpectedly, Lambda resumes from the last checkpoint, replaying your event handler from the beginning while skipping completed operations.

Getting started with AWS Lambda durable functions
Let me walk you through how to use durable functions.

First, I create a new Lambda function in the console and select Author from scratch. In the Durable execution section, I select Enable. Note that, durable function setting can only be set during function creation and currently can’t be modified for existing Lambda functions.

After I create my Lambda durable function, I can get started with the provided code.

Lambda durable functions introduces two core primitives that handle state management and recovery:

  • Steps—The context.step() method adds automatic retries and checkpointing to your business logic. After a step is completed, it will be skipped during replay.
  • Wait—The context.wait() method pauses execution for a specified duration, terminating the function, suspending and resuming execution without compute charges.

Additionally, Lambda durable functions provides other operations for more complex patterns: create_callback() creates a callback that you can use to await results for external events like API responses or human approvals, wait_for_condition() pauses until a specific condition is met like polling a REST API for process completion, and parallel() or map() operations for advanced concurrency use cases.

Building a production-ready order processing workflow
Now let’s expand the default example to build a production-ready order processing workflow. This demonstrates how to use callbacks for external approvals, handle errors properly, and configure retry strategies. I keep the code intentionally concise to focus on these core concepts. In a full implementation, you could enhance the validation step with Amazon Bedrock to add AI-powered order analysis.

Here’s how the order processing workflow works:

  • First, validate_order() checks order data to ensure all required fields are present.
  • Next, send_for_approval() sends the order for external human approval and waits for a callback response, suspending execution without compute charges.
  • Then, process_order() completes order processing.
  • Throughout the workflow, try-catch error handling distinguishes between terminal errors that stop execution immediately and recoverable errors inside steps that trigger automatic retries.

Here’s the complete order processing workflow with step definitions and the main handler:

import random
from aws_durable_execution_sdk_python import (
    DurableContext,
    StepContext,
    durable_execution,
    durable_step,
)
from aws_durable_execution_sdk_python.config import (
    Duration,
    StepConfig,
    CallbackConfig,
)
from aws_durable_execution_sdk_python.retries import (
    RetryStrategyConfig,
    create_retry_strategy,
)


@durable_step
def validate_order(step_context: StepContext, order_id: str) -> dict:
    """Validates order data using AI."""
    step_context.logger.info(f"Validating order: {order_id}")
    # In production: calls Amazon Bedrock to validate order completeness and accuracy
    return {"order_id": order_id, "status": "validated"}


@durable_step
def send_for_approval(step_context: StepContext, callback_id: str, order_id: str) -> dict:
    """Sends order for approval using the provided callback token."""
    step_context.logger.info(f"Sending order {order_id} for approval with callback_id: {callback_id}")
    
    # In production: send callback_id to external approval system
    # The external system will call Lambda SendDurableExecutionCallbackSuccess or
    # SendDurableExecutionCallbackFailure APIs with this callback_id when approval is complete
    
    return {
        "order_id": order_id,
        "callback_id": callback_id,
        "status": "sent_for_approval"
    }


@durable_step
def process_order(step_context: StepContext, order_id: str) -> dict:
    """Processes the order with retry logic for transient failures."""
    step_context.logger.info(f"Processing order: {order_id}")
    # Simulate flaky API that sometimes fails
    if random.random() > 0.4:
        step_context.logger.info("Processing failed, will retry")
        raise Exception("Processing failed")
    return {
        "order_id": order_id,
        "status": "processed",
        "timestamp": "2025-11-27T10:00:00Z",
    }


@durable_execution
def lambda_handler(event: dict, context: DurableContext) -> dict:
    try:
        order_id = event.get("order_id")
        
        # Step 1: Validate the order
        validated = context.step(validate_order(order_id))
        if validated["status"] != "validated":
            raise Exception("Validation failed")  # Terminal error - stops execution
        context.logger.info(f"Order validated: {validated}")
        
        # Step 2: Create callback
        callback = context.create_callback(
            name="awaiting-approval",
            config=CallbackConfig(timeout=Duration.from_minutes(3))
        )
        context.logger.info(f"Created callback with id: {callback.callback_id}")
        
        # Step 3: Send for approval with the callback_id
        approval_request = context.step(send_for_approval(callback.callback_id, order_id))
        context.logger.info(f"Approval request sent: {approval_request}")
        
        # Step 4: Wait for the callback result
        # This blocks until external system calls SendDurableExecutionCallbackSuccess or SendDurableExecutionCallbackFailure
        approval_result = callback.result()
        context.logger.info(f"Approval received: {approval_result}")
        
        # Step 5: Process the order with custom retry strategy
        retry_config = RetryStrategyConfig(max_attempts=3, backoff_rate=2.0)
        processed = context.step(
            process_order(order_id),
            config=StepConfig(retry_strategy=create_retry_strategy(retry_config)),
        )
        if processed["status"] != "processed":
            raise Exception("Processing failed")  # Terminal error
        
        context.logger.info(f"Order successfully processed: {processed}")
        return processed
        
    except Exception as error:
        context.logger.error(f"Error processing order: {error}")
        raise error  # Re-raise to fail the execution

This code demonstrates several important concepts:

  • Error handling—The try-catch block handles terminal errors. When an unhandled exception is thrown outside of a step (like the validation check), it terminates the execution immediately. This is useful when there’s no point in retrying, such as invalid order data.
  • Step retries—Inside the process_order step, exceptions trigger automatic retries based on the default (step 1) or configured RetryStrategy (step 5). This handles transient failures like temporary API unavailability.
  • Logging—I use context.logger for the main handler and step_context.logger inside steps. The context logger suppresses duplicate logs during replay.

Now I create a test event with order_id and invoke the function asynchronously to start the order workflow. I navigate to the Test tab and fill in the optional Durable execution name to identify this execution. Note that, durable functions provides built-in idempotency. If I invoke the function twice with the same execution name, the second invocation returns the existing execution result instead of creating a duplicate.

I can monitor the execution by navigating to the Durable executions tab in the Lambda console:

Here I can see each step’s status and timing. The execution shows CallbackStarted followed by InvocationCompleted, which indicates the function has terminated and execution is suspended to avoid idle charges while waiting for the approval callback.

I can now complete the callback directly from the console by choosing Send success or Send failure, or programmatically using the Lambda API.

I choose Send success.

After the callback completes, the execution resumes and processes the order. If the process_order step fails due to the simulated flaky API, it automatically retries based on the configured strategy. Once all retries succeed, the execution completes successfully.

Monitoring executions with Amazon EventBridge
You can also monitor durable function executions using Amazon EventBridge. Lambda automatically sends execution status change events to the default event bus, allowing you to build downstream workflows, send notifications, or integrate with other AWS services.

To receive these events, create an EventBridge rule on the default event bus with this pattern:

{
  "source": ["aws.lambda"],
  "detail-type": ["Durable Execution Status Change"]
}

Things to know
Here are key points to note:

  • Availability—Lambda durable functions are now available in US East (Ohio) AWS Region. For the latest Region availability, visit the AWS Capabilities by Region page.
  • Programming language support—At launch, AWS Lambda durable functions supports JavaScript/TypeScript (Node.js 22/24) and Python (3.13/3.14). We recommend bundling the durable execution SDK with your function code using your preferred package manager. The SDKs are fast-moving, so you can easily update dependencies as new features become available.
  • Using Lambda versions—When deploying durable functions to production, use Lambda versions to ensure replay always happens on the same code version. If you update your function code while an execution is suspended, replay will use the version that started the execution, preventing inconsistencies from code changes during long-running workflows.
  • Testing your durable functions—You can test durable functions locally without AWS credentials using the separate testing SDK with pytest integration and the AWS Serverless Application Model (AWS SAM) command line interface (CLI) for more complex integration testing.
  • Open source SDKs—The durable execution SDKs are open source for JavaScript/TypeScript and Python. You can review the source code, contribute improvements, and stay updated with the latest features.
  • Pricing—To learn more on AWS Lambda durable functions pricing, refer to the AWS Lambda pricing page.

Get started with AWS Lambda durable functions by visiting the AWS Lambda console. To learn more, refer to AWS Lambda durable functions documentation page.

Happy building!

— Donnie

Introducing AWS Transform custom: Crush tech debt with AI-powered code modernization

Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/introducing-aws-transform-custom-crush-tech-debt-with-ai-powered-code-modernization/

Technical debt is one of the most persistent challenges facing enterprise development teams today. Studies show that organizations spend 20% of their IT budget on technical debt instead of advancing new capabilities. Whether it’s upgrading legacy frameworks, migrating to newer runtime versions, or refactoring outdated code patterns, these essential but repetitive tasks consume valuable developer time that could be spent on innovation.

Today, we’re excited to announce AWS Transform custom, a new agent that fundamentally changes how organizations approach modernization at scale. This intelligent agent combines pre-built transformations for Java, Node.js, and Python upgrades with the ability to define custom transformations. By learning specific transformation patterns and automating them across entire codebases, customers using AWS Transform custom have achieved up to 80% reduction in execution time in many cases, freeing developers to focus on innovation.

You can define transformations using your documentation, natural language descriptions, and code samples. The service then applies these specific patterns consistently across hundreds or thousands of repositories, improving its effectiveness through both explicit feedback and implicit signals like developers’ manual fixes within your transformation projects.

AWS Transform custom offers both CLI and web interfaces to suit different modernization needs. You can use the CLI to define transformations through natural language interactions and execute them on local codebases, either interactively or autonomously. You can also integrate it into code modernization pipelines or workflows, making it ideal for machine-driven automation. Meanwhile, the web interface provides comprehensive campaign management capabilities, helping teams track and coordinate transformation progress across multiple repositories at scale.

Language and framework modernization
AWS Transform supports runtime upgrades without the need to provide additional information, understanding not only the syntax changes required but also the subtle behavioral differences and optimization opportunities that come with newer versions. The same intelligent approach applies to Node.js, Python and Java runtime upgrades, and even extends to infrastructure-level transitions, such as migrating workloads from x86 processors to AWS Graviton.

It also navigates framework modernization with sophistication. When organizations need to update their Spring Boot applications to take advantage of newer features and security patches, AWS Transform custom doesn’t merely update version numbers but understands the cascading effects of dependency changes, configuration updates, and API modifications.

For teams facing more dramatic shifts, such as migrating from Angular to React, AWS Transform custom can learn the patterns of component translation, state management conversion, and routing logic transformation that make such migrations successful.

Infrastructure and enterprise-scale transformations
The challenge of keeping up with evolving APIs and SDKs becomes particularly acute in cloud-based environments where services are continuously improving. AWS Transform custom supports AWS SDK updates across a broad spectrum of programming languages that enterprises use including Java, Python, and JavaScript. The service understands not only the mechanical aspects of API changes, but also recognizes best practices and optimization opportunities available in newer SDK versions.

Infrastructure as Code transformations represent another critical capability, especially as organizations evaluate different tooling strategies. Whether you’re converting AWS Cloud Development Kit (AWS CDK) templates to Terraform for standardization purposes, or updating AWS CloudFormation configurations to access new service features, AWS Transform custom understands the declarative nature of these tools and can maintain the intent and structure of your infrastructure definitions.

Beyond these common scenarios, AWS Transform custom excels at addressing the unique, organization-specific code patterns that accumulate over years of development. Every enterprise has its own architectural conventions, utility libraries, and coding standards that need to evolve over time. It can learn these custom patterns and help refactor them systematically so that institutional knowledge and best practices are applied consistently across the entire application portfolio.

AWS Transform custom is designed with enterprise development workflows in mind, enabling center of excellence teams and system integrators to define and execute organization-wide transformations while application developers focus on reviewing and integrating the transformed code. DevOps engineers can then configure integrations with existing continuous integration and continuous delivery (CI/CD) pipelines and source control systems. It also includes pre-built transformations for Java, Node.js and Python runtime updates which can be particularly useful for AWS Lambda functions, along with transformations for AWS SDK modernization to help teams get started immediately.

Getting started
AWS Transform makes complex code transformations manageable through both pre-built and custom transformation capabilities. Let’s start by exploring how to use an existing transformation to address a common modernization challenge: upgrading AWS Lambda functions due to end-of-life (EOL) runtime support.

For this example, I’ll demonstrate migrating a Python 3.8 Lambda function to Python 3.13, as Python 3.8 reached EOL and is no longer receiving security updates. I’ll use the CLI for this demo, but I encourage you to also explore the web interface’s powerful campaign management capabilities.

First, I use the command atx custom def list to explore the available transformation definitions. You can also access this functionality through a conversational interface by typing only atx instead of issuing the command directly, if you prefer.

This command displays all available transformations, including both AWS-managed defaults and any existing custom transformations created by users in my organization. AWS-managed transformations are identified by the AWS/ prefix, indicating they’re maintained and updated by AWS. In the results, I can see several options such as AWS/java-version-upgrade for Java runtime modernization, AWS/python-boto2-to-boto3-migration for updating Python AWS SDK usage, AWS/nodejs-version-upgrade for Node.js runtime updates.

For my Python 3.8 to 3.13 migration, I’ll use the AWS/python-version-upgrade transformation.

You run a migration by using the atx custom def exec command.  Please consult the documentation for more details about the command and all its options. Here, I run it against my project repository specifying the transformation name. I also add pytest to run unit tests for validation. More importantly, I use the additionalPlanContext section in the  --configuration input to specify which Python version I want to upgrade to. For reference, here’s the command I have for my demo (I’ve used multiple lines and indented it here for clarity):

atx custom def exec 
-p /mnt/c/Users/vasudeve/Documents/Work/Projects/ATX/lambda/todoapilambda 
-n AWS/python-version-upgrade
-C "pytest" 
--configuration 
    "additionalPlanContext= The target Python version to upgrade to is Python 3.13" 
-x -t

AWS Transform then starts the migration process. It analyzes my Lambda function code, identifies Python 3.8-specific patterns, and automatically applies the necessary changes for Python 3.13 compatibility. This includes updating syntax for deprecated features, modifying import statements, and adjusting any version-specific behaviors.

After execution, it provides a comprehensive summary including a report on dependencies updated in requirements.txt with Python 3.13-compatible package versions, instances of deprecated syntax replaced with current equivalents, updated runtime configuration notes for AWS Lambda deployment, suggested test cases to validate the migration, and more. It also provides a body of evidence that serve as proof of success.

The migrated code lives in a local branch so you can review and merge when satisfied. Alternatively, you can keep providing feedback and reiterating until yo’re happy that the migration is fully complete and meets your expectations.

This automated process changes what would typically require hours of manual work into a streamlined, consistent upgrade that maintains code quality while maintaining compatibility with the newer Python runtime.

Creating a new custom transformation
While AWS-managed transformations handle common scenarios effectively, you can also create custom transformations tailored to your organization’s specific needs. Let’s explore how to create a custom transformation to see how AWS Transform learns from your specific requirements.

I type atx to initialize the atx cli and start the process.

The first thing it asks me is if I want to use one of the existing transformations or create a new one. I choose to create a new one. Notice that from here on the whole conversation takes place using natural language, not commands. I typed new one but I could have typed I want to create a new one and it would’ve understood it exactly the same.

It then prompts me to provide more information about the kind of transformation I’d like to perform. For this demo, I’m going to migrate an Angular application, so I type angular 16 to 19 application migration which prompts the CLI to search for all transformations available for this type of migration. In my case, my team has already created and made available a few Angular migrations, so it shows me those. However, it warns me that none of them is an exact match to my specific request for migrating from Angular 16 to 19. It then asks if I’d like to select from one of the existing transformations listed or create a custom one.

I choose to create a custom one by continuing to use natural language and typing create a new one as a command. Again, this could be any variation of that statement provided that you indicate your intentions clearly. It follows by asking me a few questions including whether I have any useful documentation, example code or migration guides that I can provide to help customize the transformation plan.

For this demo, I’m only going to rely on AWS Transform to provide me with good defaults. I type I don't have these details. Follow best practices. and the CLI responds by telling me that it will create a comprehensive transformation definition for migrating Angular 16 to Angular 19.  Of course, I relied on the pre-trained data to generate results based on best practices. As usual, the recommendation is to provide as much information and relevant data as possible at this stage of the process for better results. However, you don’t need to have all the data upfront. You can keep on providing data at any time› as you iterate through the process of creating the custom transformation definition.

The transformation definition is generated as a markup file containing a summary and a comprehensive sequence of implementation steps grouped logically into phases such as premigration preparation, processing and partitioning, static dependency analysis, searching and applying specific transformation rules, and step-by-step migration and iterative validation.

It’s interesting to see that AWS Transform opted for the best practice of doing incremental framework updates creating steps for migrating the application first to 17 then 18 then 19 instead of trying to go directly from 16 to 19 to minimize issues.

Note that the plan includes various stages of testing and verification to confirm that the various phases can be concluded with confidence. At the very end, it also includes a final validation stage listing exit criteria that performs a comprehensive set of tests against all aspects of the application that will be used to accept the migration as successfully complete.

After the transformation definition is created, AWS Transform asks me about what I would like to do next. I can choose to review or modify the transformation definition and I can reiterate through this process as much as I need until I arrive at one that I’m satisfied with. I can also choose to already apply this transformation definition to an Angular codebase. However, first I want to make this transformation available to my team members as well as myself so we can all use it again in the future. So, I choose option 4 to publish this transformation to the registry.

This custom transformation needs a name and a description of its objective which is displayed when users browse the registry. AWS Transforms automatically extracts those from context for me and asks me if I would like to modify them before going ahead. I like the sensible default of “Angular-16-to-19-Migration”, and the objective is clearly stated, so I choose to accept the suggestions and publish it by answering with yes, looks good.

Now that the transformation definition is created and published, I can use it and run it multiple times against any code repository. Let’s apply the transformation to a code repository with a project written in Angular 16. I now choose option 1 from the follow-up prompt and the CLI asks me for the path in my file system to the application that I want to migrate and, optionally, the build command that it should use.

After I provide that information, AWS Transform proceeds to analyze the code base and formulate a thorough step-by-step transformation plan based on the definition created earlier. After it’s done, it creates a JSON file containing the detailed migration plan specifically designed for applying our transformation definition to this code base. Similar to the process of creating the transformation definition, you can review and iterate through this plan as much as you need, providing it with feedback and adjusting it to any specific requirements you might have.

When I’m ready to accept the plan, I can use natural language to tell AWS Transform that we can start the migration process. I type looks good, proceed and watch the progress in my shell as it starts executing the plan and making the changes to my code base one step at a time.

The time it takes will vary depending on the complexity of the application. In my case, it took a few minutes to complete. After it has finished, it provides me with a transformation summary and the status of each one of the exit criteria that were included in the final verification phase of the plan alongside all the evidence to support the reported status. For example, the Application Build – Production criteria was listed as passed and some of the evidence provided included the incremental Git commits, the time that it took to complete the production build, the bundle size, the build output message, and the details about all the output files created.

Conclusion
AWS Transform represents a fundamental shift in how organizations approach code modernization and technical debt. The service helps to transform what was at one time a fragmented, team-by-team effort into a unified, intelligent capability that eliminates knowledge silos, keeping your best practices and institutional knowledge available as scalable assets across the entire organization. This helps to accelerate modernization initiatives while freeing developers to spend more time on innovation and driving business value instead of focusing on repetitive maintenance and modernization tasks.

Things to know

AWS Transform custom is now generally available. Visit the get started guide to start your first transformation campaign or check out the documentation to learn more about setting up custom transformation definitions.

Introducing AWS Lambda Managed Instances: Serverless simplicity with EC2 flexibility

Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/introducing-aws-lambda-managed-instances-serverless-simplicity-with-ec2-flexibility/

Today, we’re announcing AWS Lambda Managed Instances, a new capability you can use to run AWS Lambda functions on your Amazon Elastic Compute Cloud (Amazon EC2) compute while maintaining serverless operational simplicity. This enhancement addresses a key customer need: accessing specialized compute options and optimizing costs for steady-state workloads without sacrificing the serverless development experience you know and love.

Although Lambda eliminates infrastructure management, some workloads require specialized hardware, such as specific CPU architectures, or cost optimizations from Amazon EC2 purchasing commitments. This tension forces many teams to manage infrastructure themselves, sacrificing the serverless benefits of Lambda only to access the compute options or pricing models they need. This often leads to a significant architectural shift and greater operational responsibility.

Lambda Managed Instances
You can use Lambda Managed Instances to define how your Lambda functions run on EC2 instances. Amazon Web Services (AWS) handles setting up and managing these instances in your account. You get access to the latest generation of Amazon EC2 instances, and AWS handles all the operational complexity—instance lifecycle management, OS patching, load balancing, and auto scaling. This means you can select compute profiles optimized for your specific workload requirements, like high-bandwidth networking for data-intensive applications, without taking on the operational burden of managing Amazon EC2 infrastructure.

Each execution environment can process multiple requests rather than handling just one request at a time. This can significantly reduce compute consumption, because your code can efficiently share resources across concurrent requests instead of spinning up separate execution environments for each invocation. Lambda Managed Instances provides access to Amazon EC2 commitment-based pricing models such as Compute Savings Plans and Reserved Instances, which can provide up to a 72% discount over Amazon EC2 On-Demand pricing. This offers significant cost savings for steady-state workloads while maintaining the familiar Lambda programming model.

Let’s try it out
To take Lambda Managed Instances for a spin, I first need to create a Capacity provider. As shown in the following image, there is a new tab for creating these in the navigation pane under Additional resources.

Lambda Managed Instances Console

Creating a Capacity provider is where I specify the virtual private cloud (VPC), subnet configuration and security groups. With a capacity provider configuration, I can also tell Lambda where to provision and manage the instances.

I can also specify the EC2 instance types I’d like to include or exclude, or I can choose to include all instance types for high diversity. Additionally, I can specify a few controls related to auto scaling, including the Maximum vCPU count, and if I want to use Auto scaling or use a CPU policy.

After I have my capacity provider configured, I can choose it through its Amazon Resource Name (ARN) when I go to create a new Lambda function. Here I can also select the memory allocation I want along with a memory-to-vCPU ratio.

Working with Lambda Managed Instances
Now that we’ve seen the basic setup, let’s explore how Lambda Managed Instances works in more detail. The feature organizes EC2 instances into capacity providers that you configure through the Lambda console, AWS Command Line Interface (AWS CLI), or infrastructure as code (IaC) tools such as AWS CloudFormation, AWS Serverless Application Model (AWS SAM), AWS Cloud Development Kit (AWS CDK) and Terraform. Each capacity provider defines the compute characteristics you need, including instance type, networking configuration, and scaling parameters.

When creating a capacity provider, you can choose from the latest generation of EC2 instances to match your workload requirements. For cost-optimized general-purpose compute, you could choose AWS Graviton4 based instances that deliver excellent price performance. If you’re not sure which instance type to select, AWS Lambda provides optimized defaults that balance performance and cost based on your function configuration.

After creating a capacity provider, you attach your Lambda functions to it through a straightforward configuration change. Before attaching a function, you should review your code for programming patterns that can cause issues in multiconcurrency environments, such as writing to or reading from file paths that aren’t unique per request or using shared memory spaces and variables across invocations.

Lambda automatically routes requests to preprovisioned execution environments on the instances, eliminating cold starts that can affect first-request latency. Each execution environment can handle multiple concurrent requests through the multiconcurrency feature, maximizing resource utilization across your functions. When additional capacity is needed during traffic increases, AWS automatically launches new instances within tens of seconds and adds them to your capacity provider. The capacity provider can absorb traffic spikes of up to 50% without needing to scale by default, but built-in circuit breakers protect your compute resources during extreme traffic surges by temporarily throttling requests with 429 status codes if the capacity provider reaches maximum provisioned capacity and additional capacity is still being spun up.

The operational and architectural model remains serverless throughout this process. AWS handles instance provisioning, OS patching, security updates, load balancing across instances, and automatic scaling based on demand. AWS automatically applies security patches and bug fixes to operating system and runtime components, often without disrupting running applications. Additionally, instances have a maximum 14-day lifetime to align with industry security and compliance standards. You don’t need to write automatic scaling policies, configure load balancers, or manage instance lifecycle yourself, and your function code, event source integrations, AWS Identity and Access Management (AWS IAM) permissions, and Amazon CloudWatch monitoring remain unchanged.

Now available
You can start using Lambda Managed Instances today through the Lambda console, AWS CLI, or AWS SDKs. The feature is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland) Regions. For Regional availability and future roadmap, visit the AWS Capabilities by Region. Learn more about it in the AWS Lambda documentation.

Pricing for Lambda Managed Instances has three components. First, you pay standard Lambda request charges of $0.20 per million invocations. Second, you pay standard Amazon EC2 instance charges for the compute capacity provisioned. Your existing Amazon EC2 pricing agreements, including Compute Savings Plans and Reserved Instances, can be applied to these instance charges to reduce costs for steady-state workloads. Third, you pay a compute management fee of 15% calculated on the EC2 on-demand instance price to cover AWS’s operational management of your instances. Note that unlike traditional Lambda functions, you are not charged separately for execution duration per request. The multiconcurrency feature helps further optimize costs by reducing the total compute time required to process your requests.

The initial release supports the latest versions of Node.js, Java, .NET and Python runtimes, with support for other languages coming soon. The feature integrates with existing Lambda workflows including function versioning, aliases, AWS CloudWatch Lambda Insights, AWS AppConfig extensions, and deployment tools like AWS SAM and AWS CDK. You can migrate existing Lambda functions to Lambda Managed Instances without changing your function code (as long as it has been validated to be thread safe for multiconcurrency) making it easy to adopt this capability for workloads that would benefit from specialized compute or cost optimization.

Lambda Managed Instances represents a significant expansion of Lambda’s capabilities, which means you can run a broader range of workloads while preserving the serverless operational model. Whether you’re optimizing costs for high-traffic applications, or accessing the latest processor architectures like Graviton4, this new capability provides the flexibility you need without operational complexity. We’re excited to see what you build with Lambda Managed Instances.

Node.js 24 runtime now available in AWS Lambda

Post Syndicated from Andrea Amorosi original https://aws.amazon.com/blogs/compute/node-js-24-runtime-now-available-in-aws-lambda/

You can now develop AWS Lambda functions using Node.js 24, either as a managed runtime or using the container base image. Node.js 24 is in active LTS status and ready for production use. It is expected to be supported with security patches and bugfixes until April 2028.

The Lambda runtime for Node.js 24 includes a new implementation of the Runtime Interface Client (RIC), which integrates your functions code with the Lambda service. Written in TypeScript, the new RIC streamlines and simplifies Node.js support in Lambda, removing several legacy features. In particular, callback-based function handlers are no longer supported.

Node.js 24 includes several additions to the language, such as Explicit Resource Management, as well as changes to the runtime implementation and the standard library. With this release, Node.js developers can take advantage of these new features and enhancements when creating serverless applications on Lambda.

You can develop Node.js 24 Lambda functions using the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS SDK for JavaScript, AWS Serverless Application Model (AWS SAM), AWS Cloud Development Kit (AWS CDK), and other infrastructure as code tools. You can use Node.js 24 with Powertools for AWS Lambda (TypeScript), a developer toolkit to implement serverless best practices and increase developer velocity. Powertools includes libraries to support common tasks such as observability, AWS Systems Manager Parameter Store integration, idempotency, batch processing, and more. You can also use Node.js 24 with Lambda@Edge to customize low-latency content delivered through Amazon CloudFront.

This blog post highlights important changes to the Node.js runtime, notable Node.js language updates, and how you can use the new Node.js 24 runtime in your serverless applications.

Node.js 24 runtime changes

The Lambda Runtime for Node.js 24 includes the following changes relative to the Node.js 22 and earlier runtimes.

Removing support for callback-based function handlers

Starting with the Node.js 24 runtime, Lambda no longer supports the callback-based handler signature for asynchronous operations. Callback-based handlers take three parameters, with the third parameter a callback. For example:

export const handler = (event, context, callback) => {
    try {
        // Some processing...
        
        // Success case
        // First parameter (error) is null, second is the result
        callback(null, {
            statusCode: 200,
            body: JSON.stringify({
                message: "Operation completed successfully"
            })
        });
        
    } catch (error) {
        // Error case
        // First parameter contains the error
        callback(error);
    }
};

The modern approach to asynchronous programming in Node.js is to use the async/await pattern. Lambda introduced support for async handlers with the Node.js 8 runtime, launched in 2018. Here’s how the above function looks when using an async handler:

export const handler = async (event, context) => {
    try {
	  // Some processing
        
        return {
            statusCode: 200,
            body: JSON.stringify({
                message: "Operation completed successfully"
            })
        };
        
    } catch (error) {
        throw error;
    }
};

The Node.js 24 runtime still supports synchronous function handlers that do not use callbacks:

export const handler = (event, context) => {
    // Perform some synchronous data processing
    // Return response
    return {
        statusCode: 200,
        body: JSON.stringify(response)
    };
};

And Node.js 24 still supports response streaming, enabling more responsive applications by accelerating the time-to-first-byte:

export const handler = awslambda.streamifyResponse(async (event, responseStream, context) => {
    // Convert event to a readable stream
    const requestStream = Readable.from(Buffer.from(JSON.stringify(event)));
    // Stream the response using pipeline
    await pipeline(requestStream, responseStream);
});

This change to remove support for callback-based function handlers only affects Node.js 24 (and later) runtimes. Existing runtimes for Node.js 22 and earlier continue to support callback-based function handlers. When migrating functions that use callback-based handlers to Node.js 24, you need to modify your code to use one of the supported function handler signatures

As part of this change, context.callbackWaitsForEmptyEventLoop is removed. In addition, the previously deprecated context.succeed, context.fail, and context.done methods have also been removed. This aligns the runtime with modern Node.js patterns for clearer, more consistent error and result handling.

Harmonizing streaming and non-streaming behavior for unresolved promises

The Node.js 24 runtime also resolves a previous inconsistency in how unresolved promises were handled. Previously, Lambda would not wait for unresolved promises once the handler returns except when using response streaming. Starting with Node.js 24, the response streaming behavior is now consistent with non-streaming behavior, and Lambda no longer waits for unresolved promises once your handler returns or the response stream ends. Any background work (for example, pending timers, fetches, or queued callbacks) is not awaited implicitly. If your response depends on additional asynchronous operations, ensure you await them in your handler or integrate them into the streaming pipeline before closing the stream or returning, so the response only completes after all required work has finished.

Experimental Node.js features

Node.js enables certain experimental features by default in the upstream language releases. Such features include support for importing modules using require() in ECMAScript modules (ES modules) and automatically detecting ES vs CommonJS modules. As they are experimental, these features may be unstable or undergo breaking changes in future Node.js updates. To provide a stable experience, Lambda disables these features by default in the corresponding Lambda runtimes.

Lambda allows you to re-enable these features by adding the --experimental-require-module flag or the --experimental-detect-module flag to the NODE_OPTIONS environment variable. Enabling experimental Node.js features may affect performance and stability, and these features can change or be removed in future Node.js releases; such issues are not covered by AWS Support or the Lambda SLA.

ES modules in CloudFormation inline functions

With AWS CloudFormation inline functions, you provide your function code directly in the CloudFormation template. They’re particularly useful when deploying custom resources. With inline functions, the code filename is always index.js, which by default Node.js interprets as a CommonJS module. With the Node.js 24 runtime, you can use ES modules when authoring inline functions by passing the --experimental-detect-module flag via the NODE_OPTIONS environment variable. Previously, you needed a zip or container package to use ES modules. With Node.js 24, you can write inline functions using standard ESM syntax (import/export) and top‑level await), which simplifies small utilities and bootstrap logic without requiring a packaging step.

Node.js 24 language features

Node.js 24 introduces several language updates and features that enhance developer productivity and improve application performance.

Node.js 24 includes Undici 7, a newer version of the HTTP client that powers global ⁠fetch. This version brings performance improvements and broader protocol capabilities. Network‑heavy Lambda functions that call AWS services or external APIs can benefit from better connection management and throughput, especially when reusing clients or using HTTP/2 where supported. Most applications should work without changes, but you should validate behavior for advanced scenarios, such as custom headers or streaming bodies, and continue to define HTTP clients outside of the handler to maximize connection reuse across invocations.

The JavaScript Explicit Resource Management syntax (⁠using and ⁠await using) enables deterministic clean-up of resources when a block completes. For Lambda handlers, this makes it easier to ensure short‑lived objects, such as streams, temporary buffers, or file handles, are disposed of promptly, which reduces the risk of resource leaks across warm invocations. You should continue to define long‑lived clients, for example SDK clients or database pools, outside the handler to benefit from connection reuse, and apply explicit disposal only to resources you want to tear down at the end of each invocation.

Finally, the ⁠AsyncLocalStorage API now uses ⁠AsyncContextFrame by default, improving the performance and reliability of async context propagation. This benefits common serverless patterns such as timers, correlating logs, managing tracing IDs and request‑scoped metadata across async and await boundaries, and streams without manual parameter threading. If you already use ⁠AsyncLocalStorage‑based libraries for logging or observability, you may see lower overhead and more consistent context propagation in Node.js 24.

For a detailed overview of Node.js 24 language features, see the Node.js 24 release blog post and the Node.js 24 changelog.

Performance considerations

At launch, new Lambda runtimes receive less usage than existing established runtimes. This can result in longer cold start times due to reduced cache residency within internal Lambda sub-systems. Cold start times typically improve in the weeks following launch as usage increases. As a result, AWS recommends not drawing conclusions from side-by-side performance comparisons with other Lambda runtimes until the performance has stabilized. Since performance is highly dependent on workload, customers with performance-sensitive workloads should conduct their own testing, instead of relying on generic test benchmarks.

Builders should continue to measure and test function performance and optimize function code and configuration for any impact. To learn more about how to optimize Node.js performance in Lambda, see our blog post Optimizing Node.js dependencies in AWS Lambda.

Migration from earlier Node.js runtimes

We’ve already discussed changes that are new to the Node.js 24 runtime, such as removing support for callback-based function handlers. As a reminder, we’ll recap some previous changes for customers upgrading from older Node.js functions.

AWS SDK for JavaScript

Up until Node.js 16, Lambda’s Node.js runtimes included the AWS SDK for JavaScript version 2. This has since been superseded by the AWS SDK for JavaScript version 3, which was released in December 2024. Starting with Node.js 18, and continuing with Node.js 24, the Lambda Node.js runtimes include version 3. When upgrading from Node.js 16 or earlier runtimes and using the included version 2, you must upgrade your code to use the v3 SDK.

For optimal performance, and to have full control over your code dependencies, we recommend bundling and minifying the AWS SDK in your deployment package, rather than using the SDK included in the runtime. For more information, see Optimizing Node.js dependencies in AWS Lambda.

Amazon Linux 2023

The Node.js 24 runtime is based on the provided.al2023 runtime, which is based on the Amazon Linux 2023 minimal container image. The Amazon Linux 2023 minimal image uses microdnf as a package manager, symlinked as dnf. This replaces the yum package manager used in Node.js 18 and earlier AL2-based images. If you deploy your Lambda function as a container image, you must update your Dockerfile to use dnf instead of yum when upgrading to the Node.js 24 base image from Node.js 18 or earlier.

Learn more about the provided.al2023 runtime in the blog post Introducing the Amazon Linux 2023 runtime for AWS Lambda and the Amazon Linux 2023 launch blog post.

Using the Node.js 24 runtime in AWS Lambda

Finally, we’ll review how to configure your functions to use Node.js 24, using a range of deployment tools.

AWS Management Console

When using the AWS Lambda Console, you can choose Node.js 24.x in the Runtime dropdown when creating a function:

Creating Node.js function in the AWS Management Console

Creating Node.js function in the AWS Management Console

To update an existing Lambda function to Node.js 24, navigate to the function in the Lambda console, click Edit in the Runtime settings panel, then choose Node.js 24.x from the Runtime dropdown:

Editing Node.js function runtime

Editing Node.js function runtime

AWS Lambda container image

Change the Node.js base image version by modifying the FROM statement in your Dockerfile.

FROM public.ecr.aws/lambda/nodejs:24
# Copy function code
COPY lambda_handler.mjs ${LAMBDA_TASK_ROOT}

AWS Serverless Application Model

In AWS SAM, set the Runtime attribute to node24.x to use this version:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: nodejs24.x
      CodeUri: my_function/.
      Description: My Node.js Lambda Function

AWS SAM supports generating this template with Node.js 24 for new serverless applications using the sam init command. For more information, refer to the AWS SAM documentation.

AWS Cloud Development Kit (AWS CDK)

In AWS CDK, set the runtime attribute to Runtime.NODEJS_24_X to use this version.

import * as cdk from "aws-cdk-lib";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as path from "path";
import { Construct } from "constructs";
export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    // The code that defines your stack goes here
    // The Node.js 24 enabled Lambda Function
    const lambdaFunction = new lambda.Function(this, "node24LambdaFunction", {
      runtime: lambda.Runtime.NODEJS_24_X,
      code: lambda.Code.fromAsset(path.join(__dirname, "/../lambda")),
      handler: "index.handler",
    });
  }
}

Conclusion

AWS Lambda now supports Node.js 24 as a managed runtime and container base image. This release uses a new runtime interface client, removes support for callback-based function handlers, and includes several other changes to streamline and simplify Node.js support in Lambda.

You can build and deploy functions using Node.js 24 using the AWS Management Console, AWS CLI, AWS SDK, AWS SAM, AWS CDK, or your choice of infrastructure as code tool. You can also use the Node.js 24 container base image if you prefer to build and deploy your functions using container images.

To find more Node.js examples, use the Serverless Patterns Collection. For more serverless learning resources, visit Serverless Land