Post Syndicated from D Surya Sai original https://aws.amazon.com/blogs/compute/observability-best-practices-for-lambda-durable-functions-2/

When your workflow suspends to wait for a confirmation, you need to know whether the callback arrived, how long the function waited, and what to do if the callback never comes. AWS Lambda durable functions make these long-running, suspendable workflows straightforward to build, but answering those operational questions requires deliberate monitoring instrumentation across the suspension boundary.

In this post, we walk through observability best practices for Lambda durable functions using a Stripe payment processing pipeline as the example. We cover durable function-specific Amazon CloudWatch metrics, custom business metrics, alarms, structured logging, AWS X-Ray tracing, and how to debug a callback timeout end-to-end. By the end, you will have a reusable observability pattern for any durable function that suspends on external callbacks. The GitHub repository contains the complete implementation.

Architecture overview

Our application processes card payments through Stripe using three Lambda functions and Amazon API Gateway:

1. Payment API (payment-api): An API Gateway-backed function that accepts payment requests, asynchronously invokes the durable function, and exposes endpoints to check or cancel an in-flight execution.

2. Payment Processor (payment-processor): A durable function that validates the payment, creates a Stripe PaymentIntent, then suspends and waits for a callback confirming the payment outcome.

3. Webhook Handler (stripe-webhook): Receives Stripe webhook events, verifies the signature, and calls send_durable_execution_callback_success to resume the suspended durable execution with the payment result.

Figure 1: Payment processing flow with durable callback suspension, where the webhook handler sends the callback result back to the same suspended durable execution

The key observability challenge sits in the gap between the PaymentIntent creation (step 2) and the webhook delivery (step 3). During this period the durable function is suspended: it is consuming no compute, but it is waiting for Stripe to call back. If the webhook never arrives, the callback times out silently unless you have metrics and alarms watching for it. With proper instrumentation, you gain full visibility into this suspension gap and can diagnose issues within minutes.

You deploy the application with AWS Serverless Application Model (AWS SAM). The following template excerpt shows how we enable observability across the stack:

Globals:
  Function:
    Runtime: python3.13
    Tracing: Active # X-Ray on all functions
    Environment:
      Variables:
        POWERTOOLS_METRICS_NAMESPACE: DurablePayments
        LOG_LEVEL: INFO

Resources:
  PaymentApi:
    Type: AWS::Serverless::Api
    Properties:
      TracingEnabled: true # X-Ray on API Gateway

  PaymentProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      AutoPublishAlias: live
      DurableConfig:
        ExecutionTimeout: 600 # Bounds the whole workflow
        RetentionPeriodInDays: 5 # Keep execution history

Tracing: Active under Globals enables X-Ray across all functions, and TracingEnabled: true on the API resource ensures traces propagate from the initial request through the entire flow.

Durable function CloudWatch metrics, custom business metrics, and alarms

Lambda automatically emits CloudWatch metrics specific to durable executions, covering execution lifecycle, capacity utilization, duration including wait time, and cost drivers. For the full list, see Monitoring durable functions.

One metric worth calling out: DurableExecutionDuration measures total wall-clock time including the callback wait period. For a payment that takes 2 seconds to process but waits 30 seconds for a webhook, this metric reports approximately 32 seconds. This is distinct from the standard Duration metric, which only measures active compute time.

Custom business metrics for the callback funnel

The built-in metrics tell you whether executions succeeded or failed. To understand where in the business flow the issue occurred, we emit custom metrics at each stage using Powertools for AWS Lambda Metrics with Embedded Metric Format (EMF):

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit

metrics = Metrics(namespace="DurablePayments", service="payment-processor")

# In the durable handler, after each stage:
metrics.add_metric(name="PaymentIntentCreated", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="PaymentSucceeded", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="PaymentFailed", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="PaymentTimeout", unit=MetricUnit.Count, value=1)

In the webhook handler:

metrics.add_metric(name="WebhookReceived", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="WebhookSucceeded", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="WebhookSignatureFailure", unit=MetricUnit.Count, value=1)

These metrics create an end-to-end funnel:

PaymentRequested → PaymentIntentCreated → WebhookReceived → WebhookSucceeded → PaymentSucceeded

Any drop-off between stages pinpoints the problem. If PaymentIntentCreated is higher than WebhookReceived, Stripe is not delivering webhooks. If WebhookReceived is higher than WebhookSucceeded, signature verification is failing. No corresponding PaymentSucceeded for a PaymentIntentCreated means the callback timed out.

Alarms for callback failure modes

Durable functions with callbacks have specific failure modes: callbacks that never arrive, webhook signatures that fail verification, and executions that time out waiting. We define alarms for each:

DurableExecutionFailureAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    Namespace: AWS/Lambda
    MetricName: DurableExecutionFailed
    Dimensions:
      - Name: FunctionName
        Value: !Ref PaymentProcessorFunction
    Threshold: 1
    ComparisonOperator: GreaterThanOrEqualToThreshold
    TreatMissingData: notBreaching
    ...

PaymentTimeoutAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    Namespace: DurablePayments
    MetricName: PaymentTimeout
    Dimensions:
      - Name: service
        Value: payment-processor
    Threshold: 1
    ...

WebhookSignatureFailureAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    Namespace: DurablePayments
    MetricName: WebhookSignatureFailure
    Dimensions:
      - Name: service
        Value: stripe-webhook
    Threshold: 3

These alarm definitions are abbreviated for readability. Each alarm in the deployed template.yaml also sets Dimensions (scoping DurableExecutionFailed to the payment-processor function, and the custom metrics to their service). It also includes Statistic, Period, EvaluationPeriods, and AlarmActions/OKActions wired to an SNS topic. See the GitHub repository for the deployable definitions.

Unified dashboard

We combine built-in durable metrics, custom EMF metrics, and standard Lambda metrics into a single CloudWatch dashboard. The dashboard includes widgets for execution state, payment outcomes, end-to-end flow metrics, quota utilization, cost drivers, error breakdown, and API/webhook latency.

Figure 2: CloudWatch dashboard showing durable execution state, payment outcomes, end-to-end flow metrics, running executions and quota utilization

Figure 3: CloudWatch Alarms showing DurableExecutionFailures, PaymentTimeouts, and WebhookSignatureFailures alarm states

Tracing callbacks across the suspension boundary

When a durable function suspends at a callback, the execution pauses. An external system (Stripe) fires a webhook to your API Gateway, which invokes the webhook handler. The webhook handler then calls send_durable_execution_callback_success to deliver the result back to the suspended execution, which resumes and completes. The challenge is correlating these two separate invocations so you can reconstruct the full payment timeline from a single query.

Structured logging with correlation keys

Using Lambda Powertools Logger, we progressively append correlation keys as they become available. Each subsequent log entry automatically includes all previously appended keys:

from aws_lambda_powertools import Logger
from aws_durable_execution_sdk_python import (
    DurableContext, durable_execution, durable_step,
)
from aws_durable_execution_sdk_python.config import CallbackConfig, Duration
from aws_durable_execution_sdk_python.exceptions import CallbackError

logger = Logger(service="payment-processor")

@durable_execution
def handler(event, context: DurableContext):
    payment = context.step(validate_payment_request(event), name="validate-payment")
    logger.append_keys(customer_id=payment["customer_id"])

    callback = context.create_callback(
        name="stripe-payment-result",
        config=CallbackConfig(timeout=Duration.from_minutes(5)),
    )
    logger.info("Callback created", callback_id=callback.callback_id)

    intent = context.step(
        create_stripe_payment_intent(payment, callback.callback_id),
        name="create-payment-intent",
    )
    logger.append_keys(payment_intent_id=intent["payment_intent_id"])
    logger.info("Suspending, waiting for Stripe webhook callback")

    try:
        result = callback.result()  # Function suspends here
    except CallbackError:
        logger.warning("Payment timed out")
        return {"status": "timeout", "message": "No confirmation within 5 minutes"}

In the webhook handler, we append the same keys so a single Logs Insights query reconstructs the full timeline:

logger = Logger(service="stripe-webhook")

def handler(event, context):
    # ... verify signature, parse event
    logger.append_keys(event_type=event_type, payment_intent_id=payment_intent_id)
    logger.append_keys(callback_id=callback_id)
    logger.info("Processing webhook event")

Query across all three log groups for a single payment:

fields @timestamp, service, message, customer_id, payment_intent_id, callback_id
| filter payment_intent_id = "pi_3TJafD04vzZc6RmP0RrCWhix"
| sort @timestamp asc

Figure 4: CloudWatch Logs Insights query showing the timeline of a single payment across payment-api, payment-processor, and stripe-webhook

Durable steps and X-Ray annotations

The SDK’s @durable_step decorator checkpoints each step. If the function crashes and replays, completed steps return their cached result without re-executing. We combine this with Powertools Tracer to add searchable X-Ray annotations at each business-critical point:

from aws_durable_execution_sdk_python import StepContext, durable_step

@durable_step
@tracer.capture_method
def create_stripe_payment_intent(step_context: StepContext, payment: dict, callback_id: str) -> dict:
    tracer.put_annotation("callback_id", callback_id)
    tracer.put_annotation("customer_id", payment["customer_id"])

    try:
        intent = stripe.PaymentIntent.create(
            amount=payment["amount"], currency=payment["currency"],
            payment_method=payment["payment_method_id"], confirm=True,
            metadata={"callback_id": callback_id},
            automatic_payment_methods={"enabled": True, "allow_redirects": "never"},
            ...
        )
    except stripe.error.CardError as exc:
        # Hard declines (e.g. pm_card_chargeDeclined) raise synchronously. Return a
        # structured decline so the step doesn't retry and fail the whole execution.
        ...
        metrics.add_metric(name="PaymentDeclinedAtCreate", unit=MetricUnit.Count, value=1)
        return {"declined": True, ...}  # decline_code, error_message, payment_intent_id

    metrics.add_metric(name="PaymentIntentCreated", unit=MetricUnit.Count, value=1)
    ...
    return {"payment_intent_id": intent.id, "status": intent.status}

Note: The preceding code is abbreviated for readability. Refer to the GitHub repository for the complete code. The main durable handler runs within a FacadeSegment X-Ray context that does not support put_annotation(). Annotations work normally inside @durable_step functions. In the main handler, use a try/except wrapper if you need annotations outside of steps.

Note: When calling PaymentIntent.create with confirm=True, some cards decline synchronously (no webhook fires). The deployed code handles this by detecting the decline in the step return value and skipping the callback suspension, preventing an indefinite wait.

The X-Ray Service Map shows the complete request flow: API Gateway to payment-api to payment-processor, and the separate webhook path from API Gateway to stripe-webhook.

Figure 5: X-Ray Service Map showing API Gateway connected to payment-api and stripe-webhook, with payment-api connected to payment-processor

Durable executions tab

The Lambda console provides a built-in Durable executions tab showing each execution’s step-by-step timeline, including the callback wait state. You can see which steps completed, where the function suspended, and when (or if) the callback arrived.

Figure 6: Lambda console Durable executions tab showing a completed execution with steps: validate-payment succeeded, create-payment-intent succeeded, stripe-payment-result callback received, and final result succeeded

Putting it together: debugging real failure modes

The following three scenarios demonstrate how all of these observability layers work together. You can reproduce each one from the demo checkout page.

Scenario 1: Webhook never arrives

A customer reports that their payment was charged but they never received a confirmation.

1. Alarm fires. The PaymentTimeoutAlarm triggers, indicating a durable execution timed out waiting for a callback.

2. Check the dashboard. The Payment Outcomes widget shows a spike in PaymentTimeout. The End-to-End Flow Metrics widget reveals the drop-off: PaymentIntentCreated count is higher than WebhookReceived, meaning the webhook never arrived.

3. Query logs. Search Amazon CloudWatch Logs Insights for the timed-out payment:

fields @timestamp, service, message, payment_intent_id, callback_id
| filter message = "Payment timed out"
| sort @timestamp desc
| limit 5

This returns the payment_intent_id of the timed-out payment.

4. Cross-reference the webhook handler. Search for that payment_intent_id in the webhook handler logs. No results means Stripe never delivered the webhook. Results with WebhookSignatureFailure mean the webhook secret is misconfigured.

5. Inspect the X-Ray trace. Filter traces by the payment_intent_id annotation. The trace shows the durable function start but no corresponding webhook handler span, confirming the webhook never arrived.

6. Check the durable executions tab. The execution shows validate-payment and create-payment-intent as succeeded, with the stripe-payment-result callback in a timed-out state.

Figure 7: Durable executions tab showing the timed-out execution: validate-payment succeeded, create-payment-intent succeeded, stripe-payment-result callback timed out

Within minutes, you have identified the root cause (the Stripe webhook endpoint was misconfigured) without adding a single debug statement or redeploying code.

Scenario 2: The whole workflow runs too long

The callback timeout in Scenario 1 is a per-callback bound (5 minutes in this example). There is also an outer bound: DurableConfig.ExecutionTimeout (600 seconds), which caps the total wall-clock time of the whole execution. If you set a callback to wait an hour but the overall ExecutionTimeout is 10 minutes, the execution itself terminates first. This shows up as a distinct terminal state in the durable executions tab, on the Durable Execution State widget, and as its own alarm (DurableExecutionTimedOutAlarm).

Choose the “Simulate timeout (no webhook)” option on the demo checkout page to reproduce this. The durable function skips the Stripe call, suspends on a long-timeout callback, and lets ExecutionTimeout catch it. The dashboard distinguishes the two failure modes cleanly: per-callback timeouts show up on the custom Payment Outcomes widget as PaymentTimeout. Whole-execution timeouts appear on the built-in Durable Execution State widget alongside started/succeeded/failed counts. This distinction matters operationally because the remediation is different: callback timeouts point to external system issues (Stripe), while execution timeouts point to configuration issues (your timeout values).

Scenario 3: Customer abandons checkout

Real checkout flows have a third outcome: the customer cancels while the durable function is still suspended. The demo wires this up to StopDurableExecution, which terminates the in-flight execution and surfaces on the same Durable Execution State widget as a separate terminal state.

Choose “Simulate timeout” and then “Cancel Payment” on the demo page to see this happen. Looking at the dashboard after running all three scenarios, the execution-state widget tells the full story: started, succeeded, failed, timed-out, and stopped. Each state answers a different operational question about what is happening to your workflows.

Conclusion

In this post, we walked through observability best practices for Lambda durable functions using a Stripe payment processing pipeline. Callbacks can time out, whole executions can expire, and running workflows can be canceled. Each shows up as a distinct terminal state, and each deserves its own alarm. Layering custom business metrics, structured logging with correlation keys, X-Ray annotations, and the durable executions tab on top of the built-in CloudWatch metrics gives you a clear picture of where in the lifecycle any given execution is. It also reveals where in the business funnel any failure occurred.

Deploy the payment processing application from the GitHub repository and try the three demo scenarios to see the dashboards, alarms, and execution history in your own account. For core concepts, see Lambda durable functions. For the durable execution SDK, see the Python SDK, JavaScript SDK, and Java SDK. Browse Serverless Land for reference architectures.