Post Syndicated from Giedrius Praspaliauskas original https://aws.amazon.com/blogs/compute/customize-amazon-api-gateway-destinations-for-execution-logs/

Amazon API Gateway execution logs help you trace request processing step by step through your REST API stages. They capture authorization results, integration latency, mapping template output, and error details that are otherwise invisible at the API surface. When a production request fails in a way the access log cannot explain, the execution log is usually where you find the explanation.

Until now, execution logs had two constraints. Every log event was truncated at 1 KB, so a request carrying a moderately sized JSON body would exceed that limit and the remainder was dropped. Logs could only go to the auto-managed log group that API Gateway creates for you (API-Gateway-Execution-Logs_{rest-api-id}/{stage_name}).

With Amazon CloudWatch Logs delivery for REST API execution logs, you can now route execution logs to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose. Log events can be up to 1 MB per entry, and you benefit from vended logs pricing.

In this post, you learn how CloudWatch Logs delivery works with API Gateway execution logs, how to configure it, and what patterns work best for common observability scenarios.

Understanding API Gateway execution logs

API Gateway produces two categories of logs: access logs and execution logs. Access logs record a summary line per request, similar to an HTTP server access log. You configure the format and destination yourself.

Execution logs are different. They capture the internal processing of each request as it moves through the API Gateway pipeline: authorizer evaluation, request validation, integration dispatch, response mapping, and error handling. These logs exist so you can answer questions such as “why did my authorizer reject this token?” or “what did the mapping template produce before it reached my backend integration?”

API Gateway manages execution log creation automatically. When you set loggingLevel to INFO or ERROR in your stage’s method settings, the service writes execution log events to a CloudWatch Logs log group it manages on your behalf. You do not choose the log group name or configure retention directly on it.

The auto-managed model works for many customers but may create friction for teams with specific observability requirements. Compliance frameworks that require logs in S3 with a particular prefix structure need an extra subscription filter and delivery mechanism. Sending execution logs into a security information and event management (SIEM) tool through a Firehose stream requires a forwarding layer.

Configurable log delivery with CloudWatch Logs

CloudWatch Logs delivery separates log routing from log content. Two concepts control the behavior:

DeliverySource is scoped to your API Gateway stage ARN. It defines where logs go. You create a delivery source, then attach one or more delivery destinations (CloudWatch Logs log group, S3 bucket, or Firehose stream).

MethodSettings controls what gets logged. The loggingLevel setting (INFO, ERROR, or OFF) and dataTraceEnabled flag still determine which log events API Gateway produces. These settings work the same way regardless of whether you use the auto-managed log group or CloudWatch Logs delivery.

When you create a delivery using the CloudWatch Logs APIs, CloudWatch Logs activates your log delivery on your API Gateway stage. When you delete the delivery, CloudWatch Logs disables it accordingly. You do not need to flip any flags on the API Gateway side, and the execution logs automatically resume flowing to the auto-managed log group.

Your existing method settings keep their meaning. The loggingLevel and dataTraceEnabled values continue to control log content. If loggingLevel is already INFO or ERROR, creating a delivery redirects those logs to your chosen destination with no further configuration.

The following diagram shows how the pieces fit together.

Figure 1 — A single delivery source scoped to an API Gateway stage feeds one or more deliveries, each of which writes to a delivery destination backed by CloudWatch Logs, Amazon S3, or Amazon Data Firehose

The following table summarizes what changes when log delivery is active.

What stays the same

Only execution log routing changes. Access logs continue to flow through accessLogSettings to whatever log group you configure, and unrelated stage features such as AWS X-Ray tracing, detailed CloudWatch metrics, throttling, and caching behave exactly as they did before.

Configuration and integration options

Before you create a delivery, confirm the following requirements:

• The API Gateway REST API is deployed to a stage.
• loggingLevel is set to INFO or ERROR in MethodSettings.
• The account-level CloudWatch Logs IAM role is configured. For setup steps, see Set up CloudWatch logging for REST APIs in API Gateway.
• For cross-account delivery, the destination has an appropriate resource policy attached through PutDeliveryDestinationPolicy.

Sending logs to a custom CloudWatch Logs log group

The most common starting point is redirecting execution logs to a log group you own. You get direct control over retention policies, metric filters, and subscription filters. The following steps use the AWS Command Line Interface (AWS CLI) with the fictitious REST API ID abc123, stage prod, Region us-east-1, and account 111122223333.

1. Create a delivery source referencing your stage ARN. The log type for REST API execution logs is EXECUTION_LOGS:

   aws logs put-delivery-source \
       --name my-apigw-execution-logs \
       --resource-arn arn:aws:apigateway:us-east-1:111122223333:/restapis/abc123/stages/prod \
       --log-type EXECUTION_LOGS

2. Create a delivery destination pointing to your custom (existing) log group, then create the delivery that connects them:

   aws logs put-delivery-destination \
       --name my-execution-log-destination \
       --delivery-destination-configuration \
           destinationResourceArn=arn:aws:logs:us-east-1:111122223333:log-group:/my-api/execution-logs

   aws logs create-delivery \
       --delivery-source-name my-apigw-execution-logs \
       --delivery-destination-arn arn:aws:logs:us-east-1:111122223333:delivery-destination:my-execution-log-destination

3. Verify that the delivery is active by listing deliveries for the source:

   aws logs describe-deliveries

The response includes the delivery ID, source, and destination ARN after delivery is established. Execution logs flow to /my-api/execution-logs instead of the auto-managed group.

Note: Log delivery adds structured fields (resource_arn, event_timestamp, api_id, stage, resource_path, http_method, and payload) to each event, so a new delivery emits more than your previous logs. To keep the traditional execution log format with nothing extra, set output format and record fields while creating delivery destination and creating delivery:

aws logs put-delivery-destination \
    --output-format "plain" ...

aws logs create-delivery \
    --record-fields "payload" \
    --field-delimiter "" ...

Routing logs to Amazon S3

S3 works well for long-term retention at lower cost, or for feeding logs into analytics tools such as Amazon Athena. The bucket must be in the same region as your API. Create a delivery destination pointing to your bucket:

aws logs put-delivery-destination \
    --name s3-archive-destination \
    --delivery-destination-configuration \
        destinationResourceArn=arn:aws:s3:::amzn-s3-demo-apigw-logs

Then create a delivery using the same source name. CloudWatch Logs delivers the events to your bucket, where you can query them with Athena or catalog them with AWS Glue.

Streaming to Amazon Data Firehose

For real-time analytics pipelines or third-party SIEM integration, Firehose delivery sends execution log events directly to your stream. The setup is identical: create a delivery destination with your Firehose stream ARN, then create a delivery. With direct Firehose delivery, you no longer need to maintain CloudWatch Logs subscription filters and AWS Lambda forwarders to route execution logs to external analytics systems.

Multi-destination delivery and per-destination shaping

A single delivery source supports multiple destinations. You can route the same execution logs to CloudWatch Logs for real-time alerting, S3 for long-term compliance retention, and Firehose for your SIEM, all from one stage. Create additional deliveries using the same delivery source with different destination ARNs.

Each destination receives identical log events. To shape what reaches each destination, apply a CloudWatch Logs subscription filter on the CloudWatch Logs destination. For example, you can forward only ERROR-level events to a Lambda function that pushes alerts to a SIEM, while the same delivery source writes the full event stream to S3 for compliance.

Management console experience

You can also add a log delivery destination in the management console after you enable logging for the stage.

You can specify multiple destinations, both in the current or in a different account:

Keeping existing monitoring intact

If you have dashboards or alarms on the auto-managed log group, use that same log group as one of your delivery destinations. Your existing monitoring keeps working, and you gain the ability to send logs to additional destinations such as S3 or Firehose in parallel.

Best practices

Update dashboards and alarms before enabling log delivery. When you activate log delivery, the auto-managed log group stops receiving logs. Any CloudWatch alarms, dashboards, or Contributor Insights rules pointing to API-Gateway-Execution-Logs_{rest-api-id}/{stage_name} stop working. Migrate these references to your new log group before creating the delivery.

Keep loggingLevel at INFO or ERROR. Log delivery controls routing, not content. If loggingLevel is OFF, no execution log events are produced regardless of whether a delivery exists. Verify your method settings before troubleshooting missing logs.

Treat the 1 MB log event capacity as a security decision, not only a debugging convenience. With dataTraceEnabled set to true, execution logs include complete request and response payloads up to 1 MB. Those payloads might contain personally identifiable information (PII) or other sensitive data. Confirm your log destinations have appropriate access controls, encryption, and retention policies. Mask or filter sensitive fields in mapping templates upstream of logging and enable data tracing selectively per method or only in non-production stages.

Start with a single destination, then expand. Validate that your log group or bucket receives events correctly before adding Firehose or additional destinations.

Log delivery is best-effort. In rare cases, some log events might not be delivered. For audit-critical workloads, build retention and reconciliation that account for occasional missing events rather than treating execution logs as the system of record.

Cleaning up

To avoid ongoing charges from the resources you created while following this post, delete the delivery and then remove the destinations and any example S3 bucket or Data Firehose delivery stream you no longer need. Deleting the delivery returns the stage to standard auto-managed logging.

aws logs delete-delivery --id <delivery-id>

When the delivery is deleted, CloudWatch Logs disables log delivery on the API Gateway stage automatically. The delivery source and delivery destination remain as independent objects. Delete them with delete-delivery-source and delete-delivery-destination if you do not plan to reuse them.

Conclusion

CloudWatch Logs delivery for API Gateway REST API execution logs helps address the 1 KB event truncation and single managed destination constraints. You can now route full execution logs to CloudWatch Logs, Amazon S3, or Amazon Data Firehose, use multiple destinations from a single stage, and pay vended logs pricing.

The feature works alongside existing method settings. No changes to your current logging configuration are required beyond creating the delivery itself.

To get started, refer to Route execution logs with Amazon CloudWatch Logs delivery in the API Gateway documentation. For more about CloudWatch Logs delivery configuration, see Enable logging from AWS services. For pricing details, review the Amazon CloudWatch pricing page. Try it on a test stage and share your experience in the comments.

Post Syndicated from Biswanath Mukherjee original https://aws.amazon.com/blogs/compute/bring-your-own-client-certificate-for-backend-mtls-in-amazon-api-gateway/

Enterprises that use Amazon API Gateway in front of internal or partner backends often want to bring their own client certificate for backend mutual TLS (mTLS) authentication. During mTLS, the backend presents its own server certificate and also requests the caller to present a client certificate to validate it against a trusted certificate authority (CA). Until now, you could use only an API Gateway-generated, self-signed SSL certificate for the outbound connection, because there was no CA behind it for the backend to trust. Backends that enforce a specific corporate or partner CA reject that self-signed certificate, and the mutual TLS handshake fails. Bringing your own CA-signed certificate is necessary for scenarios such as migrating APIs off legacy gateways or meeting your internal PKI mandates that require certificates from an approved CA.

With API Gateway, you can now bring your own client certificate for backend mutual TLS (mTLS) authentication. You can either use a third-party certificate or a certificate issued by AWS Private Certificate Authority. If you’re using a third-party certificate, you must import the certificate in AWS Certificate Manager (ACM). Then you configure the ACM certificate ARN in your REST API stage. API Gateway presents that certificate during the backend mTLS handshake.

Solution overview

In this post, you build a REST API with an outbound mTLS connection using this newly launched API Gateway feature. This solution demonstrates an outbound mTLS connection between Amazon API Gateway and a backend application running on Amazon Elastic Container Service (Amazon ECS).

The following diagram shows the solution architecture.

The solution uses:

1. AWS Private Certificate Authority with a root-subordinate CA hierarchy to issue both the client and server certificates through AWS Certificate Manager (ACM).

2. Amazon API Gateway REST API stage configured with the ACM client certificate ARN (ClientCertificateId), so that the API Gateway presents the certificate during the outbound TLS handshake.

3. Amazon ECS on AWS Fargate running an NGINX sidecar that holds the server certificate and validates the incoming client certificate against a CA bundle (root and subordinate chain).

A request goes through the following steps:

1. Client application invokes the REST API exposed by API Gateway. The API Gateway stage is configured with an ACM client certificate ARN.

2. API Gateway opens an outbound connection to the Network Load Balancer (NLB) to begin the TLS handshake. The API Gateway presents an ACM client certificate configured at the stage level when the backend requests one.

3. The NLB listens for the incoming TCP request on port 443 and forwards the call to Amazon ECS Fargate. The NLB acts as a passthrough and does not terminate the TLS connection.

4. The NGINX sidecar container running on Amazon ECS performs the inbound mTLS handshake:

• NGINX presents the backend server certificate and verifies the client certificate against a mounted CA bundle (root and subordinate chain).

• After verification, NGINX forwards the request and parsed certificate details to the validator app container over local HTTP.

• The validator app re-checks the certificate validity window, matches the common name against an allowlist, and returns a structured JSON response.

Note: The NGINX sidecar is not mandatory for this flow. It demonstrates separation of concerns: NGINX handles the mTLS handshake, and the validator app contains the business logic.

Prerequisites for demo

To follow along, you need the following:

• Create an AWS account if you do not already have one.

• Access to an AWS account through the AWS Management Console and the AWS Command Line Interface (AWS CLI). The AWS Identity and Access Management (IAM) principal you use must have permissions to make the necessary AWS service calls and manage the resources in this post. Follow the principle of least privilege.

• Git installed.

• AWS Serverless Application Model (AWS SAM) CLI installed.

• Docker installed and running, to build and push the two container images.

• Python 3.14 installed.

• jq command line tools installed.

Environment setup

Run the following commands to set up the demo environment:

1. Create a new folder and clone the GitHub repository:

   git clone https://github.com/aws-samples/sample-api-backend-mtls
   cd sample-api-backend-mtls

2. Set the environment variables after replacing the placeholders:

   ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   REGION=<Your AWS Region, for example, us-east-1>
   STACK_NAME=<Your stack name e.g. outbound-mtls-backend>

Build the container images

Run the following commands to create container images of NGINX sidecar container and the validator app containers:

1. Create two Amazon Elastic Container Registry (Amazon ECR) repositories, one for NGINX and another for validator app containers respectively:

   NGINX_REPO_URI=$(aws ecr create-repository \
     --repository-name $STACK_NAME-nginx-sidecar \
     --image-tag-mutability IMMUTABLE \
     --image-scanning-configuration scanOnPush=true \
     --region "$REGION" \
     --query "repository.repositoryUri" --output text)
   
   VALIDATOR_REPO_URI=$(aws ecr create-repository \
     --repository-name $STACK_NAME-validator-app \
     --image-tag-mutability IMMUTABLE \
     --image-scanning-configuration scanOnPush=true \
     --region "$REGION" \
     --query "repository.repositoryUri" --output text)
   
   aws ecr get-login-password --region "$REGION" \
     | docker login --username AWS --password-stdin "${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com"

2. Build and push the NGINX and validator app containers:

   docker build --platform linux/amd64 -t $STACK_NAME-nginx-sidecar nginx/
   docker tag $STACK_NAME-nginx-sidecar:latest "${NGINX_REPO_URI}:latest"
   docker push "${NGINX_REPO_URI}:latest"
   docker build --platform linux/amd64 -t $STACK_NAME-validator-app validator_app/
   docker tag $STACK_NAME-validator-app:latest "${VALIDATOR_REPO_URI}:latest"
   docker push "${VALIDATOR_REPO_URI}:latest"

Deploy and test the solution

You first deploy the stack without the client certificate configured in the API Gateway and perform negative testing. The mTLS handshake will fail because of a missing client certificate in the request. Then you update the stack to configure client certificate in API Gateway stage and retest mTLS.

1. Run the following command to build and deploy the overall stack without client certificate configured at API Gateway stage:

   sam build
   sam deploy \
     --stack-name $STACK_NAME \
     --resolve-s3 \
     --capabilities CAPABILITY_IAM \
     --region "$REGION" \
     --parameter-overrides \
     NginxRepositoryUri="$NGINX_REPO_URI" \
     ValidatorRepositoryUri="$VALIDATOR_REPO_URI" \
     EnableOutboundMtls=false

2. Wait for the task to reach RUNNING and pass its target group health check:

   EcsClusterName=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region "$REGION" \
     --query "Stacks[0].Outputs[?OutputKey=='EcsClusterName'].OutputValue" \
     --output text)
   
   TargetGroupArn=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region "$REGION" \
     --query "Stacks[0].Outputs[?OutputKey=='TargetGroupArn'].OutputValue" \
     --output text)
   
   aws ecs list-tasks --cluster "$EcsClusterName" --region "$REGION"
   
   aws elbv2 describe-target-health \
     --target-group-arn "$TargetGroupArn" --region "$REGION"

3. Capture the front API invoke URL from the stack outputs:

   FRONT_API_URL=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region "$REGION" \
     --query "Stacks[0].Outputs[?OutputKey=='FrontApiUrl'].OutputValue" \
     --output text)
   
   NLB_DNS_NAME=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region "$REGION" \
     --query "Stacks[0].Outputs[?OutputKey=='NlbDnsName'].OutputValue" \
     --output text)
   
   FRONT_CLIENT_CERT_ARN=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region "$REGION" \
     --query "Stacks[0].Outputs[?OutputKey=='FrontClientCertArn'].OutputValue" \
     --output text)
   
   FRONT_API_ID=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region "$REGION" \
     --query "Stacks[0].Outputs[?OutputKey=='FrontApiId'].OutputValue" \
     --output text)

4. Wait a minute or two after the stack finishes, then invoke the front API:

   curl -v "$FRONT_API_URL"

   The following is the NGINX configuration for mTLS:

   ...
   server {
       listen 443 ssl;
       # Server identity (issued by the private CA).
       ssl_certificate /etc/nginx/certs/server.crt;
       ssl_certificate_key /etc/nginx/certs/server.key;
       # Inbound mutual TLS: require and validate the client certificate
       # against the CA bundle (root + subordinate CA chain).
       ssl_client_certificate /etc/nginx/certs/ca_bundle.pem;
       ssl_verify_client on;
       ssl_verify_depth 2;
       ssl_protocols TLSv1.2;
   ...}

   The curl command returns HTTP/2 400, with a response body containing 400 No required SSL certificate was sent. Because the API Gateway is not presenting a client certificate on the outbound handshake, the NGINX sidecar container in Amazon ECS rejects the mTLS connection. The following screenshot shows the response:

   

5. Now redeploy the solution with outbound mTLS enabled:

   sam deploy \
     --stack-name $STACK_NAME \
     --resolve-s3 \
     --capabilities CAPABILITY_IAM \
     --region "$REGION" \
     --parameter-overrides \
     NginxRepositoryUri="$NGINX_REPO_URI" \
     ValidatorRepositoryUri="$VALIDATOR_REPO_URI" \
     EnableOutboundMtls=true

6. Wait a minute or two after the stack finishes, then invoke the front API again:

   curl -v "$FRONT_API_URL"

   Because the client certificate is now presented during the mTLS handshake, the handshake completes successfully, as shown in the following response snippet:

   

Automatic certificate renewal

When a certificate changes in ACM, API Gateway detects the update and propagates the new certificate automatically. You do not redeploy the stage, and the API experiences no downtime during rotation. Certificate propagation is eventually consistent. During an update, the backend might briefly receive either the old or the new certificate. ACM also emits certificate expiration notifications through Amazon EventBridge, which you can use to set alarms before a certificate expires.

Clean up

If you followed along only for demonstration purposes, to avoid incurring future charges, run the following commands to delete the resources created in this demo:

1. Clean up the S3 buckets:

   NLB_LOGS_BUCKET=$(aws cloudformation describe-stacks \
     --stack-name $STACK_NAME --region $REGION \
     --query "Stacks[0].Outputs[?OutputKey=='NlbAccessLogsBucketName'].OutputValue" \
     --output text)
   
   aws s3api list-object-versions --bucket "$NLB_LOGS_BUCKET" \
     --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' \
     --output json | \
     jq -c '.Objects[]? // empty' | \
     while read -r obj; do
       aws s3api delete-object --bucket "$NLB_LOGS_BUCKET" \
         --key "$(echo "$obj" | jq -r .Key)" \
         --version-id "$(echo "$obj" | jq -r .VersionId)" \
         --region $REGION
     done
   
   aws s3api list-object-versions --bucket "$NLB_LOGS_BUCKET" \
     --query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' \
     --output json | \
     jq -c '.Objects[]? // empty' | \
     while read -r obj; do
       aws s3api delete-object --bucket "$NLB_LOGS_BUCKET" \
         --key "$(echo "$obj" | jq -r .Key)" \
         --version-id "$(echo "$obj" | jq -r .VersionId)" \
         --region $REGION
     done

2. Delete the stack:

   sam delete --stack-name $STACK_NAME --region $REGION --no-prompts

3. Delete the ECR repository:

   aws ecr delete-repository --repository-name $STACK_NAME-nginx-sidecar --force --region $REGION
   aws ecr delete-repository --repository-name $STACK_NAME-validator-app --force --region $REGION

Conclusion

In this post, you configured a REST API with an outbound mTLS connection using Amazon API Gateway and an ECS Fargate backend. With this new feature launch in API Gateway, you can now bring your own client certificate for outbound mTLS handshake for your REST APIs. You can now meet your internal PKI mandates to authenticate backends that pin a specific certificate issuer.

To get started, import a certificate from your own PKI into ACM and configure your API Gateway REST API stage for outbound mTLS authentication. For more information, see Present client certificates to backend services with mutual TLS in API Gateway. If you have feedback about this post, leave it in the comments section. For technical questions, you can start a thread on AWS re:Post.

Further reading

• Present client certificates to backend services with mutual TLS in API Gateway

• Amazon API Gateway REST API client certificates

• Integration tlsConfig reference

• Importing certificates into AWS Certificate Manager

• Requesting a private certificate with AWS Private CA

• Automating mutual TLS setup for Amazon API Gateway