Tag Archives: CloudWatch Events

Streamline your GitHub journey with AWS CodePipeline and AWS DevOps Agent

Post Syndicated from Anjani Reddy original https://aws.amazon.com/blogs/devops/streamline-your-github-journey-with-aws-codepipeline-and-aws-devops-agent/

Introduction

When CI/CD deployment failures occur for GitHub hosted applications,  AWS DevOps Agent reduces the hours that Development and Site Reliability Engineering (SRE) teams typically spend manually investigating across multiple AWS services, logs, and pipeline stages. This process delays critical deployments and impacts software delivery velocity. This is especially true when teams need to correlate data between GitHub commit histories, AWS CodePipeline execution logs, and Amazon CloudWatch metrics. When continuous integration and continuous delivery (CI/CD) pipelines fail, engineers often find themselves context-switching between GitHub pull requests, code build logs, deployment artifacts, and downstream service health metrics. This process of identifying root causes can extend resolution time from minutes to hours, especially in multi-service architectures.

AWS DevOps Agent reduces this manual investigation by automatically correlating pipeline failures with specific code changes. Rather than spending hours manually tracing deployment failures through multiple systems, engineers can use AWS DevOps Agent to perform this correlation. It identifies which specific code changes caused pipeline failures and provides remediation guidance. The agent analyzes pipeline failures, correlates them with specific commits and pull requests, and identifies root causes across the deployment chain.

AWS CodePipeline combined with AWS DevOps Agent helps address this challenge by creating a streamlined path from GitHub repositories to AWS deployments. This solution reduces manual handoffs, reduces configuration complexity, and provides end-to-end visibility across the entire development lifecycle.

In this post, you learn how to integrate AWS DevOps Agent with your GitHub repositories to automatically correlate deployment failures with specific commits, providing root cause analysis and remediation steps across your entire CI/CD pipeline.

Solution overview 

Modern software delivery teams face a persistent challenge: when deployments fail, engineers spend valuable time manually correlating logs, tracing pipeline errors, and diagnosing root causes across disconnected tools. This reactive cycle slows recovery and increases mean time to resolution (MTTR). By integrating the AWS DevOps Agent with GitHub, AWS CodePipeline, Amazon CloudWatch, and AWS Lambda, teams can shift from manual triage to automated incident investigation, directly within their existing GitHub-based workflows.

This solution integrates AWS DevOps Agent with GitHub to automate deployment failure investigation. The following sections explain the architecture and operational benefits.

How it works​ 

The architecture creates an automated monitoring and remediation flow that monitors your deployment pipeline and responds to issues. Your source code resides in a GitHub repository, and AWS CodePipeline orchestrates the build, test, and deployment stages. Amazon CloudWatch continuously monitors pipeline execution metrics and logs and generates alarms when it detects anomalies or failures, such as failed build stages, deployment rollbacks, or threshold breaches in downstream application of health metrics. When a failure occurs, it generates an error metric in CloudWatch. The CloudWatch Alarm detects this error and transitions to an ALARM state, which directly invokes the WebHook Executor Lambda. The WebHook Executor then sends an authenticated HTTP POST request to DevOps Agent, which receives the incident and begins an investigation.

Webhook integration acts as the bridge between the Amazon CloudWatch, the monitoring layer. Lambda parses the alarm payload and extracts contextual metadata and then invokes the DevOps Agent with a structured investigation request.

Integration with Operational Excellence

This solution directly supports the AWS Well-Architected Framework’s Operational Excellence pillar by automating the investigation process and reducing the MTTR. The investigation capability of AWS DevOps Agent aligns with AWS Incident Detection and Response (IDR) best practices, helping teams to detect, diagnose, and develop mitigation plans for pipeline failures faster while maintaining a full audit trail of agent actions and findings. This creates a delivery pipeline that accelerates resolution workflows through automated diagnostics and actionable remediation recommendations, keeping deployments moving and engineering teams focused on building rather than firefighting.

Architecture diagram showing GitHub repository connected to AWS CodePipeline, CloudWatch, Lambda, and DevOps Agent in an automated investigation flow 

Figure 1: GitHub and DevOps Agent integration

Prerequisites 

For this walkthrough, you should have access to and understanding of the following:

  •  An AWS account with permissions to create AWS Identity and Access Management (IAM) roles:
    1. Agent Space role – for basic service operations.
    2. Agent Space web app role – for using the Agent Space web app functionality.
    3. (Optional) Secondary source account roles if monitoring multiple AWS accounts. Refer to the DevOps Agent user guide for the details on setting up these roles.
  • A GitHub account:
    1. You have a GitHub account with administrative permissions for your repositories, or an organization you belong to.
    2. Your repositories contain code that deploys to AWS resources you want to monitor.
    3. You have identified the GitHub repositories you want AWS DevOps agent to access.
  • Access to register DevOps Agent with your GitHub Account or Organization.
  • CloudWatch monitoring enabled for your application.

​​Implementation steps​ 

Note: For this blog we used a sample application  from the AWS-samples.

  1. ​​Create an AWS DevOps Agent Space and configure the webhook​
    The first step is to create a dedicated Agent Space that serves as the central hub for your automated investigation workflow. The Agent Space connects your monitoring infrastructure to the DevOps Agent’s analysis capabilities.
    Create the DevOps Agent space by following the steps outlined in the Getting Started with AWS DevOps Agent guide Navigate to the DevOps Agent console.
    Create an Agent Space named after your application (for example, `myhotelapp`)
    1) “Auto-create both IAM roles”.
    2) “Edit the role names to be descriptive (for example, DevOpsAgentRole-AgentSpace-hotel-app and DevOpsAgentRole-WebappAdmin-hotel-app)”
Screenshot of AWS DevOps Agent console showing the Agent Space creation interface with IAM role configuration options

Figure 2: Agent Spaces Screen

On the Capabilities tab, generate a webhook and save the credentials

Store the webhook credentials in AWS Secrets Manager:

```bash

aws secretsmanager create-secret \

--name devops-agent-webhook-credentials \

--secret-string '{"webhookUrl":"YOUR-WEBHOOK-URL","webhookSecret":"YOUR-WEBHOOK-SECRET"}' \

--region us-east-1

```

2. Configure GitHub integration with your AgentSpace

With your Agent Space created and webhook configured, the next step is to connect your GitHub repositories. This integration allows the DevOps Agent to access commit histories, pull request data, and code changes when investigating pipeline failures.

To configure GitHub integration with your AgentSpace:
1. From the Capabilities tab within your configured AgentSpace, navigate to the GitHub Configuration section and choose “Register”

Screenshot of the GitHub Configuration section in the AgentSpace Capabilities tab showing the Register button

Figure 3: Capability Providers

2.     Your GitHub repositories will be listed with their connection status.

3.     To connect to a repository, verify that the Status shows “Ready to connect” and choose the + button in the Actions column.

4.     Upon successful connection, the Status updates to ‘Connected’.

To automatically trigger AWS DevOps Agent investigations via Webhook when a CloudWatch enters the ALARM state, you can refer to sample-aws-devops-agent-cloudwatch and build based on your use case.

3. Troubleshooting application deployment 5XX errors with CloudWatch and AWS DevOps Agent

When your application encounters 5XX errors during deployment, CloudWatch alarms detect the anomaly and trigger the DevOps Agent investigation workflow. The following dashboard shows the alarm state that initiates the automated investigation process.

Screenshot of CloudWatch dashboard displaying alarm metrics triggered by application 5XX errors

Figure 4: CloudWatch Dashboard

4. Resolving deployment/build errors during CI/CD deployment

The following use cases demonstrate how AWS DevOps Agent investigates and resolves common CI/CD pipeline failures. Each scenario walks through the failure trigger, the automated investigation, and the remediation guidance that the agent provides

Use case 1: Push a code change that introduces an invalid DynamoDB table name

Simulate: Push a code change that breaks the DynamoDB table name — e.g., change DYNAMODB_TABLE_NAME env var but don’t update CloudFormation to make the CodePipeline unit testing fail

A – dynamodb_table: process.env.DYNAMODB_TABLE_NAME || “Rooms”,

B + dynamodb_table: “HotelRooms”

The CodePipeline triggers 5xx alarms and the webhook triggers a DevOps Agent investigation.

DevOps Agent analyzes the 500 errors in relation to the configuration change, identifies the invalid DynamoDB endpoint, and shows the timeline: configuration update → service redeployment → requests fail with connection errors.

Screenshot of CodePipeline execution view showing a failed unit test stage highlighted in red

Figure 5: Unit test failed for the CodePipeline

Use case 2: Identifying dependency resolution failures from bad commits

1. Navigate to `package.json`

2. Change any dependency name to something invalid — for example, change `”express”` to `”expresss”` (extra ‘s’)

3. Commit the change directly to `main`

CodePipeline detects the push and starts a new execution. The CI stage runs `npm install`, which fails because the misspelled package doesn’t exist. The Amazon EventBridge rule catches the stage failure and invokes the webhook executor Lambda, which triggers a DevOps Agent investigation.

In the DevOps Agent console, select your Agent Space, then choose Operator access to open the web app.  Navigate to the Incident Response tab to view the new investigation.

Screenshot of DevOps Agent showing the first step of the mitigation plan identifying the root cause

Figure 6: Mitigation plan step1

Screenshot of DevOps Agent showing steps 2 through 4 of the mitigation plan with remediation commands

Figure 7: Mitigation plan steps 2-4

DevOps Agent investigates the pipeline failure, examines the CodeBuild logs showing the `npm install` error, and correlates it with the recent commit to the repository. It identifies the root cause as a dependency resolution failure introduced by the latest code change.

Clean up

This walkthrough creates AWS resources that incur charges, including AWS DevOps Agent (pay-per-use), Lambda functions, CodePipeline executions, CloudWatch alarms, and Secrets Manager secrets. Follow the cleanup steps when finished to avoid ongoing charges.

1. Delete the Secrets Manager secret devops-agent-webhook-credentials using: aws secretsmanager delete-secret –secret-id devops-agent-webhook-credentials –region us-east-1

2. Delete your Agent Space from the AWS DevOps Agent console

3. Remove the GitHub pipeline connection from your settings.

4. Delete the IAM roles created for the Agent Space.

5. Delete the Lambda function, EventBridge rule, and CloudWatch alarms created for webhook integration.

6. (Optional) If you created additional source account roles, remove those as well.

Conclusion

The AWS DevOps Agent integration with GitHub fundamentally transforms how engineering teams approach CI/CD reliability by shifting from reactive troubleshooting to proactive incident prevention. By autonomously correlating CodePipeline failures with specific GitHub commits, analyzing root causes across the deployment chain, and providing intelligent remediation recommendations, this solution reduces mean time to resolution from hours to minutes while maintaining the human oversight necessary for production environments.

Organizations implementing this integration gain a resilient software delivery pipeline that combines the collaborative strengths of GitHub source control with AWS’s intelligent automation capabilities. This helps teams maintain deployment velocity, strengthen operational excellence, and focus engineering effort on innovation rather than incident response.

AWS CodePipeline, Amazon CloudWatch, AWS Lambda, and the AWS DevOps Agent integrate natively to provide end-to-end visibility and autonomous investigation capabilities. Together, they accelerate recovery workflows, reduce operational friction, and build the foundation for continuous delivery at scale.

About authors

Anjani Reddy

Anjani is a Sr. Solutions Architect at AWS. She works with Enterprise customers to provide operational guidance to innovate and build a secure, scalable cloud on the AWS platform. Outside of work, she is an Indian classical & salsa dancer, loves to travel and Volunteers for American Red Cross & Hands on Atlanta.

Jared Thompson
Jared Thompson is a Senior Technical Account Manager at AWS, where he partners with strategic enterprise customers to optimize cloud operations and accelerate AI/ML workloads at scale. Jared specializes in GPU-accelerated computing, capacity planning, and cloud observability, with a passion for turning complex infrastructure challenges into automated, self-healing systems. He is a recipient of the AWS Golden Jacket award and when not at work, he can be found on a cruise ship.

Aneesh Varghese is a Senior Technical Account Manager at AWS with more than 19 years of Information Technology industry experience. Aneesh supports enterprise customers in cost optimization strategies, Cloud operations, MLOps, providing advocacy and strategic technical guidance to help plan and build solutions using AWS best practices. Outside of work, Aneesh likes to spend time with family, play Basketball and Badminton.

Create a Multi-Region Python Package Publishing Pipeline with AWS CDK and CodePipeline

Post Syndicated from Brian Smitches original https://aws.amazon.com/blogs/devops/create-a-multi-region-python-package-publishing-pipeline-with-aws-cdk-and-codepipeline/

Customers can author and store internal software packages in AWS by leveraging the AWS CodeSuite (AWS CodePipeline, AWS CodeBuild, AWS CodeCommit, and AWS CodeArtifact).  As of the publish date of this blog post, there is no native way to replicate your CodeArtifact Packages across regions. This blog addresses how a custom solution built with the AWS Cloud Development Kit and AWS CodePipeline can create a Multi-Region Python Package Publishing Pipeline.

Whether it’s for resiliency or performance improvement, many customers want to deploy their applications across multiple regions. When applications are dependent on custom software packages, the software packages should be replicated to multiple regions as well. This post will walk through how to deploy a custom package publishing pipeline in your own AWS Account. This pipeline connects a Python package source code repository to build and publish pip packages to CodeArtifact Repositories spanning three regions (the primary and two replica regions). While this sample CDK Application is built specifically for pip packages, the underlying architecture can be reused for different software package formats, such as npm, Maven, NuGet, etc.

Solution overview

The following figure demonstrates the solution workflow:

  1. A CodePipeline pipeline orchestrates the building and publishing of the software package
    1. This pipeline is triggered by commits on the main branch of the CodeCommit repository
    2. A CodeBuild job builds the pip packages using twine to be distributed
    3. The publish stage (third column) uses three parallel CodeBuild jobs to publish the distribution package to the two CodeArtifact repositories in separate regions
  1. The first CodeArtifact Repository stores the package contents in the primary region.
  2. The second and third CodeArtifact Repository act as replicas and store the package contents in other regions.
Figure 1. A figure showing the architecture diagram

Figure 1.  Architecture diagram

All of these resources are defined in a single AWS CDK Application. The resources are defined in CDK Stacks that are deployed as AWS CloudFormation Stacks. AWS CDK can deploy the different stacks across separate regions.

Prerequisites

Before getting started, you will need the following:

  1. An AWS account
  2. An instance of the AWS Cloud9 IDE or an alternative local compute environment, such as your personal computer
  3. The following installed on your compute environment:
    1. AWS CDK
    2. AWS Command Line Interface (AWS CLI)
    3. npm
  1. The AWS Accounts must be bootstrapped for CDK in the necessary regions. The default configuration uses us-east-1, us-east-2 and us-west-2  as these three regions support CodeArtifact.

A new AWS Cloud9 IDE is recommended for this tutorial to isolate these actions in this post from your normal compute environment. See the Cloud9 Documentation for Creating an Environment.

Deploy the Python Package Publishing Pipeline into your AWS Account with the CDK

The source code can be found in this GitHub Repository.

  1. Fork the GitHub Repo into your account. This way you can experiment with changes as necessary to fit your workload.
  2. In your local compute environment, clone the GitHub Repository and cd into the project directory:
git clone [email protected]:<YOUR_GITHUB_USERNAME>/multi-region-
python-package-publishing-pipeline.git && cd multi-region-
python-package-publishing-pipeline
  1. Install the necessary node packages:
npm i
  1. (Optional) Override the default configurations for the CodeArtifact domainName, repositoryName, primaryRegion, and replicaRegions.
    1. navigate to ./bin/multiregion_package_publishing.ts and update the relevant fields.
    2. From the project’s root directory (multi-region-python-package-publishing-pipeline), deploy the AWS CDK application. This step may take 5-10 minutes.
cdk deploy --all
  1. When prompted “Do you wish to deploy these changes (y/n)?”, Enter y.

Viewing the deployed CloudFormation stacks

After the deployment of the AWS CDK application completes, you can view the deployed AWS CDK Stacks via CloudFormation. From the AWS Console, search “CloudFormation’ in the search bar and navigate to the service dashboard. In the primary region (us-east-1(N. Virginia)) you should see two stacks: CodeArtifactPrimaryStack-<region> and PackagePublishingPipelineStack.

Screenshot showing the CloudFormation Stacks in the primary region

Figure 2. Screenshot showing the CloudFormation Stacks in the primary region

Switch regions to one of the secondary regions us-west-2 (Oregon) or us-east-2 (Ohio) to see the remaining stacks named CodeArtifactReplicaStack-<region>. These correspond to the three AWS CDK Stacks from the architecture diagram.

Screenshot showing the CloudFormation stacks in a separate region

Figure 3. Screenshot showing the CloudFormation stacks in a separate region

Viewing the CodePipeline Package Publishing Pipeline

From the Console, select the primary region (us-east-1) and navigate to CodePipeline by utilizing the search bar. Select the Pipeline titled packagePipeline and inspect the state of the pipeline. This pipeline triggers after every commit from the CodeCommit repository named PackageSourceCode. If the pipeline is still in process, then wait a few minutes, as this pipeline can take approximately 7–8 minutes to complete all three stages (Source, Build, and Publish). Once it’s complete, the pipeline should reflect the following screenshot:

A screenshot showing the CodePipeline flow

Figure 4. A screenshot showing the CodePipeline flow

Viewing the Published Package in the CodeArtifact Repository

To view the published artifacts, go to the primary or secondary region and navigate to the CodeArtifact dashboard by utilizing the search bar in the Console. You’ll see a repository named package-artifact-repo. Select the repository and you’ll see the sample pip package named mypippackage inside the repository. This package is defined by the source code in the CodeCommit repository named PackageSourceCode in the primary region (us-east-1).

Screenshot of the package repository

Figure 5. Screenshot of the package repository

Create a new package version in CodeCommit and monitor the pipeline release

Navigate to your CodeCommit’s PackageSourceCode (us-east-1 CodeCommit > Repositories > PackageSourceCode. Open the setup.py file and select the Edit button. Make a simple modification, change the version = '1.0.0' to version = '1.1.0' and commit the changes to the Main branch.

A screenshot of the source package's code repository in CodeCommit

Figure 6. A screenshot of the source package’s code repository in CodeCommit

Now navigate back to CodePipeline and watch as the pipeline performs the release automatically. When the pipeline finishes, this new package version will live in each of the three CodeArtifact Repositories.

Install the custom pip package to your local Python Environment

For your development team to connect to this CodeArtifact Repository to download repositories, you must configure the pip tool to look in this repository. From your Cloud9 IDE (or local development environment), let’s test the installation of this package for Python3:

  1. Copy the connection instructions for the pip tool. Navigate to the CodeArtifact repository of your choice and select View connection instructions
    1. Select Copy to copy the snippet to your clipboard
Screenshot showing directions to connect to a code artifact repository

Figure 7. Screenshot showing directions to connect to a code artifact repository

  1. Paste the command from your clipboard
  2. Run pip install mypippackage==1.0.0
Screenshot showing CodeArtifact login

Figure 8. Screenshot showing CodeArtifact login

  1. Test the package works as expected by importing the modules
  2. Start the Python REPL by running python3 in the terminal
Screenshot of the package being imported

Figure 9. Screenshot of the package being imported

Clean up

Destroy all of the AWS CDK Stacks by running cdk destroy --all from the root AWS CDK application directory.

Conclusion

In this post, we walked through how to deploy a CodePipeline pipeline to automate the publishing of Python packages to multiple CodeArtifact repositories in separate regions. Leveraging the AWS CDK simplifies the maintenance and configuration of this multi-region solution by using Infrastructure as Code and predefined Constructs. If you would like to customize this solution to better fit your needs, please read more about the AWS CDK and AWS Developer Tools. Some links we suggest include the CodeArtifact User Guide (with sections covering npm, Python, Maven, and NuGet), the CDK API Reference, CDK Pipelines, and the CodePipeline User Guide.

About the authors:

Andrew Chen

Andrew Chen is a Solutions Architect with an interest in Data Analytics, Machine Learning, and DevOps. Andrew has previous experience in management consulting in which he worked as a technical architect for various cloud migration projects. In his free time, Andrew enjoys fishing, hiking, kayaking, and keeping up with financial markets.

Brian Smitches

Brian Smitches is a Solutions Architect with an interest in Infrastructure as Code and the AWS Cloud Development Kit. Brian currently supports Federal SMB Partners and has previous experience with Full Stack Application Development. In his personal time, Brian enjoys skiing, water sports, and traveling with friends and family.