All posts by Michael Ohde

AWS CodePipeline adds support for Branch-based development and Monorepos

Post Syndicated from Michael Ohde original https://aws.amazon.com/blogs/devops/aws-codepipeline-adds-support-for-branch-based-development-and-monorepos/

AWS CodePipeline is a managed continuous delivery service that automates your release pipelines for application and infrastructure updates. Today, CodePipeline adds triggers and new execution modes to support teams with various delivery strategies. These features give customers more choice in the pipelines they build.

In this post, I am going to show you how to use triggers and pipeline execution modes together to create three pipeline designs. These examples are requested by customers that practice branch-based development or manage multiple projects within a monorepo.

  • Pipeline #1: Create a GitFlow (multi-branch) release pipeline.
  • Pipeline #2: Run a pipeline on all pull requests (PRs).
  • Pipeline #3: Run a pipeline on a single folder within a monorepo.

As I walkthrough each of the pipelines you will learn more about these features and how to use them. After completing the blog, you can use triggers and execution modes to adapt these examples to your pipeline needs.

Pipeline #1 – Create a GitFlow (multi-branch) release pipeline

GitFlow is a development model that manages large projects with parallel development and releases using long-running branches. GitFlow uses two permanent branches, main and develop, along with supporting feature, release, and hotfix branches. Since I will cover triggering a pipeline from multiple branches, these concepts can be applied to simplify other multi-branch pipeline strategies such as GitHub flow.

I can create pipelines using the AWS Management Console, AWS CLI, AWS CloudFormation, or by writing code that calls the CodePipeline CreatePipeline API. In this blog, I will keep things simple by creating two pipelines, a release pipeline and a feature development pipeline. I start by navigating to the CodePipeline console and choosing Create pipeline.

In the first step, Pipeline settings, as per Figure 1 below, you will now see options for the newly added Execution modes – Queued and Parallel.

Example GitFlow release pipeline settings for Queued execution mode.
Figure 1. Example GitFlow release pipeline settings for Queued execution mode.

The execution mode of the pipeline determines the handling of multiple executions:

  • Superseded – an execution that started more recently can overtake one that began earlier. Before today, CodePipeline only supported Superseded execution mode.
  • Queued – the executions wait and do not overtake executions that have already started. Executions are processed one by one in the order that they are queued.
  • Parallel – the executions are independent of one another and do not wait for other executions to complete before starting.

The first pipeline, release pipeline, will trigger for main, develop, hotfix, and release branches. I select Queued, since I want to run every push to these branches in the order triggered by the pipeline. I make sure the Pipeline type chosen is V2 and I click Next.

Example source connection and repository.
Figure 2. Example source connection and repository.

In step two, Add source stage, I select my Source provider, Connection, Repository name, and Default branch. I need to use a source provider that uses a connection to my external code repository, in this example I’m using GitHub so I select Connect to GitHub. Connections authorize and establish configurations that associate a third-party provider such as GitHub with CodePipeline.

Now that I have my Source setup, I’m going to configure a Trigger. Triggers define the event type that starts the pipeline, such as a code push or pull request. I select the Specify filter from the Trigger types, since I want to add a filtered trigger.

For this pipeline, I select Push for the Event type. A push trigger starts a pipeline when a change is pushed to the source repository. The execution uses the files in the branch that is pushed to, the destination branch. Next, I select the Filter type of Branch. The branch filter type specifies the branches in GitHub connected repository that the trigger monitors in order to know when to start an execution.

There are two types of branch filters:

  • Include – the trigger will start a pipeline if the branch name matches the pattern.
  • Exclude – the trigger will NOT start a pipeline if the branch name matches the pattern.

Note: If Include and Exclude both have the same pattern, then the default is to exclude the pattern.

Branching patterns are entered in the glob format, detailed in Working with glob patterns in syntax, to specify the branch I want to trigger, I enter main,develop,hotfix/**,release/** in Include and I leave Exclude empty.

Example GitFlow release pipeline for push event type and branch filters.
Figure 3. Example GitFlow release pipeline for push event type and branch filters.

I am done configuring the filters and I click Next.

To keep the focus of the blog on the pipeline and not the application, I will skip ahead to Create pipeline. If you are curious about by my application and build step, I followed the example in AWS CodeBuild adds support for AWS Lambda compute mode.

Next, I create the feature development pipeline. The feature development pipeline will trigger for feature branches. This time, I select the Parallel Execution mode, as developers should not be blocked by their peers working in other feature branches. I make sure the Pipeline type chosen is V2 and I click Next.

Example GitFlow feature pipeline settings for Queued execution mode.
Figure 4. Example GitFlow feature pipeline settings for Queued execution mode.

In Step 2, the source provider and connection is setup the same as per the previous release pipeline, see Figure 2 above. Once the Source step is complete, I configure my Trigger with an Event type of Push, but this time I only enter feature/** for Include.

Example GitFlow feature pipeline for push event type and branch filters.
Figure 5. Example GitFlow feature pipeline for push event type and branch filters.

I am done configuring the filters and I skip forward to Create pipeline.

After the pipeline is finished creating, I can now see both of the pipelines I created – the release pipeline and the feature development pipeline.

Example GitFlow pipelines.
Figure 6. Example GitFlow pipelines.

To verify my pipeline setup, I create and merge multiple code changes to feature branches and to the release branches – develop, release, and main. The Pipeline view now displays the executions that have been triggered by the matching branches. Note how these executions have been successfully added to the queue by the pipeline.

Example GitFlow release executions queued.
Figure 7. Example GitFlow release executions queued.

I have now implemented a GitFlow release pipeline. By using Branch filter types and Push event triggers, you can now extend this example to meet your branch-based development needs.

Pipeline #2 – Run a pipeline on all pull requests (PRs)

Before proceeding, I recommend you review the concepts covered in Pipeline #1, as you will build on that knowledge.

Triggering a pipeline on a pull request (PR) is a common continuous integration pattern to catch build and test failures before the PR is merged into the branch. A PR pipeline is often faster and lighter than the full release by limiting tests like security scans, validation tests, or performance tests to the changes in the PR rather than running them on every commit. Having a single pipeline triggered for all PRs allows reviewing and validating any proposed changes to the repository before merging.

To start I create a new pipeline, by clicking Create Pipeline. I change the Execution mode to Parallel. I choose Parallel because the development team will be working on multiple features at the same time and it is wasteful to wait for other executions to finish. I make sure the Pipeline type chosen is V2 and I click Next.

Example PR pipeline settings for Parallel execution mode.
Figure 8. Example PR pipeline settings for Parallel execution mode.

As per the previous pipeline, the Source provider and connection is setup as show in Figure 2 above. Once the Source step is setup, I configure my Pull Request Trigger.

For this pipeline, I select Pull Request for the Event type. A pull request trigger starts a pipeline when a pull request is opened, updated, or closed in the source repository. The execution will use the files in the branch that the change is being pulled from, the source branch. Next, I select Pull request is created and New revision is made to pull request for Events for pull request. To match pull requests for all branches, I enter ** under Include for Branches and leave Exclude empty.

Example PR pipeline for pull request event type.
Figure 9. Example PR pipeline for pull request event type.

I will fast-forward to the Create pipeline, skipping the details of the build and deploy steps, similar to what I did in Pipeline #1.

Once the pipeline has finished creating, I open a few PRs in my GitHub repository as a test. Back in CodePipeline when I click on my pipeline, I notice the pipeline takes me straight to the Execution history view. The reason I’m redirected to the execution history is the pipeline execution mode is Parallel and all executions are independent. From this view, I see the Trigger column displaying details about each pull request that has triggered the pipeline.

Example PR pipeline with executions in parallel.
Figure 10. Example PR pipeline with executions in parallel.

Note: To view an individual execution Pipeline, click the Execution ID.

I have now implemented a PR validation pipeline for all PRs across branches. By using Pull request event triggers and Branch filter types, you can now extend this example to meet your PR pipeline needs.

Pipeline #3 – Run a pipeline on a single folder within a monorepo

Before proceeding, I recommend you review the concepts covered in Pipeline #1, as you will build on that knowledge.

A monorepo is a software-development strategy where a single repository is used to contain the code for multiple projects. Running pipelines for each project contained in the monorepo on every commit can be inefficient, especially when each project requires different pipelines. For this pipeline example, I want to limit pipeline executions to only changes inside the infrastructure folder in the main branch. This can reduce cost, speed up deployments, and optimize resource usage.

To start, I create a new pipeline by clicking Create Pipeline. For this example, I keep the default Execution mode as Suspended, since I do not have any specific execution mode requirements. I make sure the Pipeline type chosen is V2 and I click Next.

As per the previous pipeline, the Source provider and connection is setup as per Figure 2 above. Once the Source step is complete, I configure my Trigger to focus on the infrastructure folder in the main branch.

For this pipeline, I select Push for the Event type. Next, I select the Filter type of Branch.  To match pushes to only main, I enter main under Include for Branches and leave Exclude empty. Under File paths, for Include, I enter infrastructure/** and I leave Exclude empty. The file paths filter type specifies file path names in the source repository that the trigger monitors in order to know when to start an execution. Similar to branch filters, I can specify file path name patterns in glob format under Include and Exclude.

Example monorepo pipeline for push event type and file path filters.
Figure 11. Example monorepo pipeline for push event type and file path filters.

I click Next, since I am done configuring the filters.

I will jump ahead to the Create pipeline, omitting the details of the build and deploy steps, like I did in Pipeline #1.

Once the pipeline has been created, I can test the pipeline Trigger in GitHub by making changes on the main branch inside and outside the infrastructure folder. To verify it is only invoking the pipelines inside the infrastructure folder, I open the History for the pipeline in CodePipeline. I confirm that only the changes I’m expecting are running.

Example monorepo pipeline with only infrastructure executions.
Figure 12. Example monorepo pipeline with only infrastructure executions.

I have now selectively invoked a pipeline based on repository changes in a monorepo. By using File paths filter types, you can now extend this example to meet your monorepo release pipelines.

Conclusion

AWS CodePipeline’s new triggers and execution modes unlock new patterns for building pipelines on AWS. In this post, I discussed the new features and three popular pipeline patterns you can build. If you are creating GitFlow or your own multi-branch strategy, CodePipeline simplifies managing release pipelines for multi-branch models. Whether you are using File path filter types for monorepos or leveraging Parallel execution mode to unblock developers, CodePipeline accelerates the delivery of your software. Check out the AWS CodePipeline User Guide and hands-on tutorials to automate your delivery workflows today.

Michael Ohde

Michael Ohde is a Senior Solutions Architect from Long Beach, CA. As a Product Acceleration Solution Architect at AWS, he currently assists Independent Software Vendor (ISVs) in the GovTech and EdTech sectors, by building modern applications using practices like serverless, DevOps, and AI/ML.

Implementing GitFlow with Amazon CodeCatalyst

Post Syndicated from Michael Ohde original https://aws.amazon.com/blogs/devops/implementing-gitflow-with-amazon-codecatalyst/

Amazon CodeCatalyst is a unified software development service for building and delivering applications on AWS. With CodeCatalyst, you can implement your team’s preferred branching strategy. Whether you follow popular models like GitFlow or have your own approach, CodeCatalyst Workflows allow you to design your development process and deploy to multiple environments.

Introduction

In a previous post in this series, Using Workflows to Build, Test, and Deploy with Amazon CodeCatalyst, we discussed creating a continuous integration and continuous delivery (CI/CD) pipeline in CodeCatalyst and how you can continually deliver high-quality updates through the use of one workflow. I will build on these concepts by focusing on how you collaborate across your codebase by using multiple CodeCatalyst Workflows to model your team’s branching strategy.

Having a standardized process for managing changes to the codebase allows developers to collaborate predictably and focus on delivering software. Some popular branching models include GitFlow, GitHub flow, and trunk-based development.

  • GitFlow is designed to manage large projects with parallel development and releases, featuring multiple long-running branches.
  • GitHub flow is a lightweight, branch-based workflow that involves creating feature branches and merging changes into the main branch.
  • Trunk-based development is focused on keeping the main branch always stable and deployable. All changes are made directly to the main branch, and issues are identified and fixed using automated testing and deployment tools.

In this post, I am going to demonstrate how to implement GitFlow with CodeCatalyst. GitFlow uses two permanent branches, main and develop, with supporting branches. The prefix names of the supporting branches give the function they serve — feature, release, and hotfix. I will apply this strategy by separating these branches into production and integration, both with their own workflow and environment. The main branch will deploy to production and the develop branch plus the supporting branches will deploy to integration.

Implementing GitFlow with CodeCatalyst
Figure 1. Implementing GitFlow with CodeCatalyst.

Upon completing the walkthrough, you will have the ability to utilize these techniques to implement any of the popular models or even your own.

Prerequisites

If you would like to follow along with the walkthrough, you will need to:

Walkthrough

For this walkthrough, I am going use the Static Website blueprint with the default configuration. A CodeCatalyst blueprint creates new project with everything you need to get started. To try this out yourself, launch the blueprint by following the steps outlined in the Creating a project in Amazon CodeCatalyst.

Once the new project is launched, I navigate to CI/CD > Environments. I see one environment called production. This environment was setup when the project was created by the blueprint. I will now add my integration environment. To do this, I click the Create environment above the list of environments.

Initial environment list with only production.
Figure 2. Initial environment list with only production.

A CodeCatalyst environment is where code is deployed and are configured to be associated with your AWS account using AWS account connections. Multiple CodeCatalyst environments can be associated with a single AWS account, allowing you to have environments in CodeCatalyst for development, test, and staging associated with one AWS account.

In the next screen, I enter the environment name as integration, select Non-production for the environment type, provide a brief description of the environment, and select the connection of the AWS account I want to deploy to. To learn more about connecting AWS accounts review Working with AWS accounts in Amazon CodeCatalyst. I will make note of my connection Name and Role, as I will need it later in my workflow. After I have entered all the details for the integration environment, I click Create environment on the bottom of the form. When I navigate back to CI/CD > Environments I now see both environments listed.

Environment list with integration and production.
Figure 3. Environment list with integration and production.

Now that I have my production and integration environment, I want to setup my workflows to deploy my branches into each separate environment. Next, I navigate to CI/CD > Workflows. Just like with the environments, there is already a workflow setup by the blueprint created called OnPushBuildTestAndDeploy. In order to review the workflow, I select Edit under the Actions menu.

OnPushBuildTestAndDeploy workflow Actions menu.
Figure 4. OnPushBuildTestAndDeploy workflow Actions menu.

By reviewing the workflow YAML, I see the OnPushBuildTestAndDeploy workflow is triggered by the main branch and deploys to production. Below I have highlighted the parts of the YAML that define each of these. The Triggers in the definition determine when a workflow run should start and Environment where code is deployed to.

Name: OnPushBuildTestAndDeploy
...
Triggers:
  - Type: PUSH
    Branches:
      - main
...
  Deploy:
...
    Environment:
      Name: production
      Connections:
        - Name: ****
          Role: ****

Since this confirms the production workflow is already done, I will copy this YAML and use it to create my integration workflow. After copying the entire OnPushBuildTestAndDeploy YAML to my clipboard (not just the highlights above), I navigate back to CI/CD > Workflows and click Create Workflow. Then in the Create workflow dialog window click Create.

Create workflow dialog window.
Figure 5. Create workflow dialog window.

Inside the workflow YAML editor, I replace all the existing content by pasting the OnPushBuildTestAndDeploy YAML from my clipboard. The first thing I edit in the YAML is the name of the workflow. I do this by finding the property called Name and replacing OnPushBuildTestAndDeploy to OnIntegrationPushBuildTestAndDeploy.

Next, I want to change the triggers to the develop branch and match the supporting branches by their prefixes. Triggers allow you to specify multiple branches and you can use regex to define your branch names to match multiple branches. To explore triggers further read Working with triggers.

Triggers:
  - Type: PUSH
    Branches:
      - develop
      - "feature/.*"
      - "release/.*"
      - "hotfix/.*"

After my triggers are updated, I need to update the Environment property with my integration environment. I replace both the Name and the Connections properties with the correct values for my integration environment. I use the Name and Role from the integration environment connection I made note of earlier. For additional details about environments in workflows review Working with environments.

  Deploy:
...
    Environment:
      Name: integration
      Connections:
        - Name: ****
          Role: ****

Before finishing the integration workflow, I have highlighted the use of ${WorkflowSource.BranchName} in the Deploy action. The workflow uses the BranchName variable to prevent different branch deployments from overwriting one another. This is important to verify as all integration branches use the same environment. The WorkflowSource action outputs both CommitId and BranchName to workflow variables automatically. To learn more about variables in workflows review Working with variables.

  Deploy:
...	
    Configuration:
      AmplifyBranchName: ${WorkflowSource.BranchName}

I have included the complete sample OnIntegrationPushBuildTestAndDeploy workflow below. It is the developer’s responsibility to delete resources their branches create even after merging and deleting branches as there is no automated cleanup.

Entire sample integration workflow.
Figure 6. Entire sample integration workflow.

After I have validated the syntax of my workflow by clicking Validate, I then click Commit. Confirm this action by clicking Commit in the Commit workflow modal window.

Commit workflow dialog window.
Figure 7. Commit workflow dialog window.

Immediately after committing the workflow, I can see the new OnIntegrationPushBuildTestAndDeploy workflow in my list of workflows. I see that the workflow shows the “Workflow is inactive”. This is expected as I am looking at the main branch and the trigger is not invoked from main.

Now that I have finished the implementation details of GitFlow, I am now going to create the permanent develop branch and a feature branch to test my integration workflow. To add a new branch, I go to Code > Source repositories > static-website-content, select Create branch under the More menu.

Source repository Actions menu.
Figure 8. Source repository Actions menu.

Enter develop as my branch name, create the branch from main, and then click Create.

Create the develop branch from main.
Figure 9. Create the develop branch from main.

I now add a feature branch by navigating back to the create branch screen. This time, I enter feature/gitflow-demo as my branch name, create the branch from develop, and then click Create.

Create a feature branch from develop.
Figure 10. Create a feature branch from develop.

To confirm that I have successfully implemented GitFlow, I need to verify that the feature branch workflow is running. I return to CI/CD > Workflows, select feature/gitflow-demo from the branch dropdown, and see the integration workflow is running.

Feature branch running integration workflow.
Figure 11. Feature branch running integration workflow.

To complete my testing of my implementation of GitFlow, I wait for the workflow to succeed. Then I view the newly deployed branch by navigating to the workflow and clicking on the View app link located on the last workflow action.

Lastly, now that GitFlow is implemented and tested, I will step through getting the feature branch to production. After I make my code changes to the feature branch, I create a pull request to merge feature/gitflow-demo into develop. Note that pull requests were covered in the prior post in this series. When merging the pull request select Delete the source branch after merging this pull request, as the feature branch is not a permanent branch.

Deleting the feature branch when merging.
Figure 12. Deleting the feature branch when merging.

Now that my changes are in the develop branch, I create a release branch. I navigate back to the create branch screen. This time I enter release/v1 as my branch name, create the branch from develop, and then click Create.

Create the release branch from main.
Figure 13. Create the release branch from main.

I am ready to release to production, so I create a pull request to merge release/v1 into main. The release branch is not a permanent branch, so it can also be deleted on merge. When the pull request is merged to main, the OnPushBuildTestAndDeploy workflow runs. After the workflow finishes running, I can verify my changes are in production.

Cleanup

If you have been following along with this workflow, you should delete the resources you deployed so you do not continue to incur charges. First, delete the two stacks that deployed using the AWS CloudFormation console in the AWS account(s) you associated when you launched the blueprint and configured the new environment. These stacks will have names like static-web-XXXXX. Second, delete the project from CodeCatalyst by navigating to Project settings and choosing Delete project.

Conclusion

In this post, you learned how to use triggers and environments in multiple workflows to implement GitFlow with Amazon CodeCatalyst. By consuming variables inside workflows, I was able to further customize my deployment design. Using these concepts, you can now implement your team’s branching strategy with CodeCatalyst. Learn more about Amazon CodeCatalyst and get started today!

Michael Ohde

Michael Ohde is a Senior Solutions Architect from Long Beach, CA. As a Product Acceleration Solution Architect at AWS, he currently assists Independent Software Vendor (ISVs) in the GovTech and EdTech sectors, by building modern applications using practices like serverless, DevOps, and AI/ML.