Post Syndicated from Jehu Gray original https://aws.amazon.com/blogs/devops/how-to-add-notifications-and-manual-approval-to-an-aws-cdk-pipeline/
A deployment pipeline typically comprises several stages such as dev, test, and prod, which ensure that changes undergo testing before reaching the production environment. To improve the reliability and stability of release processes, DevOps teams must review Infrastructure as Code (IaC) changes before applying them in production. As a result, implementing a mechanism for notification and manual approval that grants stakeholders improved access to changes in their release pipelines has become a popular practice for DevOps teams.
Notifications keep development teams and stakeholders informed in real-time about updates and changes to deployment status within release pipelines. Manual approvals establish thresholds for transitioning a change from one stage to the next in the pipeline. They also act as a guardrail to mitigate risks arising from errors and rework because of faulty deployments.
Please note that manual approvals, as described in this post, are not a replacement for the use of automation. Instead, they complement automated checks within the release pipeline.
In this blog post, we describe how to set up notifications and add a manual approval stage to AWS Cloud Development Kit (AWS CDK) Pipeline.
Concepts
CDK Pipeline
CDK Pipelines is a construct library for painless continuous delivery of CDK applications. CDK Pipelines can automatically build, test, and deploy changes to CDK resources. CDK Pipelines are self-mutating which means as application stages or stacks are added, the pipeline automatically reconfigures itself to deploy those new stages or stacks. Pipelines need only be manually deployed once, afterwards, the pipeline keeps itself up to date from the source code repository by pulling the changes pushed to the repository.
Notifications
Adding notifications to a pipeline provides visibility to changes made to the environment by utilizing the NotificationRule construct. You can also use this rule to notify pipeline users of important changes, such as when a pipeline starts execution. Notification rules specify both the events and the targets, such as Amazon Simple Notification Service (Amazon SNS) topic or AWS Chatbot clients configured for Slack which represents the nominated recipients of the notifications. An SNS topic is a logical access point that acts as a communication channel while Chatbot is an AWS service that enables DevOps and software development teams to use messaging program chat rooms to monitor and respond to operational events.
Manual Approval
In a CDK pipeline, you can incorporate an approval action at a specific stage, where the pipeline should pause, allowing a team member or designated reviewer to manually approve or reject the action. When an approval action is ready for review, a notification is sent out to alert the relevant parties. This combination of notifications and approvals ensures timely and efficient decision-making regarding crucial actions within the pipeline.
Solution Overview
The solution explains a simple web service that is comprised of an AWS Lambda function that returns a static web page served by Amazon API Gateway. Since Continuous Deployment and Continuous Integration (CI/CD) are important components to most web projects, the team implements a CDK Pipeline for their web project.
There are two important stages in this CDK pipeline; the Pre-production stage for testing and the Production stage, which contains the end product for users.
The flow of the CI/CD process to update the website starts when a developer pushes a change to the repository using their Integrated Development Environment (IDE). An Amazon CloudWatch event triggers the CDK Pipeline. Once the changes reach the pre-production stage for testing, the CI/CD process halts. This is because a manual approval gate is between the pre-production and production stages. So, it becomes a stakeholder’s responsibility to review the changes in the pre-production stage before approving them for production. The pipeline includes an SNS notification that notifies the stakeholder whenever the pipeline requires manual approval.
After approving the changes, the CI/CD process proceeds to the production stage and the updated version of the website becomes available to the end user. If the approver rejects the changes, the process ends at the pre-production stage with no impact to the end user.
The following diagram illustrates the solution architecture.
Figure 1. This image shows the CDK pipeline process in our solution and how applications or updates are deployed using AWS Lambda Function to end users.
Prerequisites
For this walkthrough, you should have the following prerequisites:
Add notification to the pipeline
In this tutorial, perform the following steps:
- Add the import statements for AWS CodeStar notifications and SNS to the import section of the pipeline stack py
import aws_cdk.aws_codestarnotifications as notifications
import aws_cdk.pipelines as pipelines
import aws_cdk.aws_sns as sns
import aws_cdk.aws_sns_subscriptions as subs
- Ensure the pipeline is built by calling the ‘build pipeline’ function.
pipeline.build_pipeline()
topic = sns.Topic(self, "MyTopic1")
- Add a subscription to the topic. This specifies where the notifications are sent (Add the stakeholders’ email here).
topic.add_subscription(subs.EmailSubscription("[email protected]"))
- Define a rule. This contains the source for notifications, the event trigger, and the target .
rule = notifications.NotificationRule(self, "NotificationRule", )
- Assign the source the value
pipeline.pipeline The first pipeline is the name of the CDK pipeline(variable) and the .pipeline is to show it is a pipeline(function).
source=pipeline.pipeline,
- Define the events to be monitored. Specify notifications for when the pipeline starts, when it fails, when the execution succeeds, and finally when manual approval is needed.
events=["codepipeline-pipeline-pipeline-execution-started", "codepipeline-pipeline-pipeline-execution-failed","codepipeline-pipeline-pipeline-execution-succeeded",
"codepipeline-pipeline-manual-approval-needed"],
- For the complete list of supported event types for pipelines, see here
- Finally, add the target. The target here is the topic created previously.
targets=[topic]
The combination of all the steps becomes:
pipeline.build_pipeline()
topic = sns.Topic(self, "MyTopic1")
topic.add_subscription(subs.EmailSubscription("[email protected]"))
rule = notifications.NotificationRule(self, "NotificationRule",
source=pipeline.pipeline,
events=["codepipeline-pipeline-pipeline-execution-started", "codepipeline-pipeline-pipeline-execution-failed","codepipeline-pipeline-pipeline-execution-succeeded",
"codepipeline-pipeline-manual-approval-needed"],
targets=[topic]
)
Adding Manual Approval
- Add the ManualApprovalStep import to the aws_cdk.pipelines import statement.
from aws_cdk.pipelines import (
CodePipeline,
CodePipelineSource,
ShellStep,
ManualApprovalStep
)
- Add the ManualApprovalStep to the production stage. The code must be added to the add_stage() function.
prod = WorkshopPipelineStage(self, "Prod")
prod_stage = pipeline.add_stage(prod,
pre = [ManualApprovalStep('PromoteToProduction')])
When a stage is added to a pipeline, you can specify the pre and post steps, which are arbitrary steps that run before or after the contents of the stage. You can use them to add validations like manual or automated gates to the pipeline. It is recommended to put manual approval gates in the set of pre steps, and automated approval gates in the set of post steps. So, the manual approval action is added as a pre step that runs after the pre-production stage and before the production stage .
- The final version of the
pipeline_stack.py becomes:
from constructs import Construct
import aws_cdk as cdk
import aws_cdk.aws_codestarnotifications as notifications
import aws_cdk.aws_sns as sns
import aws_cdk.aws_sns_subscriptions as subs
from aws_cdk import (
Stack,
aws_codecommit as codecommit,
aws_codepipeline as codepipeline,
pipelines as pipelines,
aws_codepipeline_actions as cpactions,
)
from aws_cdk.pipelines import (
CodePipeline,
CodePipelineSource,
ShellStep,
ManualApprovalStep
)
class WorkshopPipelineStack(cdk.Stack):
def __init__(self, scope: Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
# Creates a CodeCommit repository called 'WorkshopRepo'
repo = codecommit.Repository(
self, "WorkshopRepo", repository_name="WorkshopRepo",
)
#Create the Cdk pipeline
pipeline = pipelines.CodePipeline(
self,
"Pipeline",
synth=pipelines.ShellStep(
"Synth",
input=pipelines.CodePipelineSource.code_commit(repo, "main"),
commands=[
"npm install -g aws-cdk", # Installs the cdk cli on Codebuild
"pip install -r requirements.txt", # Instructs Codebuild to install required packages
"npx cdk synth",
]
),
)
# Create the Pre-Prod Stage and its API endpoint
deploy = WorkshopPipelineStage(self, "Pre-Prod")
deploy_stage = pipeline.add_stage(deploy)
deploy_stage.add_post(
pipelines.ShellStep(
"TestViewerEndpoint",
env_from_cfn_outputs={
"ENDPOINT_URL": deploy.hc_viewer_url
},
commands=["curl -Ssf $ENDPOINT_URL"],
)
)
deploy_stage.add_post(
pipelines.ShellStep(
"TestAPIGatewayEndpoint",
env_from_cfn_outputs={
"ENDPOINT_URL": deploy.hc_endpoint
},
commands=[
"curl -Ssf $ENDPOINT_URL",
"curl -Ssf $ENDPOINT_URL/hello",
"curl -Ssf $ENDPOINT_URL/test",
],
)
)
# Create the Prod Stage with the Manual Approval Step
prod = WorkshopPipelineStage(self, "Prod")
prod_stage = pipeline.add_stage(prod,
pre = [ManualApprovalStep('PromoteToProduction')])
prod_stage.add_post(
pipelines.ShellStep(
"ViewerEndpoint",
env_from_cfn_outputs={
"ENDPOINT_URL": prod.hc_viewer_url
},
commands=["curl -Ssf $ENDPOINT_URL"],
)
)
prod_stage.add_post(
pipelines.ShellStep(
"APIGatewayEndpoint",
env_from_cfn_outputs={
"ENDPOINT_URL": prod.hc_endpoint
},
commands=[
"curl -Ssf $ENDPOINT_URL",
"curl -Ssf $ENDPOINT_URL/hello",
"curl -Ssf $ENDPOINT_URL/test",
],
)
)
# Create The SNS Notification for the Pipeline
pipeline.build_pipeline()
topic = sns.Topic(self, "MyTopic")
topic.add_subscription(subs.EmailSubscription("[email protected]"))
rule = notifications.NotificationRule(self, "NotificationRule",
source = pipeline.pipeline,
events = ["codepipeline-pipeline-pipeline-execution-started", "codepipeline-pipeline-pipeline-execution-failed", "codepipeline-pipeline-manual-approval-needed", "codepipeline-pipeline-manual-approval-succeeded"],
targets=[topic]
)
When a commit is made with git commit -am "Add manual Approval" and changes are pushed with git push, the pipeline automatically self-mutates to add the new approval stage.
Now when the developer pushes changes to update the build environment or the end user application, the pipeline execution stops at the point where the approval action was added. The pipeline won’t resume unless a manual approval action is taken.
Figure 2. This image shows the pipeline with the added Manual Approval action.
Since there is a notification rule that includes the approval action, an email notification is sent with the pipeline information and approval status to the stakeholder(s) subscribed to the SNS topic.
Figure 3. This image shows the SNS email notification sent when the pipeline starts.
After pushing the updates to the pipeline, the reviewer or stakeholder can use the AWS Management Console to access the pipeline to approve or deny changes based on their assessment of these changes. This process helps eliminate any potential issues or errors and ensures only changes deemed relevant are made.
Figure 4. This image shows the review action that gives the stakeholder the ability to approve or reject any changes.
If a reviewer rejects the action, or if no approval response is received within seven days of the pipeline stopping for the review action, the pipeline status is “Failed.”
Figure 5. This image depicts when a stakeholder rejects the action.
If a reviewer approves the changes, the pipeline continues its execution.
Figure 6. This image depicts when a stakeholder approves the action.
Considerations
It is important to consider any potential drawbacks before integrating a manual approval process into a CDK pipeline. one such consideration is its implementation may delay the delivery of updates to end users. An example of this is business hours limitation. The pipeline process might be constrained by the availability of stakeholders during business hours. This can result in delays if changes are made outside regular working hours and require approval when stakeholders are not immediately accessible.
Clean up
To avoid incurring future charges, delete the resources. Use cdk destroy via the command line to delete the created stack.
Conclusion
Adding notifications and manual approval to CDK Pipelines provides better visibility and control over the changes made to the pipeline environment. These features ideally complement the existing automated checks to ensure that all updates are reviewed before deployment. This reduces the risk of potential issues arising from bugs or errors. The ability to approve or deny changes through the AWS Management Console makes the review process simple and straightforward. Additionally, SNS notifications keep stakeholders updated on the status of the pipeline, ensuring a smooth and seamless deployment process.
Prashant Agrawal is a Sr. Search Specialist Solutions Architect with Amazon OpenSearch Service. He works closely with customers to help them migrate their workloads to the cloud and helps existing customers fine-tune their clusters to achieve better performance and save on cost. Before joining AWS, he helped various customers use OpenSearch and Elasticsearch for their search and log analytics use cases. When not working, you can find him traveling and exploring new places. In short, he likes doing Eat → Travel → Repeat.
Sandeep Adwankar is a Senior Technical Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.
Srividya Parthasarathy is a Senior Big Data Architect on the AWS Lake Formation team. She enjoys building data mesh solutions and sharing them with the community.
Mahesh Mishra is a Principal Product Manager with AWS Lake Formation team. He works with many of AWS largest customers on emerging technology needs, and leads several data and analytics initiatives within AWS including strong support for Transactional Data Lakes.