Post Syndicated from Jay McConnell original https://aws.amazon.com/blogs/devops/integrating-git-with-aws-codepipeline/
AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. The service currently supports GitHub, AWS CodeCommit, and Amazon S3 as source providers. This blog post will cover how to integrate AWS CodePipeline with GitHub Enterprise, Bitbucket, GitLab, or any other Git server that supports the webhooks functionality available in most Git software.
Webhooks notify a remote service by issuing an HTTP POST when a commit is pushed to the repository. AWS Lambda receives the HTTP POST through Amazon API Gateway, and then downloads a copy of the repository. It places a zipped copy of the repository into a versioned S3 bucket. AWS CodePipeline can then use the zip file in S3 as a source; the pipeline will be triggered whenever the Git repository is updated.
There are two methods you can use to get the contents of a repository. Each method exposes Lambda functions that have different security and scalability properties.
- Zip download uses the Git provider’s HTTP API to download an already-zipped copy of the current state of the repository.
- No need for external libraries.
- Smaller Lambda function code.
- Large repo size limit (500 MB).
- Git pull uses SSH to pull from the repository. The repository contents are then zipped and uploaded to S3.
- Efficient for repositories with a high volume of commits, because each time the API is triggered, it downloads only the changed files.
- Suitable for any Git server that supports hooks and SSH; does not depend on personal access tokens or OAuth2.
- More extensible because it uses a standard Git library.
Build the required AWS resources
For your convenience, there is an AWS CloudFormation template that includes the AWS infrastructure and configuration required to build out this integration. To launch the CloudFormation stack setup wizard, click the link for your desired region. (The following AWS regions support all of the services required for this integration.)
For a list of services available in AWS regions, see the AWS Region Table.
The stack setup wizard will prompt you to enter several parameters. Many of these values must be obtained from your Git service.
OutputBucketName: The name of the bucket where your zipped code will be uploaded. CloudFormation will create a bucket with this name. For this reason, you cannot use the name of an existing S3 bucket.
Note: By default, there is no lifecycle policy on this bucket, so previous versions of your code will be retained indefinitely. If you want to control the retention period of previous versions, see Lifecycle Configuration for a Bucket with Versioning in the Amazon S3 User Guide.
AllowedIps: Used only with the git pull method described earlier. A comma-separated list of IP CIDR blocks used for Git provider source IP authentication. The Bitbucket Cloud IP ranges are provided as defaults.
ApiSecret: Used only with the git pull method described earlier. This parameter is used for webhook secrets in GitHub Enterprise and GitLab. If a secret is matched, IP range authentication is bypassed. The secret cannot contain commas (,), slashes (\), or quotation marks (“).
GitToken: Used only with the zip download method described earlier. This is a personal access token generated by GitHub Enterprise or GitLab.
OauthKey/OuathSecret: Used only with the zip download method described earlier. This is an OAuth2 key and secret provided by Bitbucket.
At least one parameter for your chosen method and provider must be set.
The process for setting up webhook secrets and API tokens differs between vendors and product versions. Consult your Git provider’s documentation for details.
After you have entered values for these parameters, you can complete the steps in the wizard and start the stack creation. If your desired values change over time, you can use CloudFormation’s update stack functionality to modify your parameters.
After the CloudFormation stack creation is complete, make a note of the GitPullWebHookApi, ZipDownloadWebHookApi, OutputBucketName and PublicSSHKey. You will need these in the following steps.
Configure the source repository
Depending on the method (git pull or zip download) you would like to use, in your Git provider’s interface, set the destination URL of your webhook to either the GitPullWebHookApi or ZipDownloadWebHookApi. If you create a secret at this point, be sure to update the ApiSecret parameter in your CloudFormation stack.
If you are using the git pull method, the Git repo is downloaded over SSH. For this reason, the PublicSSHKey output must be imported into Git as a deployment key.
Test a commit
After you have set up webhooks on your repository, run the git push command to create a folder structure and zip file in the S3 bucket listed in your CloudFormation output as OutputBucketName. If the zip file is not created, you can check the following sources for troubleshooting help:
- Webhook logs in your Git provider’s interface
- Monitoring and Troubleshooting in API Gateway
- Accessing Amazon CloudWatch Logs for AWS Lambda
Set up AWS CodePipeline
The final step is to create a pipeline in AWS CodePipeline using the zip file as an S3 source. For information about creating a pipeline, see the Simple Pipeline Walkthrough in the AWS CodePipeline User Guide. After your pipeline is set up, commits to your repository will trigger an update to the zip file in S3, which, in turn, triggers a pipeline execution.
We hope this blog post will help you integrate your Git server. Feel free to leave suggestions or approaches on integration in the comments.