All posts by Nikunj Vaidya

Automated code reviews on Bitbucket repositories and other enhancements in Amazon CodeGuru

Post Syndicated from Nikunj Vaidya original https://aws.amazon.com/blogs/devops/automated-code-reviews-on-bitbucket-repositories-and-other-enhancements-in-amazon-codeguru/

This post covers the support for the Atlassian Bitbucket Cloud source repository for Amazon CodeGuru Reviewer, which was recently announced. It also delves into new functionalities introduced to enhance the developer experience in CodeGuru Reviewer.

CodeGuru Reviewer is a machine learning-based service that scans your pull requests and gives you recommendations against your source code in Bitbucket with a description of what’s causing the issue and how to remediate it. CodeGuru Reviewer identifies code quality issues in five broad categories:

  • AWS best practices
  • Concurrency
  • Resource leaks
  • Sensitive information leaks
  • Common code bugs

You can also use CodeGuru Reviewer to provide code quality or AWS best practice recommendations when you migrate your code base to Java or adopt AWS services to achieve scale and robustness. The CodeGuru Reviewer recommendations offer unique capabilities in static code analysis, including the following:

  • Lower false positives
  • Difficult-to-identify issues like resource leaks and concurrency
  • Machine learning to evolve continuously from Amazon code bases
  • AWS best practices

In short, Amazon CodeGuru (Preview) equips your development team with the tools to maintain a high bar of coding standards in the software development process. For more information about configuring CodeGuru (Preview) for automated code reviews and performance optimization, see Automated code reviews and application profiling with Amazon CodeGuru.

 

In this blog, we will go over the following items:

  1. Using the Getting Started wizard.
  2. Associating the Bitbucket repository with the CodeGuru Reviewer and generating a pull request to trigger an automated code review.
  3. Using the new pull request code reviews dashboard to keep track of the history of pull requests and associated CodeGuru Reviewer recommendations
  4. Using supported APIs and AWS Command Line Interface (AWS CLI) to carry out CodeGuru Reviewer functions.

 

Using the Getting Started wizard

If you’re new to CodeGuru (Preview), you should follow the wizard guidance from the Getting started drop-down menu on the CodeGuru (Preview) console. This has been recently introduced to facilitate configuring the service.

Choose CodeGuru Profiler or CodeGuru Reviewer for configuration and follow the guided steps.

Screenshot of Getting Started Wizard

Associating a Bitbucket repository

This section summarizes the high level steps to associate the Bitbucket code repository with CodeGuru Reviewer. For more information, see What is Amazon CodeGuru Reviewer?

1.  On the CodeGuru (Preview) console, choose Reviewer.

2.  Choose Associated repositories.

3.  Choose Associate repository.

4.  Select Bitbucket.

5.  For Connect to Bitbucket, you can choose an existing connection from the drop-down menu or choose Create a Bitbucket connection.

 

 

6.  After choosing Create a Bitbucket connection, for Connection name, provide a name for your connection; for example, Bitbucket-Connection.

Screenshot of Create Connection Step1

7.  Choose Connect to Bitbucket Cloud.

A window opens to log in to Bitbucket.

8.  If you’re not already logged in, enter your Bitbucket login credentials.

9.  In the Bitbucket connection settings section, for Connection name, the name you entered in earlier step will be displayed.

10.  For Bitbucket Cloud apps, you can search for an existing app in the search text box or choose Install a new app.

Screenshot to Connect to Bitbucket

After you choose Install a new app, a pop-up window appears asking for authorization to grant access for AWS CodeStar.

11.  Choose Grant Access.

You now see a connection string populated in the Bitbucket Cloud apps field.

 

12.  Choose Connect.

You return to the earlier screen, which provides you with a drop-down menu of repositories from your Bitbucket account.

 

13.  Choose an appropriate repository and choose Associate.

 

You can see your repository is in the status Associating as shown below:

Screenshot for Reviewer connection with Bitbucket in Associating State

 

When the association is complete, the status shows as Associated. The following screenshot shows two repositories in the Associated state. This indicates that CodeGuru (Preview) is now listening for any pull request notifications from these repositories.

Screenshot showing in Associated state

 

Now you can go to the Bitbucket site and access your repository.

 

14.  On the Bitbucket website, create a pull request for the Bitbucket repository.

Screenshot for Pull-Request

 

CodeGuru (Preview) is notified of any pull requests created on that repository. This triggers the code review for the code referenced in the pull request. You can see generated recommendations on the Activity tab.

 

The following screenshot shows recommendations generated on the Bitbucket dashboard.

Screenshot for Recommendation

 

You can now take further actions to address the comments and merge the code.

 

Using the pull request code reviews dashboard

AWS introduced this dashboard based on early feedback requesting a centralized place to manage details about code review history. The pull request dashboard allows you to view CodeGuru Reviewer recommendations for all code reviews. This page lists all code reviews with accompanying information such as the status of the code review, the repository, the number of recommendations, and more.

PR Dashboard

 

Every code review is assigned a unique ARN that allows you to see the details of an individual code review, including any recommendations that may have surfaced.

 

The following screenshot shows the details of a code review.

PR Dashboard with Status

 

 

The following are key details:

  • Status – Confirms that the code review is complete. This is especially useful in use cases where there are no generated recommendations.
  • Metered lines of code – Refers to the lines of code scanned for the code request, excluding the lines without significant code (for example, commented code or lines with only opening or closing braces).
  • Pull request Id – Provides a link to navigate back to the code review page on the repository and review the complete code review activity.
  • Recommendations – Provides a text search capability to locate specific recommendations

 

The following screenshot shows an individual recommendation.

Individual Recommendations from PR Dashboard

 

You can give feedback on CodeGuru recommendations by choosing either the thumbs up or thumbs down icon below each recommendation. This gives you the opportunity to provide input about whether the recommendation was useful for you. These inputs enable AWS to evolve the service with more relevant recommendations.

 

Using the supported APIs and AWS CLI

The pull request code review dashboard includes the following APIs:

  • DescribeCodeReview
  • ListCodeReviews
  • ListRecommendations
  • PutRecommendationFeedback
  • DescribeRecommendationFeedback
  • ListRecommendationFeedback

In addition, you can use equivalent AWS CLI commands. To use the AWS CLI, you need to install the AWS CLI version 2. For more information about CodeGuru Reviewer operations, see codeguru-reviewer. For more information about CodeGuru Profiler operations, see codeguruprofiler.

 

The following are a few examples exercising the above API’s using aws cli’s:

Admin:~/environment $ aws codeguru-reviewer list-repository-associations
{
{
    "RepositoryAssociationSummaries": [
        {
            "AssociationArn": "arn:aws:codeguru-reviewer:<Region>:<AcctID>:association:11c1b6a3-638f-4a6c-bfc8-3e286785632a",
            "LastUpdatedTimeStamp": "2020-05-06T04:46:16.742000+00:00",
            "AssociationId": "11c1b6a3-638f-4a6c-bfc8-3e286785632a",
            "Name": "codeguruapp",
            "Owner": "nvaidya1",
            "ProviderType": "Bitbucket",
            "State": "Associated"
        },
        {
            "AssociationArn": "arn:aws:codeguru-reviewer:<Region>:<AcctID>:association:29ec5f42-34b4-448e-909e-76fc98bd8e59",
            "LastUpdatedTimeStamp": "2020-05-06T03:13:55.878000+00:00",
            "AssociationId": "29ec5f42-34b4-448e-909e-76fc98bd8e59",
            "Name": "MyJavaProject",
            "Owner": "nvaidya1",
            "ProviderType": "Bitbucket",
            "State": "Associated"
        }
    ]
}

 

Admin:~/environment $ aws codeguru-reviewer list-code-reviews --type PullRequest
{
    "CodeReviewSummaries": [
        {
            "Name": "BITBUCKET-codeguruapp-2-3099f39ad26a2d70e93d0288dfbd8ae301e925d5",
            "CodeReviewArn": "arn:aws:codeguru-reviewer:<Region>:<AcctID>:code-review:PullRequest-BITBUCKET-codeguruapp-2-3099f39ad26a2d70e93d0288dfbd8ae301e925d5",
            "RepositoryName": "codeguruapp",
            "Owner": "<Snip>",
            "ProviderType": "Bitbucket",
            "State": "Completed",
            "CreatedTimeStamp": "2020-05-06T04:47:43.868000+00:00",
            "LastUpdatedTimeStamp": "2020-05-06T04:51:47.492000+00:00",
            "Type": "PullRequest",
            "PullRequestId": "2",
            "MetricsSummary": {
                "MeteredLinesOfCodeCount": 74,
                "FindingsCount": 5
            }
        }
    ]
} 
<SNIP>

Admin:~/environment $ aws codeguru-reviewer list-recommendations --code-review-arn arn:aws:codeguru-reviewer:<Remaining-ARN-String>
{
    "RecommendationSummaries": [
        {
            "FilePath": "src/main/java/com/company/sample/application/EventHandler.java",
            "RecommendationId": "573efe3796b8b96f455591aa6eb4b675f77c95b9cf42f5a260ecc0dee0299e53",
            "StartLine": 170,
            "EndLine": 170,
            "Description": "This code is written so that the client cannot be reused across invocations of the Lambda function.\nTo improve the performance of the Lambda function, consider using static initialization/constructor, global/static variables and singletons. It allows to keep alive and reuse HTTP connections that were established during a previous invocation.\nLearn more about [best practices for working with AWS Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html)."
        },
<SNIP>

Admin:~/environment $ aws codeguru-reviewer describe-code-review --code-review-arn arn:aws:codeguru-reviewer:<Remaining-ARN-String>
{
    "CodeReview": {
        "Name": "BITBUCKET-codeguruapp-2-3099f39ad26a2d70e93d0288dfbd8ae301e925d5",
        "CodeReviewArn": "arn:aws:codeguru-reviewer:<Region>:<AcctID>:code-review:PullRequest-BITBUCKET-codeguruapp-2-3099f39ad26a2d70e93d0288dfbd8ae301e925d5",
        "RepositoryName": "codeguruapp",
        "Owner": "<snip>",
        "ProviderType": "Bitbucket",
        "State": "Completed",
        "StateReason": "CodeGuru Reviewer successfully finished reviewing the pull request source code.",
        "CreatedTimeStamp": "2020-05-06T04:47:43.868000+00:00",
        "LastUpdatedTimeStamp": "2020-05-06T04:51:47.492000+00:00",
        "Type": "PullRequest",
        "PullRequestId": "2",
        "SourceCodeType": {
            "CommitDiff": {
                "SourceCommit": "3099f39ad26a2d70e93d0288dfbd8ae301e925d5",
                "DestinationCommit": "7338cd2fd99e6e663b3a312e80e5ca2b570a6891"
            }
        },
        "Metrics": {
            "MeteredLinesOfCodeCount": 74,
            "FindingsCount": 5
        }
    }
}

Cleaning up

When you’re finished testing, you should un-provision the service to avoid incurring further charges:

  • CodeGuru Reviewer – Remove the association of CodeGuru (Preview) to the repository, so that any further pull request notifications doesn’t trigger CodeGuru (Preview) to perform an automated code review.
  • CodeGuru Profiler – If configured, remove the profiling group.

 

Conclusion

This post reviewed CodeGuru (Preview) support for Bitbucket repositories for CodeGuru Reviewer. It also reviewed the pull request dashboard and supported APIs and AWS CLI functionalities. You can take advantage of these features to enhance your application development workflow.

 

Automating code reviews and application profiling with Amazon CodeGuru

Post Syndicated from Nikunj Vaidya original https://aws.amazon.com/blogs/devops/automating-code-reviews-and-application-profiling-with-amazon-codeguru/

Amazon CodeGuru is a machine learning-based service released during re:Invent 2019 for automated code reviews and application performance recommendations. CodeGuru equips the development teams with the tools to maintain a high bar for coding standards in their software development process.

CodeGuru Reviewer helps developers avoid introducing issues that are difficult to detect, troubleshoot, reproduce, and root-cause. It also enables them to improve application performance. This not only improves the reliability of the software, but also cuts down the time spent chasing difficult issues like race conditions, slow resource leaks, thread safety issues, use of un-sanitized inputs, inappropriate handling of sensitive data, and application performance impact, to name a few.

CodeGuru is powered by machine learning, best practices, and hard-learned lessons across millions of code reviews and thousands of applications profiled on open source projects and internally at Amazon.

The service leverages machine-learning abilities to provide following two functionalities:

a) Reviewer: provides automated code reviews for static code analysis

b) Profiler: provides visibility into and recommendations about application performance during runtime

This blog post provides a short workshop to get a feel for both the above functionalities.

 

Solution overview

The following diagram illustrates a typical developer workflow in which the CodeGuru service is used in the code-review stage and the application performance-monitoring stage. The code reviewer is used for static code analysis backed with trained machine-learning models, and the profiler is used to monitor application performance when the code artifact is deployed and executed on the target compute.

Development Workflow

 

The following diagram depicts additional details to show the big picture in the overall schema of the CodeGuru workflow:

Big Picture Development Workflow
This blog workshop automates the deployment of a sample application from a GitHub link via an AWS CloudFormation template, including the dependencies needed. It also demonstrates the Reviewer functionality.

 

Pre-requisites

Follow these steps to get set up:

1. Set up your AWS Cloud9 environment and access the bash terminal, preferably in the us-east-1 region.

2. Ensure you have an individual GitHub account.

3. Configure an Amazon EC2 key-pair (preferably in the us-east-1 region) and make the .pem file available from the terminal being used.

 

CodeGuru Reviewer

This section demonstrates how to configure CodeGuru Reviewer functionality and associate it with the GitHub repository. Execute the following configuration steps:

Step 1: Fork the GitHub repository
First, log in to your GitHub account and navigate to this sample code. Choose Fork and wait for it to create a fork in your account, as shown in the following screenshot.

Figure shows how to fork a repository from GitHub

 

Step 2: Associate the GitHub repository
Log in to the CodeGuru dashboard and follow these steps:

1. Choose Reviewer from the left panel and choose Associate repository.

2. Choose GitHub and then choose Connect to GitHub.

3. Once authenticated and connection made, you can select the repository aws-codeguru-profiler-sample-application from the Repository location drop-down list and choose Associate, as shown in the following screenshot.

Associate repository with CodeGuru service

This associates the CodeGuru Reviewer with the specified repository and continues to listen for any pull-request events.

Associated repository with CodeGuru service

Step 3: Prepare your code
From your AWS Cloud9 terminal, clone the repository, create a new branch, using the following example commands:

git clone https://github.com/<your-userid>/aws-codeguru-profiler-sample-application.git
cd aws-codeguru-profiler-sample-application
git branch dev
git checkout dev
cd src/main/java/com/company/sample/application/

Open the file: CreateOrderThread.java and goto the line 63. Below line 63 which adds an order entry, insert the if statement under the comment to introduce an order entry check. Please indent the lines with spaces so they are well aligned as shown below.

SalesSystem.orders.put(orderDate, order);
//Check if the Order entered and present
if (SalesSystem.orders.containsKey(orderDate)) {
        System.out.println("New order verified to be present in hashmap: " + SalesSystem.orders.get(orderDate)); 
}
id++;

Once the above changes are introduced in the file, save and commit it to git and push it to the Repository.

git add .
git commit -s -m "Introducing new code that is potentially thread unsafe and inefficient"
cd ../../../../../../../
ls src/main/java/com/company/sample/application/

Now, upload the new branch to the GitHub repository using the following commands. Enter your credentials when asked to authenticate against your GitHub account:

git status
git push --set-upstream origin dev
ls

Step 4: Create a Pull request on GitHub:
In your GitHub account, you should see a new branch: dev.

1. Go to your GitHub account and choose the Pull requests tab.

2. Select New pull request.

3. Under Comparing Changes, select <userid>/aws-codeguru-profiler-sample-application as the source branch.

4. Select the options from the two drop-down lists that selects a merge proposal from the dev branch to the master branch, as shown in the following screenshot.

5. Review the code diffs shown. It should say that the diffs are mergeable (able to merge). Choose Create Pull request to complete the process.

Creating Pull request

This sends a Pull request notification to the CodeGuru service and is reflected on the CodeGuru dashboard, as shown in the following screenshot.

CodeGuru Dashboard

After a short time, a set of recommendations appears on the same GitHub page on which the Pull request was created.

The demo profiler configuration and recommendations shown on the dashboard are provided by default as a sample application profile. See the profiler section of this post for further discussion.

The following screenshot shows a recommendation generated about potential thread concurrency susceptibility:

 

CodeGuru Recommendations on GitHub

 

Another example below to show how the developer can provide feedback about recommendations using emojis:

How to provide feedback for CodeGuru recommendations

As you can see from the recommendations, not only are the code issues detected, but a detailed recommendation is also provided on how to fix the issues, along with a link to examples, and documentation, wherever applicable. For each of the recommendations, a developer can give feedback about whether the recommendation was useful or not with a simple emoji selection under Pick your reaction.

Please note that the CodeGuru service is used to identify difficult-to-find functional defects and not syntactical errors. Syntax errors should be flagged by the IDE and addressed at an early stage of development. CodeGuru is introduced at a later stage in a developer workflow, when the code is already developed, unit-tested, and ready for code-review.

 

CodeGuru Profiler

CodeGuru Profiler functionality focuses on searching for application performance optimizations, identifying your most “expensive” lines of code that take unnecessarily long times or higher-than-expected CPU cycles, for which there is a better/faster/cheaper alternative. It generates recommendations with actions you can take in order to reduce your CPU use, lower your compute costs, and overall improve your application’s performance. The profiler simplifies the troubleshooting and exploration of the application’s runtime behavior using visualizations. Examples of such issues include excessive recreation of expensive objects, expensive deserialization, usage of inefficient libraries, and excessive logging.

This post provides two sample application Demo profiles by default in the profiler section to demonstrate the visualization of CPU and latency characteristics of those profiles. This offers a quick and easy way to check the profiler output without onboarding an application. Additionally, there are recommendations shown for the {CodeGuru} DemoProfilingGroup-WithIssues application profile. However, if you would like to run a proof-of-concept with real application, please follow the procedure below.

The following steps launch a sample application on Amazon EC2 and configure Profiler to monitor the application performance from the CodeGuru service.

Step 1: Create a profiling group
Follow these steps to create a profiling group:

1. From the CodeGuru dashboard, choose Profiler from the left panel.

2. Under Profiling groups, select Create profiling group and type the name of your group. This workshop uses the name DemoProfilingGroup.

3. After typing the name, choose Create in the bottom right corner.

The output page shows you instructions on how to onboard the CodeGuru Profiler Agent library into your application, along with the necessary permissions required to successfully submit data to CodeGuru. This workshop uses the AWS CloudFormation template to automate the onboarding configuration and launch Amazon EC2 with the application and its dependencies.

Step 2: Run AWS Cloudformation to launch Amazon EC2 with the Java application:
This example runs an AWS CloudFormation template that does all the undifferentiated heavy lifting of launching an Amazon EC2 machine and installing JDK, Maven, and the sample demo application.

Once done, it configures the application to use a profiling group named DemoProfilingGroup, compiles the application, and executes it as a background process. This results in the sample demo application running in the region you choose, and submits profiling data to the CodeGuru Profiler Service under the DemoProfilingGroup profiling group created in the previous step.

To launch the AWS CloudFormation template that deploys the demo application, choose the following Launch Stack button, and fill in the Stack name, Key-pair name, and Profiling Group name.

Launch Button

Once the AWS CloudFormation deployment succeeds, log in to your terminal of choice and use ssh to connect to the Amazon EC2 machine. Check the running application using the following commands:

ssh -i '<path-to-keypair.pem-file>' [email protected]<ec2-ip-address>
java -version
mvn -v
ps -ef | grep SalesSystem  => This is the java application running in the background
tail /var/log/cloud-init-output.log  => You should see the following output in 10-15 minutes as INFO: Successfully reported profile

Once the CodeGuru agent is imported into the application, a separate profiler thread spawns when the application runs. It samples the application CPU and Latency characteristics and delivers them to the backend Profiler service for building the application profile.

Step 3: Check the Profiler flame-graphs:
Wait for 10-15 minutes for your profiling-group to become active (if not already) and for profiling data to be submitted and aggregated by the CodeGuru Profiler service.

Visit the Profiling Groups page and choose DemoProfilingGroup. You should see the following page showing your application’s profiling data in a visualization called a flame-graph, as shown in the screenshot below. Detailed explanation about flame-graphs and how to read them follow.

Profiler flame-graph visualization

Profiler extensively uses flame-graph visualizations to display your application’s profiling data since they’re a great way to convey issues once you understand how to read them.

The x-axis shows the stack profile population (collection of stack traces) sorted alphabetically (not chronologically), and the y-axis shows stack depth, counting from zero at the bottom. Each rectangle represents a stack frame. The wider a frame is is, the more often it was present in the stacks. The top edge shows what is on CPU, and beneath it is its ancestry. The colors are usually not significant (they’re picked randomly to differentiate frames).

As shown in the preceding screenshot, the stack traces for the three threads are shown, which are triggered by the code in the SalesSystem.java file.

1) createOrderThread.run

2) createIllegalOrderThread.run

3) listOrderThread.run

The flame-graph also depicts the stack depth and points out specific function names when you hover over that block. The marked areas in the flame-graph highlight the top block functions on-CPU and spikes in stack-trace. This may indicate an opportunity to optimize.

It is evident from the preceding diagram that significant CPU time is being used by an exception stack trace (leftmost). It’s also highlighted in the recommendation report as described in Step 4 below.

The exception is caused by trying to instantiate an Enum class giving it invalid String values. If you review the file CreateIllegalOrderThread.java, you should notice the constructors being called with illegal product names, which are defined in ProductName.java.

Step 4: Profiler Recommendations:
Apart from the real-time visualization of application performance described in the preceding section, a recommendation report (generated after a period of time) may appear, pointing out suspected inefficiencies to fix to improve the application performance. Once the recommendation appears, select the Recommendation link to see the details.

Each section in the Recommendations report can be expanded in order to get instructions on how to resolve the issue, or to examine several locations in which there were issues in your data, as shown in the following screenshot.

CodeGuru profiler recommendations

In the preceding example, the report includes an issue named Checking for Values not in enum, in which it conveys that more time (15.4%) was spent processing exceptions than expected (less than 1%). The reason for the exceptions is described in Step 3 and the resolution recommendations are provided in the report.

 

CodeGuru supportability:

CodeGuru currently supports native Java-based applications for the Reviewer and Profiler functionality. The Reviewer functionality currently supports AWS CodeCommit and all cloud-hosted non-enterprise versions of GitHub products, including Free/Pro/Team, as code repositories.

Amazon CodeGuru Profiler does not have any code repository dependence and works with Java applications hosted on Amazon EC2, containerized applications running on Amazon ECS and Amazon EKS, serverless applications running on AWS Fargate, and on-premises hosts with adequate AWS credentials.

 

Cleanup

At the end of this workshop, once the testing is completed, follow these steps to disable the service to avoid incurring any further charges.

1. Reviewer: Remove the association of the CodeGuru service to the repository, so that any further Pull-request notifications don’t trigger the CodeGuru service to perform an automated code-review.

2. Profiler: Remove the profiling group.

3. Amazon EC2 Compute: Go to the Amazon EC2 service, select the CodeGuru EC2 machine, and select the option to terminate the Amazon EC2 compute.

 

Conclusion

This post reviewed the CodeGuru service and implemented code examples for the Reviewer and Profiler functionalities. It described Reviewer functionality providing automated code-reviews and detailed guidance on fixes. The Profiler functionality enabled you to visualize your real-time application stack for granular inspection and generate recommendations that provided guidance on performance improvements.

I hope this post was informative and enabled you to onboard and test the service, as well as to leverage this service in your application development workflow.

 

About the Author

Author Photo

 

Nikunj Vaidya is a Sr. Solutions Architect with Amazon Web Services, focusing in the area of DevOps services. He builds technical content for the field enablement and offers technical guidance to the customers on AWS DevOps solutions and services that would streamline the application development process, accelerate application delivery, and enable maintaining a high bar of software quality.

 

 

Deploying applications at FINRA using AWS CodeDeploy

Post Syndicated from Nikunj Vaidya original https://aws.amazon.com/blogs/devops/deploying-applications-at-finra-using-aws-codedeploy/

by Geethalaksmi Ramachandran (FINRA – Director, Application Engineering), Avinash Chukka (FINRA – Senior Application Engineer)

At FINRA, a financial regulatory organization that oversees the broker-dealer industry with market intelligence, we have been utilizing the AWS CodeDeploy services to deploy applications on the cloud as well as on on-premises production servers. This blog post provides insight into our operations and experience with CodeDeploy.

Migration overview

Since 2014, we have gone through a systematic effort to migrate over 100 applications from on-premises resources to the AWS Cloud for everything from case management to data ingestion and processing. The applications comprise multiple components or microservices as individually deployable units or a single monolith application with multiple shared components.

Most of the application components, running on Linux and Windows, were entirely redesigned, containerized, and gradually deployed from on-premises to cloud-based Amazon ECS clusters. Fifteen applications were to be deferred for migration or planned for retirement due to other dependencies. Those deferred applications are currently running on on-premises bare-metal servers hosted across 38 Windows servers and 15 Linux servers per environment with a total of 150 Windows servers and 60 Linux servers across all the environments.

The applications were deployed to the on-premises servers earlier using the XL Deploy tool from XebiaLabs. The tool has now been decommissioned and replaced by CodeDeploy to attain more reliability and consistency in deployments across various applications.

Infrastructure and Workflow overview

FINRA’s AWS Cloud infrastructure consists of Amazon EC2 instances, ECS, Amazon EMR clusters, and many resources from other AWS services. We host web applications in ECS clusters, running approximately 200 clusters on each environment and EC2 instances. The infrastructure uses AWS CloudFormation and AWS Java SDK as Infrastructure-as-Code (IaC).

The CI/CD pipeline comprises of:

  • A source stage (per branch) stored in a BitBucket repository
  • A build stage executed on Jenkins build slaves
  • A deploy stage involving deployment from:
    • AWS CloudFormation and SDK to ECS clusters
    • CodeDeploy to EC2 instances
    • CodeDeploy Service to on-premises servers across various development, quality assurance (QA), staging, and production environments.

Jenkins masters running on EC2 instances within an Auto Scaling group orchestrate the CI/CD pipeline. The master spawns the build slaves as ECS tasks to execute a build job. Once the image is built and containerized in the build stage, the build artifact is stored in Artifactory repos for shared common libraries or staged in S3 to be used in deployments. The Jenkins slave invokes the appropriate deployment service – AWS CloudFormation and SDK or CodeDeploy depending on the target server environment, as detailed in the preceding paragraphs. On completion of the deployment, the automated smoke tests are launched.

The following diagram depicts the CD workflow for the on-premises instances:

The Delivery pipeline deploys across various environments such as development, QA, user acceptance testing (UAT), and production. Approval gates control deployments to UAT and production.

CodeDeploy Operations

Our experience of utilizing CodeDeploy services has been “very smooth” since we moved from the XebiaLabs XL Deploy tool three months ago. The main factors that led FINRA to select CodeDeploy for our organization were:

  • Being able to use the same set of deployment tools between on-premises and cloud-based instances
  • Easy portability and reuse of scripts
  • Shared community knowledge rather than isolated expertise

The default deployment parameters were well-suited for our environments and didn’t require altering values or customization. Depending on the application being deployed, deployments can be carried out on cloud-based instances or on-premises instances. Cloud-based instances use AWS CloudFormation templates to trigger CodeDeploy; on-premises instances use AWS CLI-based scripts to trigger CodeDeploy.

The cloud-based deployments in production follow “blue-green” strategy for some of the applications, in which rollback is a critical requirement for minimal disruption. Other applications in cloud follow the “rolling updates” method, where as the on-premises servers in production are upgraded using “in-place” deployment method. The CodeDeploy agents running on on-premises servers are configured with roles to query for required artifacts stored on specific S3 buckets when deploying the package.

The applications’ deployment mappings to the instances are configured based on EC2 Auto Scaling groups in the cloud and based on tags for on-premises resources. Each component is logically mapped to a CodeDeploy deployment group. However, at one point, the maximum number of tags added to the CodeDeploy on-premises instance was restricted to a maximum tag limit of 10, but the instance needed 13 tags corresponding to 13 deployment groups.

We overcame this limitation by adding a common tenth tag on the on-premises instance and also on the remaining deployment groups (10-13) and stored the mapping of instances to deployment groups externally. The deployment script first looks up the mapping and proceeds with the deployment by validating if it matches the target server name, then runs deployments only on the matching target servers and skips deployments on the unmatched servers, as shown in the following diagram.

CodeDeploy offers the following benefits to FINRA:

  • Deployment configurations written as code: CodeDeploy configuration uses CloudFormation templates as Infrastructure-as-Code, which makes it easier to create and maintain.
  • Version controlled deployment code: AWS CloudFormation templates, deployment configuration, and deployment scripts are maintained in the source code repository and version-controlled.
  • Reusability: Most CodeDeploy resource provisioning code is reusable across all the on-premises instances and on different platforms, such as Linux (RHEL) and Windows.
  • Zero maintenance of deployment tool: As a managed service, CodeDeploy does not require maintenance and upgrade.
  • Secrets Management: CodeDeploy integrates with central secrets management systems, and externalizes environment configurations.

Monitoring

FINRA uses the in-house developed DevOps Dashboard to monitor the build and deploy stages, based upon a Grafana UI extracting CI and CD data from Jenkins.

The cloud instances and on-premises servers are configured with agents to stream the real-time logs to a central Splunk Server, where the logs are analyzed and health-monitored. Optionally, the deployment logs are forwarded to the functional owners via email attachments. These logs become critical for the troubleshooting activity in post-mortems of past events. Due to the restrictions on accessing the base instances across various functional teams, the above mechanism enables us to gain visibility into health of the CI/CD infrastructure.

Looking Forward

We plan to migrate the remaining on-premises applications to the cloud after necessary refactoring and retiring of the application dependencies over the next few years.

Our eventual goal is to move towards serverless technologies to eliminate server infrastructure management.

Conclusion

This post reviewed the Infrastructure at FINRA on both AWS Cloud and on-premises, the CI/CD pipeline, and the CodeDeploy workflow integration, as well as examining insights into CodeDeploy use.

The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this post.