Tag Archives: AWS CodeArtifact

Using NuGet with AWS CodeArtifact

Post Syndicated from John Standish original https://aws.amazon.com/blogs/devops/using-nuget-with-aws-codeartifact/

Managing NuGet packages for .NET development can be a challenge. Tasks such as initial configuration, ongoing maintenance, and scaling inefficiencies are the biggest pain points for developers and organizations. With its addition of NuGet package support, AWS CodeArtifact now provides easy-to-configure and scalable package management for .NET developers. You can use NuGet packages stored in CodeArtifact in Visual Studio, allowing you to use the tools you already know.

In this post, we show how you can provision NuGet repositories in 5 minutes. Then we demonstrate how to consume packages from your new NuGet repositories, all while using .NET native tooling.

All relevant code for this post is available in the aws-codeartifact-samples GitHub repo.

Prerequisites

For this walkthrough, you should have the following prerequisites:

Architecture overview

Two core resource types make up CodeArtifact: domains and repositories. Domains provide an easy way manage multiple repositories within an organization. Repositories store packages and their assets. You can connect repositories to other CodeArtifact repositories, or popular public package repositories such as nuget.org, using upstream and external connections. For more information about these concepts, see AWS CodeArtifact Concepts.

The following diagram illustrates this architecture.

AWS CodeArtifact core concepts

Figure: AWS CodeArtifact core concepts

Creating CodeArtifact resources with AWS CloudFormation

The AWS CloudFormation template provided in this post provisions three CodeArtifact resources: a domain, a team repository, and a shared repository. The team repository is configured to use the shared repository as an upstream repository, and the shared repository has an external connection to nuget.org.

The following diagram illustrates this architecture.

Example AWS CodeArtifact architecture

Figure: Example AWS CodeArtifact architecture

The following CloudFormation template used in this walkthrough:

AWSTemplateFormatVersion: '2010-09-09'
Description: AWS CodeArtifact resources for dotnet

Resources:
  # Create Domain
  ExampleDomain:
    Type: AWS::CodeArtifact::Domain
    Properties:
      DomainName: example-domain
      PermissionsPolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS: 
              - !Sub arn:aws:iam::${AWS::AccountId}:root
            Resource: "*"
            Action:
              - codeartifact:CreateRepository
              - codeartifact:DescribeDomain
              - codeartifact:GetAuthorizationToken
              - codeartifact:GetDomainPermissionsPolicy
              - codeartifact:ListRepositoriesInDomain

  # Create External Repository
  MyExternalRepository:
    Type: AWS::CodeArtifact::Repository
    Condition: ProvisionNugetTeamAndUpstream
    Properties:
      DomainName: !GetAtt ExampleDomain.Name
      RepositoryName: my-external-repository       
      ExternalConnections:
        - public:nuget-org
      PermissionsPolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS: 
              - !Sub arn:aws:iam::${AWS::AccountId}:root
            Resource: "*"
            Action:
              - codeartifact:DescribePackageVersion
              - codeartifact:DescribeRepository
              - codeartifact:GetPackageVersionReadme
              - codeartifact:GetRepositoryEndpoint
              - codeartifact:ListPackageVersionAssets
              - codeartifact:ListPackageVersionDependencies
              - codeartifact:ListPackageVersions
              - codeartifact:ListPackages
              - codeartifact:PublishPackageVersion
              - codeartifact:PutPackageMetadata
              - codeartifact:ReadFromRepository

  # Create Repository
  MyTeamRepository:
    Type: AWS::CodeArtifact::Repository
    Properties:
      DomainName: !GetAtt ExampleDomain.Name
      RepositoryName: my-team-repository
      Upstreams:
        - !GetAtt MyExternalRepository.Name
      PermissionsPolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS: 
              - !Sub arn:aws:iam::${AWS::AccountId}:root
            Resource: "*"
            Action:
              - codeartifact:DescribePackageVersion
              - codeartifact:DescribeRepository
              - codeartifact:GetPackageVersionReadme
              - codeartifact:GetRepositoryEndpoint
              - codeartifact:ListPackageVersionAssets
              - codeartifact:ListPackageVersionDependencies
              - codeartifact:ListPackageVersions
              - codeartifact:ListPackages
              - codeartifact:PublishPackageVersion
              - codeartifact:PutPackageMetadata
              - codeartifact:ReadFromRepository

Getting the CloudFormation template

To use the CloudFormation stack, we recommend you clone the following GitHub repo so you also have access to the example projects. See the following code:

git clone https://github.com/aws-samples/aws-codeartifact-samples.git
cd aws-codeartifact-samples/getting-started/dotnet/cloudformation/

Alternatively, you can copy the previous template into a file on your local filesystem named deploy.yml.

Provisioning the CloudFormation stack

Now that you have a local copy of the template, you need to provision the resources using a CloudFormation stack. You can deploy the stack using the AWS CLI or on the AWS CloudFormation console.

To use the AWS CLI, enter the following code:

aws cloudformation deploy \
--template-file deploy.yml \
--region <YOUR_PREFERRED_REGION> \
--stack-name CodeArtifact-GettingStarted-DotNet

To use the AWS CloudFormation console, complete the following steps:

  1. On the AWS CloudFormation console, choose Create stack.
  2. Choose With new resources (standard).
  3. Select Upload a template file.
  4. Choose Choose file.
  5. Name the stack CodeArtifact-GettingStarted-DotNet.
  6. Continue to choose Next until prompted to create the stack.

Configuring your local development experience

We use the CodeArtifact credential provider to connect the Visual Studio IDE to a CodeArtifact repository. You need to download and install the AWS Toolkit for Visual Studio to configure the credential provider. The toolkit is an extension for Microsoft Visual Studio on Microsoft Windows that makes it easy to develop, debug, and deploy .NET applications to AWS. The credential provider automates fetching and refreshing the authentication token required to pull packages from CodeArtifact. For more information about the authentication process, see AWS CodeArtifact authentication and tokens.

To connect to a repository, you complete the following steps:

  1. Configure an account profile in the AWS Toolkit.
  2. Copy the source endpoint from the AWS Explorer.
  3. Set the NuGet package source as the source endpoint.
  4. Add packages for your project via your CodeArtifact repository.

Configuring an account profile in the AWS Toolkit

Before you can use the Toolkit for Visual Studio, you must provide a set of valid AWS credentials. In this step, we set up a profile that has access to interact with CodeArtifact. For instructions, see Providing AWS Credentials.

Visual Studio Toolkit for AWS Account Profile Setup

Figure: Visual Studio Toolkit for AWS Account Profile Setup

Copying the NuGet source endpoint

After you set up your profile, you can see your provisioned repositories.

  1. In the AWS Explorer pane, navigate to the repository you want to connect to.
  2. Choose your repository (right-click).
  3. Choose Copy NuGet Source Endpoint.
AWS CodeArtifact repositories shown in the AWS Explorer

Figure: AWS CodeArtifact repositories shown in the AWS Explorer

 

You use the source endpoint later to configure your NuGet package sources.

Setting the package source using the source endpoint

Now that you have your source endpoint, you can set up the NuGet package source.

  1. In Visual Studio, under Tools, choose Options.
  2. Choose NuGet Package Manager.
  3. Under Options, choose the + icon to add a package source.
  4. For Name , enter codeartifact.
  5. For Source, enter the source endpoint you copied from the previous step.
Configuring Nuget package sources for AWS CodeArtifact

Figure: Configuring NuGet package sources for AWS CodeArtifact

 

Adding packages via your CodeArtifact repository

After the package source is configured against your team repository, you can pull packages via the upstream connection to the shared repository.

  1. Choose Manage NuGet Packages for your project.
    • You can now see packages from nuget.org.
  2. Choose any package to add it to your project.
Exploring packages while connected to a AWS CodeArtifact repository

Exploring packages while connected to a AWS CodeArtifact repository

Viewing packages stored in your CodeArtifact team repository

Packages are stored in a repository you pull from, or referenced via the upstream connection. Because we’re pulling packages from nuget.org through an external connection, you can see cached copies of those packages in your repository. To view the packages, navigate to your repository on the CodeArtifact console.

Packages stored in a AWS CodeArtifact repository

Packages stored in a AWS CodeArtifact repository

Cleaning Up

When you’re finished with this walkthrough, you may want to remove any provisioned resources. To remove the resources that the CloudFormation template created, navigate to the stack on the AWS CloudFormation console and choose Delete Stack. It may take a few minutes to delete all provisioned resources.

After the resources are deleted, there are no more cleanup steps.

Conclusion

We have shown you how to set up CodeArtifact in minutes and easily integrate it with NuGet. You can build and push your package faster, from hours or days to minutes. You can also integrate CodeArtifact directly in your Visual Studio environment with four simple steps. With CodeArtifact repositories, you inherit the durability and security posture from the underlying storage of CodeArtifact for your packages.

As of November 2020, CodeArtifact is available in the following AWS Regions:

  • US: US East (Ohio), US East (N. Virginia), US West (Oregon)
  • AP: Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo)
  • EU: Europe (Frankfurt), Europe (Ireland), Europe (Stockholm)

For an up-to-date list of Regions where CodeArtifact is available, see AWS CodeArtifact FAQ.

About the Authors

John Standish

John Standish is a Solutions Architect at AWS and spent over 13 years as a Microsoft .Net developer. Outside of work, he enjoys playing video games, cooking, and watching hockey.

Nuatu Tseggai

Nuatu Tseggai is a Cloud Infrastructure Architect at Amazon Web Services. He enjoys working with customers to design and build event-driven distributed systems that span multiple services.

Neha Gupta

Neha Gupta is a Solutions Architect at AWS and have 16 years of experience as a Database architect/ DBA. Apart from work, she’s outdoorsy and loves to dance.

Elijah Batkoski

Elijah is a Technical Writer for Amazon Web Services. Elijah has produced technical documentation and blogs for a variety of tools and services, primarily focused around DevOps.

Publishing private npm packages with AWS CodeArtifact

Post Syndicated from Ryan Sonshine original https://aws.amazon.com/blogs/devops/publishing-private-npm-packages-aws-codeartifact/

This post demonstrates how to create, publish, and download private npm packages using AWS CodeArtifact, allowing you to share code across your organization without exposing your packages to the public.

The ability to control CodeArtifact repository access using AWS Identity and Access Management (IAM) removes the need to manage additional credentials for a private npm repository when developers already have IAM roles configured.

You can use private npm packages for a variety of use cases, such as:

  • Reducing code duplication
  • Configuration such as code linting and styling
  • CLI tools for internal processes

This post shows how to easily create a sample project in which we publish an npm package and install the package from CodeArtifact. For more information about pipeline integration, see AWS CodeArtifact and your package management flow – Best Practices for Integration.

Solution overview

The following diagram illustrates this solution.

Diagram showing npm package publish and install with CodeArtifact

In this post, you create a private scoped npm package containing a sample function that can be used across your organization. You create a second project to download the npm package. You also learn how to structure your npm package to make logging in to CodeArtifact automatic when you want to build or publish the package.

The code covered in this post is available on GitHub:

Prerequisites

Before you begin, you need to complete the following:

  1. Create an AWS account.
  2. Install the AWS Command Line Interface (AWS CLI). CodeArtifact is supported in these CLI versions:
    1. 18.83 or later: install the AWS CLI version 1
    2. 0.54 or later: install the AWS CLI version 2
  3. Create a CodeArtifact repository.
  4. Add required IAM permissions for CodeArtifact.

Creating your npm package

You can create your npm package in three easy steps: set up the project, create your npm script for authenticating with CodeArtifact, and publish the package.

Setting up your project

Create a directory for your new npm package. We name this directory my-package because it serves as the name of the package. We use an npm scope for this package, where @myorg represents the scope all of our organization’s packages are published under. This helps us distinguish our internal private package from external packages. See the following code:

npm init [email protected] -y

{
  "name": "@myorg/my-package",
  "version": "1.0.0",
  "description": "A sample private scoped npm package",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

The package.json file specifies that the main file of the package is called index.js. Next, we create that file and add our package function to it:

module.exports.helloWorld = function() {
  console.log('Hello world!');
}

Creating an npm script

To create your npm script, complete the following steps:

  1. On the CodeArtifact console, choose the repository you created as part of the prerequisites.

If you haven’t created a repository, create one before proceeding.

CodeArtifact repository details console

  1. Select your CodeArtifact repository and choose Details to view the additional details for your repository.

We use two items from this page:

  • Repository name (my-repo)
  • Domain (my-domain)
  1. Create a script named co:login in our package.json. The package.json contains the following code:
{
  "name": "@myorg/my-package",
  "version": "1.0.0",
  "description": "A sample private scoped npm package",
  "main": "index.js",
  "scripts": {
    "co:login": "aws codeartifact login --tool npm --repository my-repo --domain my-domain",
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

Running this script updates your npm configuration to use your CodeArtifact repository and sets your authentication token, which expires after 12 hours.

  1. To test our new script, enter the following command:

npm run co:login

The following code is the output:

> aws codeartifact login --tool npm --repository my-repo --domain my-domain
Successfully configured npm to use AWS CodeArtifact repository https://my-domain-<ACCOUNT ID>.d.codeartifact.us-east-1.amazonaws.com/npm/my-repo/
Login expires in 12 hours at 2020-09-04 02:16:17-04:00
  1. Add a prepare script to our package.json to run our login command:
{
  "name": "@myorg/my-package",
  "version": "1.0.0",
  "description": "A sample private scoped npm package",
  "main": "index.js",
  "scripts": {
    "prepare": "npm run co:login",
    "co:login": "aws codeartifact login --tool npm --repository my-repo --domain my-domain",
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

This configures our project to automatically authenticate and generate an access token anytime npm install or npm publish run on the project.

If you see an error containing Invalid choice, valid choices are:, you need to update the AWS CLI according to the versions listed in the perquisites of this post.

Publishing your package

To publish our new package for the first time, run npm publish.

The following screenshot shows the output.

Terminal showing npm publish output

If we navigate to our CodeArtifact repository on the CodeArtifact console, we now see our new private npm package ready to be downloaded.

CodeArtifact console showing published npm package

Installing your private npm package

To install your private npm package, you first set up the project and add the CodeArtifact configs. After you install your package, it’s ready to use.

Setting up your project

Create a directory for a new application and name it my-app. This is a sample project to download our private npm package published in the previous step. You can apply this pattern to all repositories you intend on installing your organization’s npm packages in.

npm init -y

{
  "name": "my-app",
  "version": "1.0.0",
  "description": "A sample application consuming a private scoped npm package",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

Adding CodeArtifact configs

Copy the npm scripts prepare and co:login created earlier to your new project:

{
  "name": "my-app",
  "version": "1.0.0",
  "description": "A sample application consuming a private scoped npm package",
  "main": "index.js",
  "scripts": {
    "prepare": "npm run co:login",
    "co:login": "aws codeartifact login --tool npm --repository my-repo --domain my-domain",
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

Installing your new private npm package

Enter the following command:

npm install @myorg/my-package

Your package.json should now list @myorg/my-package in your dependencies:

{
  "name": "my-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "prepare": "npm run co:login",
    "co:login": "aws codeartifact login --tool npm --repository my-repo --domain my-domain",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "dependencies": {
    "@myorg/my-package": "^1.0.0"
  }
}

Using your new npm package

In our my-app application, create a file named index.js to run code from our package containing the following:

const { helloWorld } = require('@myorg/my-package');

helloWorld();

Run node index.js in your terminal to see the console print the message from our @myorg/my-package helloWorld function.

Cleaning Up

If you created a CodeArtifact repository for the purposes of this post, use one of the following methods to delete the repository:

Remove the changes made to your user profile’s npm configuration by running npm config delete registry, this will remove the CodeArtifact repository from being set as your default npm registry.

Conclusion

In this post, you successfully published a private scoped npm package stored in CodeArtifact, which you can reuse across multiple teams and projects within your organization. You can use npm scripts to streamline the authentication process and apply this pattern to save time.

About the Author

Ryan Sonshine

Ryan Sonshine is a Cloud Application Architect at Amazon Web Services. He works with customers to drive digital transformations while helping them architect, automate, and re-engineer solutions to fully leverage the AWS Cloud.

 

 

Integrating Jenkins with AWS CodeArtifact to publish and consume Python artifacts

Post Syndicated from Matt Ulinski original https://aws.amazon.com/blogs/devops/using-jenkins-with-codeartifact/

Python packages are used to share and reuse code across projects. Centralized artifact storage allows sharing versioned artifacts across an organization. This post explains how you can set up two Jenkins projects. The first project builds the Python package and publishes it to AWS CodeArtifact using twine (Python utility for publishing packages), and the second project consumes the package using pip and deploys an application to AWS Fargate.

Solution overview

The following diagram illustrates this architecture.

Architecture Diagram

 

The solution consists of two GitHub repositories and two Jenkins projects. The first repository contains the source code of a Python package. Jenkins builds this package and publishes it to a CodeArtifact repository.

The second repository contains the source code of a Python Flask application that has a dependency on the package produced by the first repository. Jenkins builds a Docker image containing the application and its dependencies, pushes the image to an Amazon Elastic Container Registry (Amazon ECR) registry, and deploys it to AWS Fargate using AWS CloudFormation.

Prerequisites

For this walkthrough, you should have the following prerequisites:

To create a new Jenkins server that includes the required dependencies, complete the following steps:

  1. Launch a CloudFormation stack with the following link:
    Launch CloudFormation stack
  2. Choose Next.
  3. Enter the name for your stack.
  4. Select the Amazon Elastic Compute Cloud (Amazon EC2) instance type for your Jenkins server.
  5. Select the subnet and corresponding VPC.
  6. Choose Next.
  7. Scroll down to the bottom of the page and choose Next.
  8. Review the stack configuration and choose Create stack.

AWS CloudFormation creates the following resources:

  • JenkinsInstance – Amazon EC2 instance that Jenkins and its dependencies is installed on
  • JenkinsWaitCondition – CloudFormation wait condition that waits for Jenkins to be fully installed before finishing the deployment
  • JenkinsSecurityGroup – Security group attached to the EC2 instance that allows inbound traffic on port 8080

The stack takes a few minutes to deploy. When it’s fully deployed, you can find the URL and initial password for Jenkins on the Outputs tab of the stack.

CloudFormation outputs tab

Use the initial password to unlock the Jenkins installation, then follow the setup wizard to install the suggested plugins and create a new Jenkins user. After the user is created, the initial password no longer works.

On the Jenkins homepage, complete the following steps:

  1. Choose Manage Jenkins.
  2. Choose Manage Plugins.
  3. On the Available tab, search for “Docker Pipeline” and select it.
    Jenkins plugins available tab
  4. Choose Download now and install after restart.
  5. Select Restart Jenkins when installation is complete and no jobs are running.

Jenkins plugins installation complete

Jenkins is ready to use after it restarts. Log in with the user you created with the setup wizard.

Setting up a CodeArtifact repository

To get started, create a CodeArtifact repository to store the Python packages.

  1. On the CodeArtifact console, choose Create repository.
  2. For Repository name, enter a name (for this post, I use my-repository).
  3. For Public upstream repositories, choose pypi-store.
  4. Choose Next.
    AWS CodeArtifact repository wizard
  5. Choose This AWS account.
  6. If you already have a CodeArtifact domain, choose it from the drop-down menu. If you don’t already have a CodeArtifact domain, choose a name for your domain and the console creates it for you. For this post, I named my domain my-domain.
  7. Choose Next.
  8. Review the repository details and choose Create repository.
    CodeArtifact repository overview

You now have a CodeArtifact repository created, which you use to store and retrieve Python packages used by the application.

Configuring Jenkins: Creating an IAM user

  1. On the IAM console, choose User.
  2. Choose Add user.
  3. Enter a name for the user (for this post, I used the name Jenkins).
  4. Select Programmatic access as the access type.
  5. Choose Next: Permissions.
  6. Select Attach existing policies directly.
  7. Choose the following policies:
    1. AmazonEC2ContainerRegistryPowerUser – Allows Jenkins to push Docker images to ECR.
    2. AmazonECS_FullAccess – Allows Jenkins to deploy your application to AWS Fargate.
    3. AWSCloudFormationFullAccess – Allows Jenkins to update the CloudFormation stack.
    4. AWSCodeArtifactAdminAccessAllows Jenkins access to the CodeArtifact repository.
  8. Choose Next: Tags.
  9. Choose Next: Review.
  10. Review the configuration and choose Create user.
  11. Record the Access key ID and Secret access key; you need them to configure Jenkins.

Configuring Jenkins: Adding credentials

After you create your IAM user, you need to set up the credentials in Jenkins.

  1. Open Jenkins.
  2. From the left pane, choose Manage Jenkins
  3. Choose Manage Credentials.
  4. Hover over the (global) domain and expand the drop-down menu.
  5. Choose Add credentials.
    Jenkins credentials
  6. Enter the following credentials:
    1. Kind – User name with password.
    2. Scope – Global (Jenkins, nodes, items, all child items).
    3. Username – Enter the Access key ID for the Jenkins IAM user.
    4. Password – Enter the Secret access key for the Jenkins IAM user.
    5. ID – Name for the credentials (for this post, I used AWS).
  7. Choose OK.

You use the credentials to make API calls to AWS as part of the builds.

Publishing a Python package

To publish your Python package, complete the following steps:

  1. Create a new GitHub repo to store the source of the sample package.
  2. Clone the sample GitHub repo onto your local machine.
  3. Navigate to the package_src directory.
  4. Place its contents in your GitHub repo.
    Package repository contents

When your GitHub repo is populated with the sample package, you can create the first Jenkins project.

  1. On the Jenkins homepage, choose New Item.
  2. Enter a name for the project; for example, producer.
  3. Choose Freestyle project.
  4. Choose OK.
    Jenkins new project wizard
  5. In the Source Code Management section, choose Git.
  6. Enter the HTTP clone URL of your GitHub repo into the Repository URL
  7. To make sure that the workspace is clean before each build, under Additional Behaviors, choose Add and select Clean before checkout.
    Jenkins source code managnment
  8. To have builds start automatically when a change occurs in the repository, under Build Triggers, select Poll SCM and enter * * * * * in the Schedule
    Jenkins build triggers
  9. In the Build Environment section, select Use secret text(s) or file(s).
  10. Choose Add and choose Username and password (separated).
  11. Enter the following information:
    1. UsernameAWS_ACCESS_KEY_ID
    2. PasswordAWS_SECRET_ACCESS_KEY
    3. Credentials – Select Specific Credentials and from the drop-down menu and choose the previously created credentials.
      Jenkins credential binding
  12. In the Build section, choose Add build step.
  13. Choose Execute shell.
  14. Enter the following command and replace my-domain, my-repository, and my-region with the name of your CodeArtifact domain, repository, and Region:
    python3 setup.py sdist bdist_wheel
    aws codeartifact login --tool twine --domain my-domain --repository my-repository --region my-region
    python3 -m twine upload dist/* --repository codeartifact

    These commands do the following:

    • Build the Python package
    • Run the aws codeartifact login AWS Command Line Interface (AWS CLI) command, which retrieves the access token for CodeArtifact and configures the twine client
    • Use twine to publish the Python package to CodeArtifact
  15. Choose Save.
  16. Start a new build by choosing Build Now in the left pane.After a build starts, it shows in the Build History on the left pane. To view the build’s details, choose the build’s ID number.
    Jenkins project builds
  17. To view the results of the run commands, from the build details page, choose Console Output.
  18. To see that the package has been successfully published, check the CodeArtifact repository on the console.
    CodeArtifact console showing package

When a change is pushed to the repo, Jenkins will start a new build and attempt to publish the package. CodeArtifact will prevent publishing duplicates of the same package version, failing the Jenkins build.

If you want to publish a new version of the package, you will need to increment the version number.

The sample package uses semantic versioning (major.minor.maintenance), to change the version number modify the version='1.0.0' value in the setup.py file. You can do this manually before pushing any changes to the repo, or automatically as part of the build process by using the python-semantic-release package, or a similar solution.

Consuming a package and deploying an application

After you have a package published, you can use it in an application.

  1. Create a new GitHub repo for this application.
  2. Populate it with the contents of the application_src directory from the sample repo.
    Sample application repository

The version of the sample package used by the application is defined in the requirements.txt file. If you have published a new version of the package and want the application to use it modify the fantastic-ascii==1.0.0 value in this file.

After the repository created, you need to deploy the CloudFormation template application.yml. The template creates the following resources:

  • ECRRepository – Amazon ECR repository to store your Docker image.
  • ClusterAmazon Elastic Container Service (Amazon ECS) cluster that contains the service of your application.
  • TaskDefinition – ECS task definition that defines how your Docker image is deployed.
  • ExecutionRole – IAM role that Amazon ECS uses to pull the Docker image.
  • TaskRole – IAM role provided to the ECS task.
  • ContainerSecurityGroup – Security group that allows outbound traffic to ports 8080 and 80.
  • Service – Amazon ECS service that launches and manages your Docker containers.
  • TargetGroup – Target group used by the Load Balancer to send traffic to Docker containers.
  • Listener – Load Balancer Listener that listens for incoming traffic on port 80.
  • LoadBalancer – Load Balancer that sends traffic to the ECS task.
  1. Choose the following link to create the application’s CloudFormation stack:
    Launch CloudFormation stack
  2. Choose Next.
  3. Enter the following parameters:
    1. Stack name – Name for the CloudFormation stack. For this post, I use the name Consumer.
    2. Container Name – Name for your application (for this post, I use application).
    3. Image Tag – Leave this field blank. Jenkins populates it when you deploy the application.
    4. VPC – Choose a VPC in your account that contains two public subnets.
    5. SubnetA – Choose a public subnet from the previously chosen VPC.
    6. SubnetB – Choose a public subnet from the previously chosen VPC.
  4. Choose Next.
  5. Scroll down to the bottom of the page and choose Next.
  6. Review the configuration of the stack.
  7. Acknowledge the IAM resources warning to allow CloudFormation to create the TaskRole IAM role.
  8. Choose Create Stack.

After the stack is created, the Outputs tab contains information you can use to configure the Jenkins project.

Application stack outputs tab

To access the sample application, choose the ApplicationUrl link. Because the application has not yet been deployed, you receive an error message.

You can now create the second Jenkins project, which uses a configured through a Jenkinsfile stored in the source repository. The Jenkinsfile defines the steps that the build takes to build and deploy a Docker image containing your application.

The Jenkinsfile included in the sample instructs Jenkins to perform these steps:

  1. Get the authorization token for CodeArtifact:
    withCredentials([usernamePassword(
        credentialsId: CREDENTIALS_ID,
        passwordVariable: 'AWS_SECRET_ACCESS_KEY',
        usernameVariable: 'AWS_ACCESS_KEY_ID'
    )]) {
        authToken = sh(
                returnStdout: true,
                script: 'aws codeartifact get-authorization-token \
                --domain $AWS_CA_DOMAIN \
                --query authorizationToken \
                --output text \
                --duration-seconds 900'
        ).trim()
    }

  2. Start a Docker build and pass the authorization token as an argument to the build:
    sh ("""
        set +x
        docker build -t $CONTAINER_NAME:$BUILD_NUMBER \
        --build-arg CODEARTIFACT_TOKEN='$authToken' \
        --build-arg DOMAIN=$AWS_CA_DOMAIN-$AWS_ACCOUNT_ID \
        --build-arg REGION=$AWS_REGION \
        --build-arg REPO=$AWS_CA_REPO .
    """)

  3. Inside of Docker, the passed argument is used to configure pip to use CodeArtifact:
    RUN pip config set global.index-url "https://aws:[email protected]$DOMAIN.d.codeartifact.$REGION.amazonaws.com/pypi/$REPO/simple/"
    RUN pip install -r requirements.txt

  4. Test the image by starting a container and performing a simple GET request.
  5. Log in to the Amazon ECR repository and push the Docker image.
  6. Update the CloudFormation template and start a deployment of the application.

Look at the Jenkinsfile and Dockerfile in your repository to review the exact commands being used, then take the following steps to setup the second Jenkins projects:

  1. Change the variables defined in the environment section at the top of the Jenkinsfile:
    environment {
        AWS_ACCOUNT_ID = 'Your AWS Account ID'
        AWS_REGION = 'Region you used for this project'
        AWS_CA_DOMAIN = 'Name of your CodeArtifact domain'
        AWS_CA_REPO = 'Name of your CodeArtifact repository'
        AWS_STACK_NAME = 'Name of the CloudFormation stack'
        CONTAINER_NAME = 'Container name provided to CloudFormation'
        CREDENTIALS_ID = 'Jenkins credentials ID
    }
  2. Commit the changes to the GitHub repo.
  3. To create a new Jenkins project, on the Jenkins homepage, choose New Item.
  4. Enter a name for the project, for example, Consumer.
  5. Choose Pipeline.
  6. Choose OK.
    Jenkins pipeline wizard
  7. To have a new build start automatically when a change is detected in the repository, under Build Triggers, select Poll SCM and enter * * * * * in the Schedule field.
    Jenkins source polling configuration
  8. In the Pipeline section, choose Pipeline script from SCM from the Definition drop-down menu.
  9. Choose Git for the SCM
  10. Enter the HTTP clone URL of your GitHub repo into the Repository URL
  11. To make sure that your workspace is clean before each build, under Additional Behaviors, choose Add and select Clean before checkout.
    Jenkins source configuration
  12. Choose Save.

The Jenkins project is now ready. To start a new job, choose Build Now from the navigation pane. You see a visualization of the pipeline as it moves through the various stages, gathering the dependencies and deploying your application.

Jenkins application pipeline visualization

When the Deploy to ECS stage of the pipeline is complete, you can choose ApplicationUrl on the Outputs tab of the CloudFormation stack. You see a simple webpage that uses the Python package to display the current time.

Deployed application displaying in browser

Cleaning up

To avoid incurring future charges, delete the resources created in this post.

To empty the Amazon ECR repository:

  1. Open the application’s CloudFormation stack.
  2. On the Resources tab, choose the link next to the ECRRepository
  3. Select the check-box next to each of the images in the repository.
  4. Choose Delete.
  5. Confirm the deletion.

To delete the CloudFormation stacks:

  1. On the AWS CloudFormation console, select the application stack you deployed earlier.
  2. Choose Delete.
  3. Confirm the deletion.

If you created a Jenkins as part of this post, select the Jenkins stack and delete it.

To delete the CodeArtifact repository:

  1. On the CodeArtifact console, navigate to the repository you created.
  2. Choose Delete.
  3. Confirm the deletion.

If you’re not using the CodeArtifact domain for other repositories, you should follow the previous steps to delete the pypi-store repository, because it contains the public packages that were used by the application, then delete the CodeArtifact domain:

  1. On the CodeArtifact console, navigate to the domain you created.
  2. Choose Delete.
  3. Confirm the deletion.

Conclusion

In this post I showed how you can use Jenkins to publish and consume a Python package with Jenkins and CodeArtifact. I walked you through creating two Jenkins projects, a Jenkins freestyle project that built a package and published it to CodeArtifact, and a Jenkins pipeline project that built a Docker image that used the package in an application that was deployed to AWS Fargate.

About the author

Matt Ulinski is a Cloud Support Engineer with Amazon Web Services.

 

 

Software Package Management with AWS CodeArtifact

Post Syndicated from Steve Roberts original https://aws.amazon.com/blogs/aws/software-package-management-with-aws-codeartifact/

Software artifact repositories and their associated package managers are an essential component of development. Downloading and referencing pre-built libraries of software with a package manager, at the point in time the libraries are needed, simplifies both development and build processes. A variety of package repositories can be used, for example Maven Central, npm public registry, and PyPi (Python Package Index), among others. Working with a multitude of artifact repositories can present some challenges to organizations that want to carefully control both versions of, and access to, the software dependencies of their applications. Any changes to dependencies need to be controlled, to try and prevent undetected and exploitable vulnerabilities creeping into the organization’s applications. By using a centralized repository, it becomes easier for organizations to manage access control and version changes, and gives teams confidence that when updating package versions, the new versions have been approved for use by their IT leaders. Larger organizations may turn to traditional artifact repository software to solve these challenges, but these products can introduce additional challenges around installation, configuration, maintenance, and scaling. For smaller organizations, the price and maintenance effort of traditional artifact repository software may be prohibitive.

Generally available today, AWS CodeArtifact is a fully managed artifact repository service for developers and organizations to help securely store and share the software packages used in their development, build, and deployment processes. Today, CodeArtifact can be used with popular build tools and package managers such as Maven and Gradle (for Java), npm and yarn (for Javascript), and pip and twine (for Python), with more to come. As new packages are ingested, or published to your repositories, CodeArtifact automatically scales, and as a fully managed service, CodeArtifact requires no infrastructure installation or maintenance on your part. Additionally, CodeArtifact is a polyglot artifact repository, meaning it can store artifact packages of any supported type. For example, a single CodeArtifact repository could be configured to store packages from Maven, npm and Python repositories side by side in one location.

CodeArtifact repositories are organized into a domain. We recommend that you use a single domain for your organization, and then add repositories to it. For example you might choose to use different repositories for different teams. To publish packages into your repositories, or ingest packages from external repositories, you simply use the package manager tools your developers are used to. Let’s take a look at the process of getting started.

Getting started with CodeArtifact
To get started with CodeArtifact, I first need to create a domain for my organization, which will aggregate my repositories. Domains are used to perform the actual storage of packages and metadata, even though I consume them from a repository. This has the advantage that a single package asset, for example a given npm package, would be stored only once per domain no matter how many repositories it may appear to be in. From the CodeArtifact console, I can select Domains from the left-hand navigation panel, or instead create a domain as part of creating my first repository, which I’ll do here by clicking Create repository.

First, I give my repository a name and optional description, and I then have the option to connect my repository to several upstream repositories. When requests are made for packages not present in my repository, CodeArtifact will pull the respective packages from these upstream repositories for me, and cache them into my CodeArtifact repository. Note that a CodeArtifact repository can also act as an upstream for other CodeArtifact repositories. For the example here, I’m going to pull packages from the npm public registry and PyPi. CodeArtifact will refer to the repositories it creates on my behalf to manage these external connections as npm-store and pypi-store.

Clicking Next, I then select, or create, a domain which I do by choosing the account that will own the domain and then giving the domain a name. Note that CodeArtifact encrypts all assets and metadata in a domain using a single AWS Key Management Service (KMS) key. Here, I’m going to use a key that will be created for me by the service, but I can elect to use my own.

Clicking Next takes me to the final step to review my settings, and I can confirm the package flow from my selected upstream repositories is as I expect. Clicking Create repository completes the process, and in this case creates the domain, my repository, and two additional repositories representing the upstreams.

After using this simple setup process, my domain and its initial repository, configured to pull upstream from npm and PyPi, are now ready to hold software artifact packages, and I could also add additional repositories if needed. However my next step for this example is to configure the package managers for my upstream repositories, npm and pip, with access to the CodeArtifact repository, as follows.

Configuring package managers
The steps to configure various package managers can be found in the documentation, but conveniently the console also gives me the instructions I need when I select my repository. I’m going to start with npm, and I can access the instructions by first selecting my npm-pypi-example-repository and clicking View connection instructions.

In the resulting dialog I select the package manager I want to configure and I am shown the relevant instructions. I have the choice of using the AWS Command Line Interface (CLI) to manage the whole process (for npm, pip, and twine), or I can use a CLI command to get the token and then run npm commands to attach the token to the repository reference.

Regardless of the package manager, or the set of instructions I follow, the commands simply attach an authorization token, which is valid for 12 hours, to the package manager configuration for the repository. So that I don’t forget to refresh the token, I have taken the approach of adding the relevant command to my startup profile so that my token is automatically refreshed at the start of each day.

Following the same guidance, I similarly configure pip, again using the AWS CLI approach:

C:\> aws codeartifact login --tool pip --repository npm-pypi-example-repository --domain my-example-domain --domain-owner ACCOUNT_ID
Writing to C:\Users\steve\AppData\Roaming\pip\pip.ini
Successfully logged in to codeartifact for pypi

That’s it! I’m now ready to start using the single repository for dependencies in my Node.js and Python applications. Any dependency I add which is not already in the repository will be fetched from the designated upstream repositories and added to my CodeArtifact repository.

Let’s try some simple tests to close out the post. First, after changing to an empty directory, I execute a simple npm install command, in this case to install the AWS Cloud Development Kit.

npm install -g aws-cdk

Selecting the repository in the CodeArtifact console, I can see that the packages for the AWS Cloud Development Kit, and its dependencies, have now been downloaded from the upstream npm public registry repository, and added to my repository.

I mentioned earlier that CodeArtifact repositories are polyglot, and able to store packages of any supported type. Let’s now add a Python package, in this case Pillow, a popular image manipulation library.

> pip3 install Pillow
Looking in indexes: https://aws:****@my-example-domain-123456789012.d.codeartifact.us-west-2.amazonaws.com/pypi/npm-pypi-example-repository/simple/
Collecting Pillow
  Downloading https://my-example-domain-123456789012.d.codeartifact.us-west-2.amazonaws.com/pypi/npm-pypi-example-repository/simple/pillow/7.1.2/Pillow-7.1.2-cp38-cp38-win_amd64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 819 kB/s
Installing collected packages: Pillow
Successfully installed Pillow-7.1.2

In the console, I can see the Python package sitting alongside the npm packages I added earlier.

Although I’ve used the console to verify my actions, I could equally well use CLI commands. For example, to list the repository packages I could have run the following command:

aws codeartifact list-packages --domain my-example-domain --repository npm-pypi-example-repository

As you might expect, additional commands are available to help with work with domains, repositories, and the packages they contain.

Availability
AWS CodeArtifact is now generally available in the Frankfurt, Ireland, Mumbai, N.Virginia, Ohio, Oregon, Singapore, Sweden, Sydney, and Tokyo regions. Tune in on June 12th at noon (PST) to Twitch.tv/aws or LinkedIn Live, where we will be showing how you can get started with CodeArtifact.

— Steve