Post Syndicated from Aditya Ranjan original https://aws.amazon.com/blogs/devops/deploying-and-managing-application-configurations-using-aws-appconfig/

The management of configurations across multiple environments and tenants poses a significant challenge in modern software development. Organizations must balance maintaining distinct settings for various environments while accommodating the unique needs of different tenants in multi-tenant architectures. This complexity is compounded by requirements for consistency, version control, security, and efficient troubleshooting.

AWS AppConfig offers a powerful solution to these challenges. AWS AppConfig centrally stores, manages, and deploys application configurations. It streamlines pushing changes without frequent code deployments. The service also enables automatic rollbacks, providing a safety net for configuration changes.

When integrated with a CI/CD pipeline, such as GitLab, AWS AppConfig becomes part of a streamlined, automated system for configuration management. This combination addresses the complexities of multi-environment and multi-tenant deployments, ensuring consistent, version-controlled, and secure configuration management across the entire application ecosystem.

Solution and Scenario Overview

The GitLab CI/CD pipeline in this blog focuses on the way application configurations are managed and deployed using AWS AppConfig. By automating the entire process from configuration updates to multi-environment deployment, it offers a streamlined approach to configuration management.

In this configuration management setup, we’re dealing with a multi-environment, multi-tenant application structure that leverages AWS AppConfig for configuration deployment.

It describes a multi-tenant configuration setup where each tenant has dedicated environments (dev and qa). Real-world examples of what these could represent:

• Development (dev): Where developers test new features and changes
• Quality Assurance (qa): Where quality assurance teams validate changes before production

The system supports multiple tenants (tenant1, tenant2), each with their own isolated environments. In real-world applications, these tenants could represent:

• Different customers:
  • A retail company (tenant1)
  • A healthcare provider (tenant2)
• Different business units:
  • North America division (tenant1)
  • EMEA division (tenant2)

Each tenant maintains separate configurations for their dev and qa environments, with three example configuration files:

1. AllowList.yml
2. FeatureFlags.yml
3. ThrottlingLimits.yml

The ‘template’ directory provides base configuration files that can be inherited and customized by each tenant’s environment-specific configurations. This hierarchical structure ensures that tenants can maintain their unique configurations while adhering to a standardized template format.

Here’s an example of how the template YAML files might look:

1. AllowList.yml

# AllowList.yml

# Network Access Controls
ip_allowlists:
  internal_networks:
    - "10.0.0.0/8"     # Internal corporate network
    - "172.16.0.0/12"  # VPC network range
    - "192.168.1.0/24" # Development network

# Domain Allowlist
domain_allowlist:
  api_consumers:
    - "api.partner1.com"
    - "services.partner2.com"
    - "*.trusted-client.com"

2. FeatureFlags.yml

# FeatureFlags.yml

features:
  new_search:
    enabled: true
    rollout_percentage: 76
    description: "Enhanced search functionality"
    
  ai_recommendations:
    enabled: true

  chat_support:
    enabled: false
    description: "In-app chat support"

3. ThrottlingLimits.yml

#ThrottlingLimits.yml
api_limits:
  global:
    requests_per_second: 100
    concurrent_requests: 50
    max_retry_attempts: 3

service_specific:
  user_service:
      requests_per_second: 80
      burst_limit: 100

These templates serve as the starting point for all environment and tenant-specific configurations.

The folder structure reflects a sophisticated approach to organizing configurations across different environments and tenants.

├── template
│   ├── AllowList.yml
│   ├── FeatureFlags.yml
│   └── ThrottlingLimits.yml

└── tenants
├── tenant1
│   ├── dev
│   │   ├── AllowList.yml
│   │   ├── FeatureFlags.yml
│   │   └── ThrottlingLimits.yml
│   └── qa
│       ├── AllowList.yml
│       ├── FeatureFlags.yml
│       └── ThrottlingLimits.yml
└── tenant2
├── dev
│   ├── AllowList.yml
│   ├── FeatureFlags.yml
│   └── ThrottlingLimits.yml
└── qa
├── AllowList.yml
├── FeatureFlags.yml
└── ThrottlingLimits.yml

1. template: Houses the base configuration templates
2. tenants: Contains tenant-specific configurations

The ‘tenants’ directory follows a hierarchical structure where each tenant (tenant1, tenant2) has their own directory. Within each tenant’s directory, there are ‘dev’ and ‘qa’ environment subdirectories. Each environment directory contains three configuration files: AllowList.yml, FeatureFlags.yml, and ThrottlingLimits.yml. These files represent different aspects of the application’s configuration and can override the base templates found in the ‘template’ directory. This structure allows for environment-specific configurations while maintaining a clear separation between tenants and their respective environments.

This structure allows for:

1. Standardization through templates: The base templates in the ‘template’ directory ensure consistency across all tenants, providing default configurations that can be selectively overridden by tenant-specific needs.
2. Tenant-specific customization: Each tenant can maintain unique configurations in their dev and qa environments while inheriting from the base templates. This allows for customization without losing standardization benefits.
3. Environment isolation: Clear separation between dev and qa environments within each tenant’s directory ensures that configuration changes in one environment don’t affect other
4. Version control of configurations: By storing configurations in a Git repository, changes can be tracked, reviewed, and rolled back if necessary.
5. AWS AppConfig integration:

1. 1. Each tenant gets their own Application in AWS AppConfig
   2. Configuration profiles map to different configuration types (AllowList, FeatureFlags, ThrottlingLimits)
   3. Separate environments (dev/qa) within each tenant’s application

The GitLab CI/CD pipeline we’re setting up will need to:

1. Generate environment and tenant-specific configurations based on these templates
2. Update the corresponding applications and configuration profiles in AWS AppConfig
3. Deploy the appropriate configurations to each tenant and environment

Pre-Requisites

1. Configuring GitLab CI/CD with AWS: Please refer Deploy to AWS from GitLab CI/CD
2. Setting up GitLab Runners: Please refer Deploy and Manage Gitlab Runners on Amazon EC2 if you want to use Gitlab runners on EC2 or you can refer Install GitLab Runner and Configure GitLab Runner guides
3. Configure Runner in .gitlab-ci.yml:
   • Use tags to specify which runner should execute your jobs:

job_name:
tags:
- aws-runner  # Tag of your specific runner

Setting Up the Directory Structure:

1. First, create the base directory structure using these commands:

# Create directory structure and files
mkdir -p template tenants/{tenant1,tenant2}/{dev,qa}

2. Create all required YAML files:

for file in AllowList.yml FeatureFlags.yml ThrottlingLimits.yml;do
  touch template/$file
  touch tenants/tenant{1,2}/{dev,qa}/$file
done

3. Populate the template files:

Copy the content of each YAML file (AllowList.yml, FeatureFlags.yml, ThrottlingLimits.yml) shown above into the corresponding files in the template directory.

4. For tenant-specific configurations:

Start by copying the template files to each tenant's environment directory

5. Verify the folder structure.

Setting Up the GitLab CI/CD Pipeline

Code for the GitLab pipeline is in this repo.

This phase begins with gaining a clear understanding of the pipeline’s structure and flow, which forms the foundation for all subsequent steps.

Configuring .gitlab-ci.yml

1. 1. Creating the .gitlab-ci.yml file in your repository root
   2. Defining the base image for the pipeline (e.g., alpine:latest)
   3. Setting up pipeline stages: update-app-config, deploy-app-config
   4. Configuring global variables and default settings
      • Locate these sections in the .gitlab-ci.yml file below and Replace them with your AWS account details

variables:
  AWS_CREDS_TARGET_ROLE: arn:aws:iam::<aws_account_ID>:role/GitLab
  AWS_DEFAULT_REGION: <aws_region>

• • •  Make sure to replace these variables in both stages (update-app-config and deploy-app-config) of the pipeline. The AWS role should have appropriate permissions to interact with AWS AppConfig service

Here’s the complete .gitlab-ci.yml file:

stages:
  - update-app-config
  - deploy-app-config

update-app-config:
  stage: update-app-config
  image:
    name: amazon/aws-cli:latest
    entrypoint:
      - '/usr/bin/env'
  script:
    - |
      # Get list of all tenant
      TENANTS=$(find tenants -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
      
      for TENANT in $TENANTS; do
        echo "Processing tenant: $TENANT"
        
        # Create/Get Application for tenant
        APP_ID=$(aws appconfig list-applications --query "Items[?Name=='$TENANT'].Id" --output text)
        if [ -z "$APP_ID" ]; then
          echo "Creating application for tenant '$TENANT'..."
          APP_ID=$(aws appconfig create-application --name $TENANT --query Id --output text)
        fi
        
        # Process each configuration type
        for CONFIG_TYPE in AllowList FeatureFlags ThrottlingLimits; do
          echo "Processing config type: $CONFIG_TYPE"
          
          # Create/Get Configuration Profile
          PROFILE_ID=$(aws appconfig list-configuration-profiles --application-id "$APP_ID" --query "Items[?Name=='$CONFIG_TYPE'].Id" --output text)
          if [ -z "$PROFILE_ID" ]; then
            echo "Creating configuration profile '$CONFIG_TYPE' for tenant '$TENANT'..."
            PROFILE_ID=$(aws appconfig create-configuration-profile --application-id "$APP_ID" --name "$CONFIG_TYPE" --description "Configuration profile for $CONFIG_TYPE" --location-uri hosted --query Id --output text)
          fi
          
          # Process each environment
          for ENV in dev qa; do
            echo "Processing environment: $ENV"
            
            # Priority: Use tenant-specific config if it exists, otherwise use template
            if [ -f "tenants/$TENANT/$ENV/$CONFIG_TYPE.yml" ]; then
              echo "Using tenant-specific configuration for $ENV"
              CONFIG_CONTENT=$(cat "tenants/$TENANT/$ENV/$CONFIG_TYPE.yml" | base64)
            else
              echo "Using template configuration for $ENV"
              CONFIG_CONTENT=$(cat "template/$CONFIG_TYPE.yml" | base64)
            fi
            
            echo "Creating new version for $CONFIG_TYPE configuration in $ENV..."
            aws appconfig create-hosted-configuration-version \
              --application-id "$APP_ID" \
              --configuration-profile-id "$PROFILE_ID" \
              --content "$CONFIG_CONTENT" \
              --content-type "application/json" \
              configuration_version_output
          done
        done
      done
  variables:
    AWS_CREDS_TARGET_ROLE: arn:aws:iam::<aws_account_ID>:role/GitLab 
    AWS_DEFAULT_REGION: <aws_region>

deploy-app-config:
  stage: deploy-app-config
  image: 
    name: amazon/aws-cli:latest
    entrypoint: 
      - '/usr/bin/env'
  script:
    - yum install -y jq
    - |
      TENANTS=$(find tenants -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
      
      for TENANT in $TENANTS; do
        echo "Processing tenant: $TENANT"
        APP_ID=$(aws appconfig list-applications --query "Items[?Name=='$TENANT'].Id" --output text)
        
        # Process each environment
        for ENV in dev qa; do
          echo "Processing environment: $ENV"
          
          # Create/Get Environment
          ENV_ID=$(aws appconfig list-environments --application-id "$APP_ID" --query "Items[?Name=='$ENV'].Id" --output text)
          if [ -z "$ENV_ID" ]; then
            echo "Creating environment '$ENV' for tenant '$TENANT'..."
            ENV_ID=$(aws appconfig create-environment --application-id "$APP_ID" --name "$ENV" --description "Environment for $ENV" --query Id --output text)
          fi
          
          # Process each configuration types
          for CONFIG_TYPE in AllowList FeatureFlags ThrottlingLimits; do
            echo "Processing $CONFIG_TYPE for $TENANT/$ENV"
            
            PROFILE_ID=$(aws appconfig list-configuration-profiles --application-id "$APP_ID" --query "Items[?Name=='$CONFIG_TYPE'].Id" --output text)

            echo " Profile ID $PROFILE_ID "
            # Get latest version for this specific profile
            LATEST_VERSION=$(aws appconfig list-hosted-configuration-versions \
              --application-id "$APP_ID" \
              --configuration-profile-id "$PROFILE_ID" \
              --query "Items[0].VersionNumber" \
              --output text)
            
            # Get current deployment for this specific profile
            CURRENT_DEPLOYMENT=$(aws appconfig list-deployments \
            --application-id "$APP_ID" \
            --environment-id "$ENV_ID" \
            --query "Items[?ConfigurationName=='$CONFIG_TYPE'].ConfigurationVersion | [0]" \
            --output text)


            echo "Current deployment $CURRENT_DEPLOYMENT"
              
            CURRENT_VERSION=$(aws appconfig list-deployments \
            --application-id "$APP_ID" \
            --environment-id "$ENV_ID" \
            --query "Items[?ConfigurationName=='$CONFIG_TYPE'].ConfigurationVersion | [0]" \
            --output text)
            
            echo "Latest Version: $LATEST_VERSION"
            echo "Current Version: $CURRENT_VERSION"
            
            if [[ "$CURRENT_DEPLOYMENT" == "None" ]] || [[ "$LATEST_VERSION" != "$CURRENT_VERSION" ]]; then
              echo "Starting deployment for $TENANT/$ENV/$CONFIG_TYPE..."
              DEPLOYMENT_RESPONSE=$(aws appconfig start-deployment \
                --application-id "$APP_ID" \
                --environment-id "$ENV_ID" \
                --deployment-strategy-id Linear50PercentEvery30Seconds \
                --configuration-profile-id "$PROFILE_ID" \
                --configuration-version "$LATEST_VERSION")
              
              DEPLOYMENT_ID=$(echo $DEPLOYMENT_RESPONSE | jq -r '.DeploymentNumber')
              
              # Monitor deployment
              max_attempts=10
              attempt=1
              while [ $attempt -le $max_attempts ]; do
                echo "Checking deployment status (attempt $attempt of $max_attempts)..."
                status=$(aws appconfig get-deployment \
                  --application-id "$APP_ID" \
                  --environment-id "$ENV_ID" \
                  --deployment-number "$DEPLOYMENT_ID" \
                  --query "State" \
                  --output text)
                
                if [ "$status" = "COMPLETE" ]; then
                  echo "Deployment completed successfully!"
                  break
                elif [ "$status" = "FAILED" ] || [ "$status" = "ROLLED_BACK" ]; then
                  echo "Deployment failed or was rolled back!"
                  exit 1
                fi
                
                if [ $attempt -eq $max_attempts ]; then
                  echo "Deployment timed out after $max_attempts attempts"
                  exit 1
                fi
                
                attempt=$((attempt + 1))
                sleep 30
              done
            else
              echo "No changes detected for $TENANT/$ENV/$CONFIG_TYPE (Current: $CURRENT_VERSION, Latest: $LATEST_VERSION). Skipping deployment..."
            fi
          done
        done
      done
  dependencies:
    - update-app-config
  variables:
    AWS_CREDS_TARGET_ROLE: arn:aws:iam::<aws_account_ID>:role/GitLab 
    AWS_DEFAULT_REGION: <aws_region>

Implementing Pipeline Stages

1. Update-App-Config Stage:

• Creates/Updates AWS AppConfig Applications:
  • Creates one application per tenant (tenant1, tenant2)
  • Uses tenant ID as application name
  • Retrieves existing application if already present
• Manages Configuration Profiles:
  • Creates three profiles per tenant application (AllowList, FeatureFlags, ThrottlingLimits)
  • Each profile represents a distinct configuration type
  • Handles profile creation if not already existing
• Creates Hosted Configuration Versions:
  • Processes changes from both template and tenant directories
  • Prioritizes tenant-specific configurations over templates
  • Creates new versions only for modified configurations
  • Uploads properly encoded configurations to AWS AppConfig

2. Deploy-App-Config Stage:

1. • Environment Deployment:
     • Manages dev and qa environments per tenant
     • Creates environments if not existing
     • Uses staged deployment strategy
   • Tenant Configuration Process:
     • Deploys per tenant and configuration type
     • Checks current deployed version against latest version
     • Only deploys if either of the follows is true:
       • No existing deployment is found
       • Latest Hosted Configuration version differs from currently deployed version
     • Maintains tenant-specific settings and version history
     • Provides clear deployment status messages, including cases where deployment is skipped

• • Deployment Management:
    • Executes AWS AppConfig deployments
    • Monitors deployment status
    • Handles failures and rollbacks
    • Times out after 10 retries

Executing the Pipeline

1. Initiation:
   • Pipeline triggered by changes pushed to the repository

2. Update-App-Config Stage:
   • Creates or updates applications and configuration profiles
   • Generates new versions of hosted configurations

3. Deploy-App-Config Stage:
   • Iterates through each environment tenant and their environments
   • Checks current deployment status for each environment and tenant
   • Initiates new deployments only for changed configurations
   • Implements specified AWS AppConfig deployment strategy

Note: Deployment Strategy used in this example is a fast one used for testing (Linear50PercentEvery30Seconds) but for real production workloads, the reader should use the slower, AWS-recommended Linear20PercentEvery6Minutes strategy. More details here

This structured execution process ensures efficient and consistent deployment of configuration changes across the entire application ecosystem, maintaining synchronization between GitLab and AWS AppConfig.

Cleaning up

To clean up all AWS AppConfig resources created by this solution, you can use the following cleanup script. Create a file named delete_appconfig_resources.sh with this content:

#!/bin/bash

# List all applications
APPS=$(aws appconfig list-applications --query 'Items[*].Id' --output text)

for APP_ID in $APPS
do
  echo "Processing application $APP_ID"
  
  # List and delete all environments for this application
  ENVS=$(aws appconfig list-environments --application-id $APP_ID --query 'Items[*].Id' --output text)
  for ENV_ID in $ENVS
  do
    echo "  Deleting environment $ENV_ID"
    aws appconfig delete-environment --application-id $APP_ID --environment-id $ENV_ID
  done

  # List and delete all configuration profiles for this application
  PROFILES=$(aws appconfig list-configuration-profiles --application-id $APP_ID --query 'Items[*].Id' --output text)
  for PROFILE_ID in $PROFILES
  do
    echo "  Deleting configuration profile $PROFILE_ID"
    
    # Delete all hosted configuration versions for this profile
    VERSIONS=$(aws appconfig list-hosted-configuration-versions --application-id $APP_ID --configuration-profile-id $PROFILE_ID --query 'Items[*].VersionNumber' --output text)
    for VERSION in $VERSIONS
    do
      echo "    Deleting hosted configuration version $VERSION"
      aws appconfig delete-hosted-configuration-version --application-id $APP_ID --configuration-profile-id $PROFILE_ID --version-number $VERSION
    done

    # Delete the configuration profile
    aws appconfig delete-configuration-profile --application-id $APP_ID --configuration-profile-id $PROFILE_ID
  done

  # Delete the application
  echo "  Deleting application $APP_ID"
  aws appconfig delete-application --application-id $APP_ID
done

echo "All AppConfig resources have been deleted."

The script is a comprehensive cleanup utility for AWS AppConfig resources.

To execute this script, you need to have the AWS CLI installed and configured with appropriate credentials that have permissions to delete AppConfig resources. Make the script delete_appconfig_resources.sh  executable by running the command:

chmod +x cleanup_appconfig.sh.

Before running the script, ensure that you’re in the correct AWS account and region, as this script will delete ALL AppConfig resources in the configured account and region. To execute the script, simply run it from your terminal:  ./ delete_appconfig_resources.sh

It’s crucial to note that this script performs irreversible deletions. Use it with extreme caution, preferably in non-production environments or when you’re absolutely certain you want to remove all AppConfig resources.

Conclusion

This blog post has explored the powerful synergy between GitLab CI/CD and AWS AppConfig for managing application configurations in multi-tenant environments. We’ve demonstrated how this integration automates and streamlines the process of updating, versioning, and deploying configuration changes, offering benefits such as scalability, version control, and the balance between consistency and flexibility. By adopting this approach, development teams can significantly reduce manual errors, save time, and focus more on building features, ultimately leading to faster development cycles and more reliable applications in our increasingly complex and distributed computing landscape.

Key resources for further reading:

• AWS AppConfig overview video
• Unlock Faster Releases with AWS AppConfig blog
• AWS AppConfig Workshop

About the Author