Exploring Fn::ForEach and Fn::FindInMap enhancements in AWS CloudFormation

Post Syndicated from Dan Blanco original https://aws.amazon.com/blogs/devops/exploring-fnforeach-and-fnfindinmap-enhancements-in-aws-cloudformation/

AWS CloudFormation, an Infrastructure as Code (IaC) service that lets you model, provision, and manage AWS and third-party resources, recently released a new language transform that enhances the core CloudFormation language. Today, we’ll be covering two more enhancements we’ve added since our initial release: Fn::FindInMap enhancements and a new looping function – Fn::ForEach.

These new language extensions are the result of open discussions with the larger CloudFormation community via our Request For Comments (RFC) proposals for new language features at our Language Discussion GitHub repository. We want to collaborate with the community to better align features and incorporate early feedback into the development cycle to meet the community’s needs. We invite you to participate in new RFCs to help shape the future of the CloudFormation language.

In this post, I’ll dive deeper into the new enhancements for Fn::FindInMap as well as explore the new Fn::ForEach looping mechanism and provide some examples.

Trending
Ferdinand Porsche – A Man ahead of His Time

Prerequisites

To use these new language features, you must add AWS::LanguageExtensions to the transform section of your template.

---
AWSTemplateFormatVersion: 2010-09-09
Transform: 'AWS::LanguageExtensions'

If you have a list of transforms, then we recommend having AWS managed transforms at the end, and AWS::LanguageExtensions must be listed before AWS::Serverless.

---
AWSTemplateFormatVersion: 2010-09-09
Transform: 
 - 'AWS::LanguageExtensions'
 - 'AWS::Serverless-2016-10-31'

This transform will cover all of the existing and future language extensions.

FindInMap enhancements

We have updated the language extension transform for CloudFormation to support Fn::FindInMap enhancements, that extend the existing functionality of the Fn::FindInMap intrinsic function so that now you can:

  • use an optional, default value in Fn::FindInMap parameters, if a given key in a Mappings section is not found, and
  • use a number of additional intrinsic functions in the parameters of Fn::FindInMap; for more information, see Supported functions.

Let’s see an example use case where Fn::FindInMap enhancements can help you simplify the business logic of your template, and make it more readable and easier to maintain. Let’s suppose you create a CloudFormation template that describes an Amazon Elastic Compute Cloud (Amazon EC2) instance, and you need to use smaller EC2 instance types for pre-production environments, and a larger EC2 instance type for production for cost savings. In this example, you choose a t2.micro instance type for the dev environment, t2.medium for the qa environment, and t2.large for the prod environment, that you start to describe as follows:

---
AWSTemplateFormatVersion: "2010-09-09"

Description: 'Sample template that describes usage for `Fn::FindInMap` enhancements'

Parameters:
 Environment:
 Description: Lifecycle environment.
 Type: String
 AllowedValues:
 - sandbox
 - dev
 - qa
 - prod
 Default: dev

 LatestAmiId:
 Description: Region-specific image to use.
 Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>  Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

Mappings:
 LifecycleEnvToInstanceType:
 dev:
 InstanceType: t2.micro
 qa:
 InstanceType: t2.medium
 prod:
 InstanceType: t2.large

You described instance types for each of your 3 lifecycle environments in the Mappings section, and you engineered your template to read environment names as input data from Environment in the Parameters section. Looking closer at Environment, you define another allowed value: sandbox, that in this example is an environment for developers to use for prototype testing only: you choose not to include this environment in the mapping you created, with the intent to do the same for any other non-formal environment (for example, a contributor’s personal environment). Next, powered by the new enhancements toFn::FindInMap, you assign a default value for environment names that are different than dev, qa, and prod; this way, the only change you’ll need to make in this context is a new value(s) to AllowedValues in Environment. You describe this business logic in your template, to which you add the sample code shown next:

Transform: AWS::LanguageExtensions

Resources: Ec2Instance: Type: AWS::EC2::Instance Properties: ImageId: !Ref 'LatestAmiId' InstanceType: !FindInMap - LifecycleEnvToInstanceType - !Ref 'Environment' - InstanceType - DefaultValue: t2.micro Tags: - Key: test Value: test

In the snippet above, you have declared the AWS::LanguageExtensions transform, and described your configuration for an EC2 instance in the Resources section. For InstanceType, you chose to use Fn::FindInMap enhancements, and pass DefaultValue as an additional parameter with t2.micro as its value. When the user uses this template to create a stack, and chooses sandbox for Environment, the !Ref 'Environment' reference to the value for Environment will evaluate to sandbox, which is not present in the mapping you created: in this case, t2.micro will be used as a value for InstanceType.

These new enhancements also allow you to use more intrinsic functions inside of Fn::FindInMap. Let’s say you have received requirements to use -env as a suffix to environment names. You choose to make a minimal set of changes to your template, and start with the Parameters section as follows:

Parameters:
  Environment:
    Description: Lifecycle environment.
    Type: String
    AllowedValues:
      - sandbox-env       - dev-env       - qa-env       - prod-env     Default: dev-env

Next, instead of modifying all of your keys the Mappings section, you choose to only change the second parameter to Fn::FindInMap enhancements as follows: first, you use the Fn::Split intrinsic function to split the user-selected environment value string (for example, dev-env) into a list of values using the ‘-‘ character as a delimiter, and next you use the Fn::Select intrinsic function to choose the first element (that is, 0) of that list:

Resources:
  Ec2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref 'LatestAmiId'
      InstanceType: !FindInMap
        - LifecycleEnvToInstanceType         - !Select           - 0           - !Split             - '-'             - !Ref 'Environment'         - InstanceType         - DefaultValue: t2.micro       Tags:
        - Key: test           Value: test

With the updated code above, if the user selects the dev-env value for Environment, Fn::FindInMap enhancements will use dev as the second parameter when looking up values in the Mappings section.

Fn::ForEach intrinsic function

Another enhancement to the language extensions is the addition of native looping inside of CloudFormation with Fn::ForEach. Imagine you have a situation where you need three EC2 instances that look exactly the same. Currently, you would have to copy and paste each instance as a separate resource with CloudFormation:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Resources:
 FirstInstance:
 Type: AWS::EC2::Instance
 Properties: 
      # ..removed for brevity..

 SecondInstance:
 Type: AWS::EC2::Instance
 Properties: 
      # ..removed for brevity..

 ThirdInstance:
 Type: AWS::EC2::Instance
 Properties: 
      # ..removed for brevity..

If you encounter the need to update one property (the AMI ID, for example), you will have to update all three separately. While this is trivially easy for our example, templates that extend into the hundreds of resources quickly becomes difficult to maintain.

With the Fn::ForEach language extension, we’re able to group all of these items together into an easy-to-manage snippet:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions
 Resources:
 Fn::ForEach::Instances:
    - InstanceLogicalId
    - [FirstInstance, SecondInstance, ThirdInstance]
    - ${InstanceLogicalId}:
 Type: AWS::EC2::Instance
 Properties: 
          # ..removed for brevity..

This results in the following output YAML, which is identical to our previous example:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Resources:
 FirstInstance:
 Type: AWS::EC2::Instance
 Properties: 
      # ..removed for brevity..

 SecondInstance:
 Type: AWS::EC2::Instance
 Properties: 
      # ..removed for brevity..

 ThirdInstance:
 Type: AWS::EC2::Instance
 Properties: 
      # ..removed for brevity..

To break down the syntax, the Fn::ForEach function requires:

  • A Logical ID for the looping function directly following the Fn::ForEach call. In our case, we named it Instances
  • The variable name we’ll be referencing in our snippet below
  • The collection of strings we’ll be iterating over. You can write these inline, or pass them as parameters or mappings.
  • A section of the template we’ll be iterating over using the variable name above. This is standard CloudFormation JSON/YAML.

These must be listed in an array immediately following the Fn::ForEach intrinsic function and in this exact order.

We can use our key to reference values found elsewhere in the template. This adds additional flexibility when combined with the aforementioned FindInMap enhancements. Imagine a similar scenario where each instance needs a specific instance type, dictated by which instance it is and which environment we’re in. As described before, we would add our parameters and mappings to our template:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Parameters:
 Environment:
 Description: Lifecycle environment.
 Type: String
 AllowedValues:
 - sandbox
 - dev
 - qa
 - prod
 Default: dev

 LatestAmiId:
 Description: Region-specific image to use.
 Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>  Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

Mappings:
 dev:
 FirstInstance:
 InstanceType: t2.micro
 SecondInstance:
 InstanceType: t2.micro
 ThirdInstance:
 InstanceType: t2.micro
 qa:
 FirstInstance:
 InstanceType: t2.medium
 SecondInstance:
 InstanceType: t2.medium
 ThirdInstance:
 InstanceType: t2.large
 prod:
 FirstInstance:
 InstanceType: t2.large
 SecondInstance:
 InstanceType: t2.xlarge
 ThirdInstance:
 InstanceType: t2.2xlarge

Given this configuration, we have different environment values as Parameters and a Mapping section that details our sizing requirements for our instance. With this, we can then use our new Fn::ForEach functionality and FindInMap enhancements:

Resources:
  Fn::ForEach::Instances:
    - InstanceLogicalId     - [FirstInstance, SecondInstance, ThirdInstance]     - ${InstanceLogicalId}:         Type: AWS::EC2::Instance
        Properties: 
          ImageId: !Ref LatestAmiId
          InstanceType: !FindInMap
            - !Ref Environment             - !Ref InstanceLogicalId             - InstanceType             - DefaultValue: t2.micro

This results in the following output:

Resources:
  FirstInstance:
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestAmiId
      InstanceType: !FindInMap
        - !Ref Environment
        - FirstInstance
        - InstanceType
        - DefaultValue: t2.micro


  SecondInstance:
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestAmiId
      InstanceType: !FindInMap
        - !Ref Environment
        - SecondInstance
        - InstanceType
        - DefaultValue: t2.micro

  ThirdInstance:
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestAmiId
      InstanceType: !FindInMap
        - !Ref Environment
        - ThirdInstance
        - InstanceType
        - DefaultValue: t2.micro

This looping feature can be used to create more than just resources – say we want to reference outputs from these EC2 instances as we create them. We can modify our above template to add an Output section and iterate over it in the same way, as well as exporting the instance ID. We can even express more than one Output per iteration. We’ll also move the instances list to a parameter for increased clarity.

Parameters:
 InstancesToManage:
 Type: CommaDelimitedList
 Description: Instances to be managed
 Default: FirstInstance,SecondInstance,ThirdInstance
 Outputs:
 Fn::ForEach::InstanceOutputs:
    - InstanceLogicalId
    - !Ref InstancesToManage
    - "${InstanceLogicalId}Id": 
 Export: 
 Name: !Sub ${AWS::AccountId}-${InstanceLogicalId}Id
 Value: !Ref 
 Ref: InstanceLogicalId

      "${InstanceLogicalId}AvailabilityZone":
 Value:
 Fn::GetAtt: 
            - !Ref InstanceLogicalId
            - AvailabilityZone

This outputs to:

Outputs:
 FirstInstanceId: 
 Export: 
 Name: !Sub ${AWS::AccountId}-FirstInstanceId
 Value: !Ref FirstInstance
 FirstInstanceAvailabilityZone:
 Value:
 Fn::GetAtt:
        - FirstInstance
        - AvailabilityZone
 SecondInstanceId: 
 Export: 
 Name: !Sub ${AWS::AccountId}-SecondInstanceId
 Value: !Ref SecondInstance
 SecondInstanceAvailabilityZone:
 Value:
 Fn::GetAtt:
        - SecondInstance
        - AvailabilityZone
 ThirdInstanceId: 
 Export: 
 Name: !Sub ${AWS::AccountId}-ThirdInstanceId
 Value: !Ref ThirdInstance
 ThirdInstanceAvailabilityZone:
 Value:
 Fn::GetAtt:
        - ThirdInstance
        - AvailabilityZone

In this snippet, we iterated over our collection and created multiple outputs. For each output, we concatenated our key with some other string. In this case, both Id and AvailabilityZone were concatenated with the key to create a unique output name based on the stack name and the logical ID of the resource.

Finally, loops can be nested inside other loops. Combined with the ability to concatenate values and do lookups inside of the Mapping section, we’re able to significantly simplify complex CloudFormation templates. Imagine an example where we are tasked with creating a Virtual Private Cloud (VPC) with three private subnets and three public subnets. This is a common configuration our customers have and we can configure it simply with looping.

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Parameters:
 AvailabilityTypes:
 Type: CommaDelimitedList
 Description: Types of subnets availability - public, private, or both
 AllowedValues:
 - Public
 - Private
 Default: Public,Private

Mappings:
 SubnetOne:
 Public: 
 Cidr:  10.215.0.0/24 
 Private: 
 Cidr:  10.215.1.0/24 
 SubnetTwo:
 Public: 
 Cidr:  10.215.2.0/24
 Private: 
 Cidr:  10.215.3.0/24
 SubnetThree:
 Public: 
 Cidr:  10.215.4.0/24
 Private: 
 Cidr:  10.215.5.0/24

Resources:
 VPC:
 Type: AWS::EC2::VPC
 Properties:
 CidrBlock: 10.215.0.0/16
 EnableDnsSupport: true
 EnableDnsHostnames: true

 Fn::ForEach::Subnets:
 - SubnetIdentifier
 - - SubnetOne
 - SubnetTwo
 - SubnetThree
 - Fn::ForEach::SubnetAvailabilityType:
 - AvailabilityType
 - !Ref AvailabilityTypes
 - "${SubnetIdentifier}${AvailabilityType}":
 Type: AWS::EC2::Subnet
 Properties:
 VpcId: !Ref VPC
 CidrBlock: !FindInMap
 - !Ref SubnetIdentifier
 - !Ref AvailabilityType
 - Cidr

which outputs the following resource section:

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCidr
      EnableDnsSupport: true
      EnableDnsHostnames: true

  SubnetOnePublic:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetOne         - Public         - Cidr 
  SubnetOnePrivate:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetOne         - Private         - Cidr 
  SubnetTwoPublic:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetTwo         - Public         - Cidr 
  SubnetTwoPrivate:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetTwo         - Private         - Cidr 
  SubnetThreePublic:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetThree         - Public         - Cidr 
  SubnetThreePrivate:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetThree         - Private         - Cidr

Combining everything we’ve learned so far, this created six subnets total, three public, three private, and attached them to the respective VPC with a relevant CIDR block.

We’re excited to share this functionality with our community, and we invite you to share feedback on future enhancements to the looping functionality here. A few enhancements we’re discussing are:

  • Iterating over a key/value pair
  • Iterating over a list of lists
  • Support in other template sections
  • And more!

Please head over and let us know what you think!

Conclusion

In this post, we walked through the new CloudFormation additions to the language extensions transform, how to enable them in your templates, and how to engage in future language extensions via our open language discussion repository. Leave us your feedback at our Language Discussion GitHub repository to help shape the future of the CloudFormation language. We look forward to hearing from you!

 

About the Author:

Dan Blanco

Dan is a senior AWS Developer Advocate based in Atlanta for the AWS IaC team. When he’s not advocating for IaC tools, you can either find him in the kitchen whipping up something delicious or flying in the Georgia sky. Find him on twitter (@TheDanBlanco) or in the AWS CloudFormation Discord server