Post Syndicated from Daniel Li original https://aws.amazon.com/blogs/architecture/field-notes-automating-migration-requests-for-reserved-instances-and-savings-plans-in-closed-accounts/

Enterprise AWS customers are often managing many accounts under a payer account, and sometimes accounts are closed before Reserved Instances (RI) or Savings Plans (SP) are fully used. Manually tracking account closures and requesting RI and SP migration from the closed accounts can become complex and error prone.

This blog post describes a solution for automating the process of requesting migration of unused RI and SP from closed accounts that are linked to a payer account. The solution helps reduce manual work and loss of unused RI and SP due to human error.

The solution automatically detects newly closed accounts that are linked to a payer account. If the closed accounts have unused RI and SP attached, it creates support cases with the required information for RI and SP migration. It can optionally send email notifications with the support case IDs. After a support case is created, the AWS Support team starts a workflow for processing the request, and updates the support case when the request is completed.

Solution Overview

The following diagram shows the overview of the solution.

Walkthrough

1. An Amazon CloudWatch Events time-based rule will create an event with the configured interval, such as one hour.
2. The event triggers an AWS Lambda function (refresh.js), which will retrieve the current status of all the linked accounts for the payer account from AWS Organizations. It saves the account status as an item in an Amazon DynamoDB table.
3. The DynamoDB table has Streams enabled and it triggers another Lambda function (process.js) when there are updates in the table. The stream view type is set to NEW_AND_OLD_IMAGES.
4.  The Lambda function performs the following steps:
   • Compares the differences between the new and the old images in the stream record. If an account’s status is SUSPENDED in the new image and is ACTIVE in the old image, the account is newly closed.
   • Retrieves the RI and SP utilization information from AWS Cost Explorer for the closed account.
   • If there are unused RI and SP in the account, it creates support cases to request RI and SP migration and/or send email notifications through Amazon SNS.

Next, let’s take a deeper look at the Lambda functions, which were written in JavaScript. The code for the Lambda functions can be downloaded from this repository.

Lambda function: refresh.js

This Lambda function is triggered by a CloudWatch Events rule at the predefined interval, for example, one hour. It retrieves the account statuses of all the linked accounts under a payer account from AWS Organizations. Then it saves the account statuses to a DynamoDB table.

It calls the AWS Organizations ListAccounts API to get the current status of all the linked accounts for the given payer account. The following sample code is for the ListAccounts API call.

const AWS = require('aws-sdk')
// Environment variable
const organizationsEndpoint = process.env.ORGANIZATIONS_ENDPOINT; 
const orgEndpoint = new AWS.Endpoint(organizationsEndpoint);
// Get region from the endpoint hostname, e.g. organizations.us-east-1.amazonaws.com
const orgRegion = organizationsEndpoint.split(".")[1];

const organizations = new AWS.Organizations({
    endpoint: orgEndpoint,
    region: orgRegion 
});

// Call the ListAccounts API to retrieve the status for all the linked accounts
function refreshAccounts() {
    console.log("Enter refreshAccounts");

    var newData = {Accounts:[]};
    
    var handleError = function(response) {
        console.log(response.err, response.err.stack); // an error occurred
        throw response.err;
    };
    
    var params = {};
    organizations.listAccounts(params).on('success', function handlePage(response) {
        console.log("ListAccounts response: \n" + JSON.stringify(response.data));    
        var account;
            
        // extract each account's Id and Status 
        for (account of response.data.Accounts) {
            var accountStatus = {Id: account.Id, Status: account.Status};
            newData.Accounts.push(accountStatus);
        }
    
        // handle response pagination
        if (response.hasNextPage()) {
            response.nextPage().on('success', handlePage).on('error', handleError)
            .send();
        } else {
            console.log("Account Status: \n" + JSON.stringify(newData)); 
            // insert code here to save the data to DynamoDB
        }
    }).on('error', handleError).send();
}

The following sample output is from the ListAccounts API call.

{
  Accounts: [
    {
      Id: '123456789000',
      Arn: 'arn:aws:organizations::123456789000:account/o-x3ub1dr53f/123456789000',
      Email: '[email protected]',
      Name: 'Payer',
      Status: 'ACTIVE',
      JoinedMethod: 'INVITED',
      JoinedTimestamp: 2020-01-09T06:49:36.035Z
    },
    {
      Id: '987654321000',
      Arn: 'arn:aws:organizations::123456789000:account/o-x3ub1dr53f/987654321000',
      Email: '[email protected]',
      Name: 'Linked Account 1',
      Status: 'SUSPENDED',
      JoinedMethod: 'INVITED',
      JoinedTimestamp: 2020-08-11T23:46:18.560Z
    }
  ]
}

From the API output, the Lambda function creates a smaller JSON with only the account ID and status information.

{
    "Accounts": [
        {
            "Id": "123456789000",
            "Status": "ACTIVE"
        },
        {
            "Id": "987654321000",
            "Status": "SUSPENDED"
        }
    ]
}

Next, it calls the DynamoDB UpdateItem API to save the current account status data as an item in the DynamoDB table. The API is configured with conditional update.

If the DynamoDB table already has an item with the same key, the item is not updated unless there is a different value. When the table is updated, DynamoDB Streams triggers a call to the next Lambda function. Below is the sample code for the UpdateItem API call.

const tableName = process.env.TABLE_NAME; // environment variable
const dynamodb = new AWS.DynamoDB();

// Save the account status JSON to the DynamoDB table 
// if it is different than the existing data in the table
function saveDataToDynamoDB(data) {
    console.log("Enter saveDataToDynamoDB");

    var params = {
        ExpressionAttributeNames: {
            "#ACC": "Accounts"
        }, 
        ExpressionAttributeValues: {
            ":t": {
                S: data
            } 
        }, 
        Key: {
            "Organization": {
                S: "default"
            }
        }, 
        ConditionExpression: "#ACC <> :t",
        ReturnValues: "ALL_NEW", 
        TableName: tableName, 
        UpdateExpression: "SET #ACC = :t"
    };
    
    dynamodb.updateItem(params, function(err, data) {
        console.log("DynamoDB callback...");  
        if (err) {
            console.log(err, err.stack); // an error occurred
        
            if (err.code != "ConditionalCheckFailedException") {
                throw err;
            }
        } else {
            console.log(data);           // successful response
        }
    });
}

Lambda function: process.js

This Lambda function processes DynamoDB stream records and compares the old and new item data in the stream record. If any of the linked accounts changed statuses to SUSPENDED from ACTIVE, it calls the Cost Explorer APIs – GetReservationUtilization API and GetSavingsPlansUtilizationDetails API– to check if the newly closed accounts have unused RI and SP attached. The following sample code is for the GetReservationUtilization API call.

const costexplorerEndpoint = process.env.COST_EXPLORER_ENDPOINT; // environment variable
const ceEndpoint = new AWS.Endpoint(costexplorerEndpoint);
// Get region from the endpoint hostname, e.g. ce.us-east-1.amazonaws.com
const ceRegion = costexplorerEndpoint.split(".")[1];

const costexplorer = new AWS.CostExplorer({
    endpoint:ceEndpoint,
    region:ceRegion
});

// Check if the source account has unused RI. If yes, send email notification 
// or create support case based on ACTION_TYPE
// Params
//   startDate: String with 'YYYY-MM-DD' format
//   endDate: String with 'YYYY-MM-DD' format
//   sourceAccountId: the AWS account to migrate RI from
//   destinationAccountId: the AWS account to migrate RI to
function processReservationUtilization(startDate,endDate,
            sourceAccountId,destinationAccountId) {
  
  console.log("Enter processReservationUtilization");
  
  // array to hold all the unused RI lease IDs from the source account
  var riLeaseIds = []; 
  
  const now = new Date();

  var params = {
    TimePeriod: {
      End: endDate, 
      Start: startDate
    },
    Filter: { 
      "Dimensions": {
          "Key": "LINKED_ACCOUNT",
          "Values": [
              sourceAccountId
          ]
      }
    },
    GroupBy: [
      {
        Key: 'SUBSCRIPTION_ID',
        Type: "DIMENSION"
      }
    ]
  };
    
  var handleError = function(response) {
      console.log(response.err, response.err.stack); // an error occurred
      throw response.err;
  };

  costexplorer.getReservationUtilization(params)
    .on('success', function handlePage(response) {
    console.log("GetReservationUtilization response \n" + response.data); 
    
    var utilizationsByTime;
    var group;
    
    // Find unused RI in the results
    for (utilizationsByTime of response.data.UtilizationsByTime) {
      for (group of utilizationsByTime.Groups) {
        const endDateTime = new Date(group.Attributes.endDateTime);
        if (endDateTime.getTime() > now.getTime()) {
          riLeaseIds.push(group.Attributes.leaseId);
        }        
      }
    }
    
    // Handle response pagination
    if (response.hasNextPage()) {
        response.nextPage().on('success', handlePage)
          .on('error', handleError).send();
    } else if (riLeaseIds.length > 0) {
      var requestBody = riRequestBody + "\nSource Account: " + sourceAccountId 
        + "\nDestination Account: " + destinationAccountId + "\nSavings Plans Arns: " + JSON.stringify(riLeaseIds);
      // Insert code here to create support case and/or send message to SNS
    }      
  }).on('error', handleError).send();
}

The following sample code is for the GetSavingsPlansUtilizationDetails API call.

// Check if the source account has unused SP. If yes, send email notification 
// or create support case based on ACTION_TYPE
// Params
//   startDate: String with 'YYYY-MM-DD' format
//   endDate: String with 'YYYY-MM-DD' format
//   sourceAccountId: the AWS account to migrate SP from
//   destinationAccountId: the AWS account to migrate SP to
function processSavingsPlansUtilization(startDate,endDate,
        sourceAccountId,destinationAccountId) {
  
  console.log("Enter processSavingsPlansUtilization");
  
  //array to hold all the unused SP ARNs from the source account
  var savingsPlansArns = []; 
  
  const now = new Date();  
  
  var params = {
    TimePeriod: { /* required */
      End: endDate, 
      Start: startDate    
    },
    Filter: { 
      "Dimensions": {
          "Key": "LINKED_ACCOUNT",
          "Values": [
            sourceAccountId
          ]
      }
    }
  };

  var handleError = function(response) {
      console.log(response.err, response.err.stack); // an error occurred
      throw response.err;
  };

  costexplorer.getSavingsPlansUtilizationDetails(params).on('success', function handlePage(response) {
    console.log("getSavingsPlansUtilizationDetails response \n" + response.data);  

    var utilizationDetail;
    
    // Find unused SP in the results
    for (utilizationDetail of response.data.SavingsPlansUtilizationDetails) {
      const endDateTime = new Date(utilizationDetail.Attributes.EndDateTime);
      if (endDateTime.getTime() > now.getTime()) {
        savingsPlansArns.push(utilizationDetail.SavingsPlanArn);
      }
    }

    // handle response pagination
    if (response.hasNextPage()) {
        response.nextPage().on('success', handlePage).on('error', handleError).send();
    } else if (savingsPlansArns.length > 0) {
      var requestBody = spRequestBody + "\nSource Account: " + sourceAccountId + "\nDestination Account: " + destinationAccountId + "\nSavings Plans Arns: " + JSON.stringify(savingsPlansArns);
      // insert code here to create support case and/or send message to SNS
    }
  }).on('error', handleError).send();
}

For those accounts that require RI or SP migration, it calls the AWS Support CreateCase API to create support cases. It can either/or send email notifications based on the value of the environment variable ACTION_TYPE.

• EMAIL_ONLY: Send the RI and SP details to an SNS topic for email notification.
• SUPPORT_CASE_ONLY: Create support cases to request RI and SP migration.
• SUPPORT_CASE_AND_EMAIL: Create support cases to request RI and SP migration and send the request details to an SNS topic for email notification. The support case IDs are included in the email subject.

The following sample code is for the CreateCase API call.

const supportEndpoint = process.env.SUPPORT_ENDPOINT; // environment variable
const supEndpoint = new AWS.Endpoint(supportEndpoint);
// get region from the endpoint hostname, e.g. support.us-east-1.amazonaws.com
const supRegion = supportEndpoint.split(".")[1];

const support = new AWS.Support({
    endpoint:supEndpoint,
    region:supRegion
});

// Create an AWS Support Case
// Params:
//   emailAddress: email address for the support case
//   caseSubject: the subject of the support case
//   caseBody: the body of the support case
function createSupportCase(emailAddresses, caseSubject, caseBody, notify) {
  
  console.log("Enter createSupportCase");
  
  var params = {
    communicationBody: caseBody, /* required */
    subject: caseSubject, /* required */
    categoryCode: 'reserved-instances',
    ccEmailAddresses: [
      emailAddresses
    ],
    issueType: 'customer-service',
    serviceCode: 'billing',
    severityCode: 'low'
  };
  
  support.createCase(params, function(err, data) {
    console.log("Support callback...");  
    
    if (err) {
      console.log(err, err.stack); // an error occurred
      throw err;
    } else {
      console.log(data);           // successful response
      if (notify) {
        // insert code here to publish the message to a SNS topic
      }
    }
  });
}

The following sample support case is content from the Lambda function:

Subject: 

Savings Plans Migration

Body: 

Please migrate the following Savings Plans from the source account to the destination account.

Source Account: 987654321000

Destination Account: 123456789000

Savings Plans Arns: [“arn:aws:savingsplans::987654321000:savingsplan/bb242919-11ba-429c-b1dc-101010101010”]

AWS Support case guidelines

The AWS RI Operations Team has provided the following guidelines for RI and SP migration support cases.

1) If a closed account is still linked to a payer account, create one support case from the payer account. Otherwise, create one support case from each account – source and destination.

2) When creating a support case, use a user ID or a role with RI and SP purchase IAM permissions.

3) In the support case body, provide RI lease IDs or SP ARNs, source account ID, destination account ID, and an explicit request to migrate the RI and SP. If there are many RI lease IDs or SP ARNs, provide the data in CSV format.

Note: Do not use PDF or images.

4) Set severity code to low, service code to billing, category code to reserved-instances, and issue type to customer-service.

Deploy the stack

Download the template for deploying the solution used in this post.

Use this template to launch the solution and all associated components. The default configuration deploys AWS Lambda functions, a DynamoDB table, an Amazon CloudWatch Events rule, an SNS topic, and AWS Identity and Access Management (IAM) roles necessary to set up the solution in your account, but you can customize the template to meet your specific needs.

Before you launch the solution, review the architecture, configuration, network security, and other considerations. Follow the step-by-step instructions in this section to configure and deploy the solution into your account. You need to have authority to launch resources in the payer account before launching the stack.

Note: You are responsible for the cost of the AWS services used while running this solution. For more details, refer to the pricing web page for each AWS service used in this solution.

Time to deploy: Approximately five minutes.

1.      Sign in to the AWS Management Console and use the following button to launch the AWS CloudFormation stack. Optionally, you can download the template as a starting point for your own implementation. Note: the CloudFormation stack needs to be created in the payer account.

2.      The template launches in the US East (N. Virginia) Region by default. You can choose to launch it in other regions.

3.      On the Create stack page, verify that the correct template URL is in the Amazon S3 URL text box and choose Next.

4.      On the Specify stack details page, assign a name to your solution stack.

5.      Under Parameters, review the parameters for this solution template and modify them as necessary. This solution uses the following default values.

Recommended documentation for reviewing these parameters:

• AWS Cost Explorer Service
• AWS Organizations endpoints and quotas
• About the AWS Support API

6.      Choose Next.

7.      On the Configure stack options page, choose Next.

8.      On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

9.      Choose Create stack to deploy the stack.

You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately five minutes.

Clean Up

To clean up the stack, select the stack in the AWS CloudFormation console and click the Delete button. In the popup window, choose Delete stack.

Conclusion

This post described a solution for automating the process of detecting account closures and requesting RI and SP migration. The solution will help enterprise customers reduce manual work and avoid losing unused RI and SP due to human errors.

If you have any comments or questions, please don’t hesitate to leave them in the comments section.

Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.