All posts by Bryan Liston

Creating an Enterprise Scheduler Using AWS Lambda and Tagging

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/creating-an-enterprise-scheduler-using-aws-lambda-and-tagging/

Co-authored by Felix Candelario and Benjamin F., AWS Solutions Architects

Many companies are looking to optimize their AWS usage and lower operational costs. While production environments are typically well-governed and reach optimal use through services such as Auto Scaling, sometimes development and testing need a different approach.

There are a number of ways to schedule resources to start or stop based on time of day (Auto Scaling Scheduled Scaling, for example); however, some companies are looking to hand over control of the schedule to the resource owner, without reassigning the burden of creating the scheduling infrastructure.

In this post, we discuss a proposed method for running an enterprise scheduler that queries running resources for a specific tag; the tag controls when resources should be switched on or off. Control over this tag can be handed to the resource owner, thereby providing a very simple method to govern the on/off schedule of a resource based on a weekly cycle.

Resource owners only need to describe the schedule they wish to enforce in human-friendly syntax and apply that in the form of a tag on their resources. The enterprise scheduler runs periodically using AWS Lambda, scans resources for the schedule tag, parses the schedule syntax, and then issue an On or Off command per the schedule that’s assigned to each resource individually.

Lambda function schedule

To enable the enterprise scheduler, a Lambda function should be added and run on a schedule. The function describes EC2 instances within the same region, and determines if there are any On or Off operations to be executed. We recommend configuring the function to run every 10 minutes. By default, the function determines On/Off actions from the last 12 minutes.

Controlling this behavior is done in two parts:

  • The Lambda function schedule
  • The max_delta parameter

The function schedule controls how often the function is invoked and begins to search for On/Off actions to perform. The max_delta parameter controls the time window in which to search for individual schedule tags on resources. For example, you may choose to run the function every 20 minutes, and then set max_delta to 22 minutes. It is necessary to keep the max_delta value slightly higher than the rate which the function is invoked so as not to miss any On/Off actions. We recommend setting it two minutes above the function invocation rate.

Required IAM permissions

The Lambda function requires permissions to query the resource tags using DescribeInstances, and then to act on them, with either a StartInstances or StopInstances API operation call.

Install the scheduler

The following procedures walk you through creating the function and validation.

Modify the code

  1. Copy the Enterprise Scheduler Lambda function code to your computer from the following code example:
from __future__ import print_function # Python 2/3 compatibility
import boto3
import datetime
import sys

def lambda_handler(event, context):
    schedule_tag = 'EntScheduler'
    max_delta = 12
    now = datetime.datetime.now()
    ec2 = boto3.resource('ec2')
    client = boto3.client('ec2')
    scheduled_instances = []
    processed_instances = []
    #filter for instances with the correct tag
    instances = ec2.instances.filter(Filters=[{'Name': 'tag-key', 'Values':[schedule_tag]}])
    #grab the scheduler string
    for instance in instances:
        for tag in instance.tags:
            if tag['Key'] == schedule_tag:
                scheduled_instances.append({'instance':instance, 'schedule':tag['Value']})

    def parse_schedule(instance_hold):
        day = now.strftime('%a').lower()
        current_time = datetime.datetime.strptime(now.strftime("%H%M"), "%H%M")
        instance_hold['disabled'] = False
        #parse the schedule string into seperate tokens
        tokenized_schedule = instance_hold['schedule'].split(';')
        #make sure the schedule string contains either 4 or 5 parameters.
        if len(tokenized_schedule) < 4:
            instance_hold['disabled'] = True
            sys.exit('Schedule string improperly formed. Fewer than 4 tokens specified.')
        if len(tokenized_schedule) > 6:
            instance_hold['disabled'] = True
            sys.exit('Schedule string improperly formed. Greater than 5 tokens specified.')
        #check to make sure today is the day to execute an on action
        if day in tokenized_schedule[0]:
            try:
                #check to make sure 24 hour string parses correctly
                scheduled_time_for_on = datetime.datetime.strptime(tokenized_schedule[1], "%H%M")
                #as long as not outside of the window of execution
                delta = scheduled_time_for_on - current_time
                margin = datetime.timedelta(minutes=max_delta)
                if(current_time - margin <= scheduled_time_for_on <= current_time):
                    instance_hold['on'] = True
                else:
                    instance_hold['on'] = False
            except Exception as e:
                print(e)
                instance_hold['disabled'] = True
                sys.exit('Time string for the on action improperly formed. Ensure in HHMM format.')
        else:
            instance_hold['on'] = False

        #check to make sure today is the day to execute an off action
        if day in tokenized_schedule[2]:
            try:
                #check to make sure 24 hour string parses correctly
                scheduled_time_for_off = datetime.datetime.strptime(tokenized_schedule[3], "%H%M")
                delta = scheduled_time_for_off - current_time
                margin = datetime.timedelta(minutes=max_delta)
                if(current_time - margin <= scheduled_time_for_off <= current_time):
                   instance_hold['off'] = True
                else:
                    instance_hold['off'] = False
            except Exception as e:
                print(e)
                instance_hold['disabled'] = True
                sys.exit('Time string for the on action improperly formed. Ensure in HHMM format.')
        else:
            instance_hold['off'] = False

        #check for disabled string
        if len(tokenized_schedule) > 4:
            if 'disable' in tokenized_schedule[4]:
                instance_hold['disabled'] = True
        return instance_hold

    for instance_hold in scheduled_instances:
        processed_instances.append(parse_schedule(instance_hold))

    for instance_hold in processed_instances:
        if(instance_hold['disabled']==False):
            if(instance_hold['off']==True and instance_hold['on']==True):
                print('Both on and off actions specified for this time window. Doing nothing.')
            if(instance_hold['off']==True and instance_hold['on']==False):
                print('Turning instance off: ' + instance_hold['instance'].id + ' ' + instance_hold['instance'].instance_type)
                client.stop_instances(InstanceIds=[instance_hold['instance'].id])
            if(instance_hold['off']==False and instance_hold['on']==True):
                print('Turning instance on: ' + instance_hold['instance'].id + ' ' + instance_hold['instance'].instance_type)
                client.start_instances(InstanceIds=[instance_hold['instance'].id])
            if(instance_hold['off']==False and instance_hold['on']==False):
                print('No action on instance: ' + instance_hold['instance'].id + ' ' + instance_hold['instance'].instance_type)
        else:
            print('Schedule disabled: ' + instance_hold['instance'].id + ' ' + instance_hold['instance'].instance_type)
  1. Edit the local copy of the code on your computer using a text editor, and make the following optional changes:
  2. Optional: Edit the _schedule_tag_ parameter if you will be using a custom tag for setting up resource schedules. By default, the value is: EntScheduler.
  3. Optional: Set _max_delta_ to a value that is two minutes higher than the rate of invocation that will be used. By default, the value is 12 mins.

  4. Save the local copy of the code with the changes made above and name it enterprise_scheduler.py.

Create the Lambda function

  1. Open the Lambda console and choose Create a Lambda function.
  2. Choose Next to skip a blueprint selection.
  3. Choose Next to skip creation of a trigger at this time.
  4. Enter a function name and note it for later use. For example: enterprise_scheduler_function.
  5. For Runtime, choose Python 2.7.
  6. For Code entry type, choose Edit code inline.
  7. Paste the function code from the local version of the scheduler that was saved in the previous section (enterprise_scheduler.py).
  8. For Handler, enter enterprise_scheduler.lambda_handler .
  9. For Role selection, choose Create a custom role.

A window pops up displaying the IAM form for creating a new IAM execution role for Lambda.

Add an IAM role

  1. In the IAM window, for Role name, enter ent_scheduler_execution_role (or similar text).
  2. Choose View Policy Document, Edit the policy, OK (to confirm that you’ve read the documentation).
  3. Replace the contents of the default policy with the following:
 {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1469047780000",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Sid": "Stmt1469047825000",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:*:*:*"
      ]
    }
  ]
}
  1. Choose Allow to save the policy and close the window.
  2. For Timeout, enter 5 minutes.
  3. Choose Next.
  4. Review the configuration and choose Create Function.

Add an event schedule

  1. Choose Triggers, Add trigger.
  2. Click on the box and choose CloudWatch Events – Schedule.
  3. For Schedule expression, enter the rate at which you would like the enterprise scheduler to be invoked. We recommend a value of "rate(10 minutes)".

At this point, a tagged Amazon EC2 instance in the region where the Lambda function is running is either turned on or off, based on its schedule.

Enabling the scheduler on a resource

Assigning a schedule to a resource using a tag requires following these guidelines:

  • The resource tag must match the tag named in the Lambda function. You can modify the schedule_tag parameter to choose a tag name such as YourCompanyName Scheduler.
  • The scheduler works on a weekly basis (7 days of the week), and you may specify up to 1 set hour for turning the resource On or Off. For each day that’s set in the On or Off section, the same hour is used.
  • The current time is interpreted by the function using UTC, therefor the schedule tag should use the UTC time for turning instances both On and Off.
  • Use the following scheduler syntax in the Lambda function:

Days-for-On;Hour-for-On;Days-for-Off;Hour-for-off;Optional-Disable;

  • The Days values should be noted with the first 3 letters of the day. Accepted values are: mon, tue, wed, thu, fri, sat, sun. Multiple days can be specified with comma separation and order is ignored.
  • The Hour value should be noted in 24H format HHMM.
  • Optional-Disable states that this tag will be ignored by the function, allowing the user to keep the configuration and have the scheduler skip over this resource temporarily. Note: The function can be configured to disregard Optional-Disable based on company policy.

Here are some examples of scheduler syntax:

  • mon,tue,wed,thu,fri,sat,sun;0830;mon,tue,wed,thu,fri,sat,sun;1700;
    Resource would be turned on at 8:30 am UTC daily and turned off at 5:00 pm UTC daily.

  • mon;0700;fri;1800;
    Resource would be turned on at 7:00 am on Mondays and turned off at 6:00 pm UTC on Fridays.

  • wed,thu,fri;2100;thu,fri,sat;0500;
    Resource would be turned on at 9:00 pm UTC on Wednesdays, Thursdays, and Fridays, and turned off at 5:00 am UTC on Thursdays, Fridays, and Saturdays.

  • wed,thu,fri;2100;thu,fri,sat;0500;disable;
    Resource would be left untouched and the schedule ignored due to the disable option.

Conclusion

This method of scheduling resource use is simple and can support several use cases, such as shutting down labs for evenings and weekends. Even this small optimization can result in significant savings when you consider that a typical workday is 8 out of 24 hours. Unlike on-premises resources, AWS resources can be managed by a quick API call.

If you have questions or suggestions, please comment below.

Create and Deploy a Chat Bot to AWS Lambda in Five Minutes

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/create-and-deploy-a-chat-bot-to-aws-lambda-in-five-minutes/

This is a guest post by Gojko Adzic, creator of ClaudiaJS

 

The new Claudia.JS Bot Builder project helps JavaScript developers to easily create chat-bots for Facebook, Telegram, Skype, and Slack, and deploy them to AWS Lambda and Amazon API Gateway in minutes.

The key idea behind this project is to remove all the boilerplate code and common infrastructure tasks, so you can focus on writing the really important part of the bot — your business workflows. Everything else is handled by the Claudia Bot Builder.

The Claudia Bot Builder library simplifies messaging workflows, automatically sets up the correct web hooks, and guides you through configuration steps, so you don’t have to research individual implementation protocols. It automatically converts the incoming messages from various platforms into a common format, so you can handle them easily. It also automatically packages the responses into the correct templates, so you do not have to worry about different message response formats. This means that you can write and deploy a single bot with just a few lines of code, and operate it on various bot platforms using AWS Lambda. Check out the two-minute video Create chat-bots easily using Claudia Bot Builder to see how easy it is to set up a bot on AWS using the new tool.

Here’s a simple example:

Prerequisites

The Claudia Bot Builder works with the Node.JS 4.3.2 AWS Lambda installation. It requires using the Claudia.JS deployment tool, which you can install using NPM:

npm install claudia -g

If you already have Claudia installed, make sure it’s up to date. The Claudia Bot Builder support requires version 1.4.0 or later.

Creating a simple text bot

First, create an empty folder, and a new NPM project inside it. Make sure to give it a descriptive name:

npm init

Then, add the claudia-bot-builder library as a project dependency:

npm install claudia-bot-builder -S

For this particular bot, generate some dynamic content using the huh excuse generator. Add that as a project dependency:

npm install huh -S

Now create the bot. Create a file called bot.js and paste the following content:

var botBuilder = require('claudia-bot-builder'),
    excuse = require('huh');

module.exports = botBuilder(function (request) {
  return 'Thanks for sending ' + request.text  + 
      '. Your message is very important to us, but ' + 
      excuse.get();
});

That’s pretty much it. You can now deploy the bot to AWS and configure it for Facebook Messenger, by using Claudia:

claudia create --region us-east-1 --api-module bot --configure-fb-bot

Now would be a good time to configure a new Facebook page and a messenger application, as explained in the Facebook Messenger Getting Started Guide. The bot installer prints the web hook URL and the verification token, which you can copy to your Facebook Messenger configuration page. You can then generate the page access token from Facebook. Copy that back to Claudia when asked, and you’re almost done.

In a few moments, your bot will be live, and you can talk to it from the page you created. That was easy, wasn’t it?

If you’d like other Facebook users to talk to it as well, submit it for application review from the Facebook App Developer page.

Deploying to other platforms

The Claudia Bot Builder can also help you set up this bot for all the other platforms. Just run claudia update and provide the additional configuration option:

  • For Slack slash commands, use –configure-slack-slash-command
  • For Skype, use –configure-skype-bot
  • For Telegram, use –configure-telegram-bot

More complex workflows

The example bot just responds with silly excuses so for homework, do something more interesting with it.

The request object passed into the message handling function contains the entire message in the text field, but it also has some other pieces of data for more complex work. The sender field identifies the user sending the message, so you can create threads of continuity and sessions. The type field contains the identifier of the bot endpoint that received the message (for example, skype or facebook) so you can respond differently to different bot systems. The originalRequest field contains the entire unparsed original message, so you can handle platform-specific requests and go beyond simple text.

For examples, check out:

  • Fact Bot, which looks up facts about topics on WikiData and creates Facebook Messenger menus.
  • Space Explorer Bot, A small FB Messenger chat bot using NASA API

Although it’s enough just to return a string value for simple cases, and the Bot Builder packages it correctly for individual bot engines, you can return a more complex object and get platform-specific features, for example, Facebook buttons. In that case, make sure to use the type field of the request to decide on additional features.

For asynchronous workflows, send back a Promise object, and resolve it with the response later. The convention is the same: if the promise gets resolved with a string, the Claudia Bot Builder automatically packages it into the correct template based on the bot endpoint that received a message. Reply with an object instead of a string, and the Bot Builder will not do any specific parsing, letting you take advantage of more advanced bot features for individual platforms. Remember to configure your Lambda function for longer execution if you plan to use asynchronous replies; by default, AWS limits this to 3 seconds.

Try it out live

You can see this bot in action and play with it live from the GitHub Claudia Examples repository.

More information

For more information on the Claudia Bot Builder, and some nice example projects, check out the Claudia Bot Builder GitHub project repository. For questions and suggestions, visit the Claudia project chat room on Gitter.

Content Replication Using AWS Lambda and Amazon S3

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/content-replication-using-aws-lambda-and-amazon-s3/

Co-authored by Felix Candelario and Benjamin F., AWS Solutions Architects

Cross-region replication in Amazon S3 lets you copy from one source bucket to one destination bucket, where the destination bucket resides in a separate region from the source bucket. In order to replicate objects to multiple destination buckets or destination buckets in the same region as the source bucket, customers must spin up custom compute resources to manage and execute the replication.

In this post, I describe a solution for replicating objects from a single S3 bucket to multiple destination S3 buckets using an AWS Lambda function. This solution is presented as a complement to cross region replication for specific use cases that require either multiple destination buckets, or a destination bucket that resides in the same region as the source.

The solution leverages S3 event notification, Amazon SNS, and a simple Lambda function to perform continuous replication of objects. Similar to cross-region replication, this solution only replicates new objects added to the source bucket after configuring the function, and does not replicate objects that existed prior to the function’s existence.

Note that while this method offers functionality not currently offered by cross region replication, it also incurs costs that cross region replication would not incur, and has limitations which are noted at the end of the post.

Solution overview

The solution requires one S3 source bucket and at least one destination bucket. The buckets can reside either in the same region or in different regions. On the source bucket, you create an event notification that publishes to an SNS topic. The SNS topic acts as the mechanism to fan out object copying to one or more destinations, each achieved by invoking a separate Lambda function. The function source code is provided as an example that accompanies this post.

You can define a new destination by creating a subscription to the SNS topic that invokes the Lambda function. There can be multiple Lambda functions subscribed to the same topic, each performing the same action, but to a different destination S3 bucket. Define which bucket and which Lambda function by naming the function with the exact name of the S3 destination bucket. There is no need to edit the function code, and the different functions can be identical with the exception of their names. After they are invoked, the functions copy new source bucket objects to the destination buckets simultaneously.

Required IAM permissions

In order for a Lambda function to be able to copy an object, it requires a Lambda function IAM execution role. The function requires S3 GET permissions on the source bucket and S3 PUT permissions on any destination bucket. This needs to be created for each instance of the function (for each destination), calling out the respective destination bucket in the policy. An example IAM policy is provided later in this post.

The user that create the IAM role is passing permissions to Lambda to assume this role. To grant these permissions, the user must already have permissions to perform the iam:PassRole action.

Solution walkthrough

The following walkthrough install the S3 replication functionality using Lambda, SNS topics, and S3 event notifications. This walkthrough assumes that you have created or identified an S3 source bucket and one or more destination buckets. Note the bucket names for later.

Configure the SNS topic and source bucket

The following steps only need to be done one time per source bucket.

Create the SNS topic to fan out

  1. In the SNS console, create a new SNS topic. Note the topic name for later. A topic is created one time per S3 bucket source, so consider naming the topic as follows: [source-bucket-name]-fanout
  2. Note the SNS topic’s ARN string and then choose Other topic actions , Edit topic policy , and Advanced View.
  3. Replace the contents of the default policy with the following:
{
  "Version": "2008-10-17",
  "Id": "<default_policy_ID>",
  "Statement": [
    {
      "Sid": "<default_statement_ID>",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "arn:aws:sns:us-east-1:123123123123:s3-source-bucket-fanout",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:*:*:s3-source-bucket-name"
        }
      }
    }
  ]
}
  1. Make the following changes in the policy that are marked in red:
    1. For Resource , change to the ARN value for the SNS topic.
    2. For AWS:SourceArn , change to the ARN value for the S3 source bucket.
  2. Choose Update policy.

Configure the source bucket

  1. In the S3 console, edit the source bucket configuration.
  2. Expand the Events section and provide a name for the new event. For example: S3 replication to dst buckets: dstbucket1 dstbucket2
  3. For Events , choose ObjectCreated (ALL).
  4. For Send to , choose SNS topic.
  5. For SNS topic , select the topic name chosen in step 2.
  6. Choose Save.

Configure the Lambda function and SNS subscription

The following steps are repeated for each destination bucket. Additional destinations can be added anytime; however, only subsequently created objects will be replicated to the new destination buckets. Note that the Lambda replication functions are executed in parallel as they are triggered simultaneously via their SNS subscriptions. The following steps only need to be done one time per destination bucket.

Create the Lambda function and IAM policy

  1. In the Lambda console, choose Create a Lambda function.
  2. Choose Skip to skip the blueprint selection.
  3. For Runtime , choose Python 2.7.
  4. For Name , enter a function name. The function name should match the name of the S3 destination bucket exactly.
  5. Enter a description that notes the source bucket and destination bucket used.
  6. For Code entry type , choose Edit code inline.
  7. Paste the following into the code editor:
import urllib
import boto3
import ast
import json
print('Loading function')

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    sns_message = ast.literal_eval(event['Records'][0]['Sns']['Message'])
    target_bucket = context.function_name
    source_bucket = str(sns_message['Records'][0]['s3']['bucket']['name'])
    key = str(urllib.unquote_plus(sns_message['Records'][0]['s3']['object']['key']).decode('utf8'))
    copy_source = {'Bucket':source_bucket, 'Key':key}
    print "Copying %s from bucket %s to bucket %s ..." % (key, source_bucket, target__bucket)
    s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)
  1. For Handler , leave the default value: lambdafunction.lambdahandler
  2. For Role , choose Create new role , basic execution role.
  3. In the IAM dialog box, create a new IAM execution role for Lambda.
  4. For Role Name , enter a value that includes the destination bucket name. For example: s3replicationexecutionroletobucket[dstbucketname]
  5. Expand View Policy Document and choose Edit the policy.
  6. Choose OK to confirm that you’ve read the documentation.
  7. Replace the contents of the default policy with the following:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::source-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::destination-bucket-name-1/*"
            ]
        }
    ]
}
  1. Make the following changes in the policy that are marked in red:
    1. Under the s3:GetObject action, change to the ARN value for the source bucket.
    2. Under the s3:PutObject action, change to the ARN value for the destination bucket.
  2. Choose Allow to save the policy and close the window.
  3. For Timeout , keep the default value 5 minutes.
  4. For VPC , leave the default value No VPC.
  5. Choose Next.
  6. Review the configuration and choose Create Function.

Create the SNS topic subscription

  1. In the SNS console, choose Topics and select the fan out topic [source-bucket-name]-fanout created earlier. Enter the topic’s details page.
  2. Choose Create Subscription.
  3. For Protocol, choose AWS Lambda.
  4. For Endpoint, select the function ARN that represents the destination bucket.
  5. Choose Create Subscription.

Validate the subscription

  1. Upload an object to the source bucket.
  2. Verify that the object was copied successfully to the destination buckets.
  3. Optional: view the CloudWatch logs entry for the Lambda function execution. For a successful execution, this should look similar to the following screenshot.

CloudWatchLogs

Conclusion

This method is simple, and addresses use cases not currently addressed by cross region replication. If you have any suggestions or comments, please feel free to comment below.

Notes

Costs : Getting and putting objects into S3 buckets incurs a cost, as well as cross region data transfer costs. Please see the S3 pricing page for more details.

Lambda : The limitation on file size that the above solution can support is variable and depends on the latency between the source and destination buckets. The Lambda function times out after 5 minutes; if the file copy has yet to complete within that time frame, it does not replicate successfully. Therefore, we recommend running multiple tests after setting up the different destination functions and, based on the results, rely on this method only for file sizes that consistently manage to replicate.

It is possible to expand this solution so that each Lambda execution reports at the end of the copy that replication has completed successfully. The result is a repository that can be queried to verify ongoing replication and that alerts on errors. One approach would be reporting to a DynamoDB table and logging each successful copy. This is beyond the scope of this post.

Also, we recommend carefully monitoring the number of files put into the source bucket as you may need to request an increase to the concurrent execution limits for Lambda.

Error Handling Patterns in Amazon API Gateway and AWS Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/error-handling-patterns-in-amazon-api-gateway-and-aws-lambda/


Ryan Green @ryangtweets
Software Development Engineer, API Gateway

A common API design practice is to define an explicit contract for the types of error responses that the API can produce. This allows API consumers to implement a robust error-handling mechanism which may include user feedback or automatic retries, improving the usability and reliability of applications consuming your API.

In addition, well-defined input and output contracts, including error outcomes, allows strongly-typed SDK client generation which further improves the application developer experience. Similarly, your API backend should be prepared to handle the various types of errors that may occur and/or surface them to the client via the API response.

This post discusses some recommended patterns and tips for handling error outcomes in your serverless API built on Amazon API Gateway and AWS Lambda.

HTTP status codes

In HTTP, error status codes are generally divided between client (4xx) and server (5xx) errors. It’s up to your API to determine which errors are appropriate for your application. The table shows some common patterns of basic API errors.

TypeHTTP status codeDescription
Data Validation400 (Bad Request)The client sends some invalid data in the request, for example, missing or incorrect content in the payload or parameters. Could also represent a generic client error.
Authentication/Authorization401 (Unauthorized)
403 (Forbidden)
The client is not authenticated (403) or is not authorized to access the requested resource (401).
Invalid Resource404 (Not Found)The client is attempting to access a resource that doesn’t exist.
Throttling429 (Too Many Requests)The client is sending more than the allowed number of requests per unit time.
Dependency Issues502 (Bad Gateway)
504 (Gateway Timeout)
A dependent service is throwing errors (502) or timing out (504).
Unhandled Errors500 (Internal Server Error)
503 (Service Unavailable)
The service failed in an unexpected way (500), or is failing but is expected to recover (503).

For more information about HTTP server status codes, see RFC2616 section 10.5 on the W3C website.

Routing Lambda function errors to API Gateway HTTP responses

In API Gateway, AWS recommends that you model the various types of HTTP responses that your API method may produce, and define a mapping from the various error outcomes in your backend Lambda implementation to these HTTP responses.

In Lambda, function error messages are always surfaced in the “errorMessage” field in the response. Here’s how it’s populated in the various runtimes:

Node.js (4.3):

exports.handler = function(event, context, callback) {
    callback(new Error("the sky is falling!");
};

Java:

public class LambdaFunctionHandler implements RequestHandler<String, String> {
  @Override
    public String handleRequest(String input, Context context) {
        throw new RuntimeException("the sky is falling!");
   }
}

Python:

def lambda_handler(event, context):
    raise Exception('the sky is falling!')
Each results in the following Lambda response body:
{
  "errorMessage" : "the sky is falling!",
    …
}

The routing of Lambda function errors to HTTP responses in API Gateway is achieved by pattern matching against this “errorMessage” field in the Lambda response. This allows various function errors to be routed to API responses with an appropriate HTTP status code and response body.

The Lambda function must exit with an error in order for the response pattern to be evaluated – it is not possible to “fake” an error response by simply returning an “errorMessage” field in a successful Lambda response.

Note: Lambda functions failing due to a service error, i.e. before the Lambda function code is executed, are not subject to the API Gateway routing mechanism. These types of errors include internal server errors, Lambda function or account throttling, or failure of Lambda to parse the request body. Generally, these types of errors are returned by API Gateway as a 500 response. AWS recommends using CloudWatch Logs to troubleshoot these types of errors.

API Gateway method response and integration response

In API Gateway, the various HTTP responses supported by your method are represented by method responses. These define an HTTP status code as well as a model schema for the expected shape of the payload for the response.

Model schemas are not required on method responses but they enable support for strongly-typed SDK generation. For example, the generated SDKs can unmarshall your API error responses into appropriate exception types which are thrown from the SDK client.

The mapping from a Lambda function error to an API Gateway method responseis defined by an integration response. An integration response defines a selection pattern used to match the Lambda function “errorMessage” and routes it to an associated method response.

Note: API Gateway uses Java pattern-style regexes for response mapping. For more information, see Pattern in the Oracle documentation.

Example:

Lambda function (Node.js 4.3):

exports.handler = (event, context, callback) => {
   callback ("the sky is falling!");
};

Lambda response body:

{
  "errorMessage": "the sky is falling!"
}

API Gateway integration response:

Selection pattern : “the sky is falling!”

Method response : 500

API Gateway response:

Status: 500

Response body:

{
  "errorMessage": "the sky is falling!"
}

In this example, API Gateway returns the Lambda response body verbatim, a.k.a. “passthrough”. It is possible to define mapping templates on the integration response to transform the Lambda response body into a different form for the API Gateway method response. This is useful when you want to format or filter the response seen by the API client.

When a Lambda function completes successfully or if none of the integration response patterns match the error message, API Gateway responds with the default integration response (typically, HTTP status 200). For this reason, it is imperative that you design your integration response patterns such that they capture every possible error outcome from your Lambda function. Because the evaluation order is undefined, it is unadvisable to define a “catch-all” (i.e., “.*”) error pattern which may be evaluated before the default response.

Common patterns for error handling in API Gateway and Lambda

There are many ways to structure your serverless API to handle error outcomes. The following section will identify two successful patterns to consider when designing your API.

Simple prefix-based

This common pattern uses a prefix in the Lambda error message string to route error types.

You would define a static set of prefixes, and create integration responses to capture each and route them to the appropriate method response. An example mapping might look like the following:

PrefixMethod response status
[BadRequest]400
[Forbidden]403
[NotFound]404
[InternalServerError]500

Example:

Lambda function (NodeJS):

exports.handler = (event, context, callback) => {
    callback("[BadRequest] Validation error: Missing field 'name'");
};

Lambda output:

{
  "errorMessage": "[BadRequest] Validation error: Missing field 'name'"
}

API Gateway integration response:

Selection pattern: “^[BadRequest].*”

Method response: 400

API Gateway response:

Status: 400

Response body:

{
  "errorMessage": "[BadRequest] Validation error: Missing field 'name'"
}

If you don’t want to expose the error prefix to API consumers, you can perform string processing within a mapping template and strip the prefix from the errorMessage field.

Custom error object serialization

Lambda functions can return a custom error object serialized as a JSON string, and fields in this object can be used to route to the appropriate API Gateway method response.

This pattern uses a custom error object with an “httpStatus” field and defines an explicit 1-to-1 mapping from the value of this field to the method response.

An API Gateway mapping template is defined to deserialize the custom error object and build a custom response based on the fields in the Lambda error.

Lambda function (Node.js 4.3):

exports.handler = (event, context, callback) => {
    var myErrorObj = {
        errorType : "InternalServerError",
        httpStatus : 500,
        requestId : context.awsRequestId,
        message : "An unknown error has occurred. Please try again."
    }
    callback(JSON.stringify(myErrorObj));
};

Lambda function (Java):

public class LambdaFunctionHandler implements RequestHandler<String, String> {
  @Override
    public String handleRequest(String input, Context context) {

        Map<String, Object> errorPayload = new HashMap();
        errorPayload.put("errorType", "BadRequest");
        errorPayload.put("httpStatus", 400);
        errorPayload.put("requestId", context.getAwsRequestId());
        errorPayload.put("message", "An unknown error has occurred. Please try again.");
        String message = new ObjectMapper().writeValueAsString(errorPayload);
        
        throw new RuntimeException(message);
    }
}

Note: this example uses Jackson ObjectMapper for JSON serialization. For more information, see ObjectMapper on the FasterXML website.

Lambda output:

{
  "errorMessage": "{\"errorType\":\"InternalServerError\",\"httpStatus\":500,\"requestId\":\"40cd9bf6-0819-11e6-98f3-415848322efb\",\"message\":\"An unknown error has occurred. Please try again.\"}"
}

Integration response:

Selection pattern: “.*httpStatus”:500.*”

Method response: 500

Mapping template:

#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))
{
  "type" : "$errorMessageObj.errorType",
  "message" : "$errorMessageObj.message",
  "request-id" : "$errorMessageObj.requestId"
}

Note: This template makes use of the $util.parseJson() function to parse elements from the custom Lambda error object. For more information, see Accessing the $util Variable.

API Gateway response:

Status: 500

Response body:

{
  "type": "InternalServerError",
  "message": " An unknown error has occurred. Please try again.",
  "request-id": "e308b7b7-081a-11e6-9ab9-117c7feffb09"
}

This is a full Swagger example of the custom error object serialization pattern. This can be imported directly into API Gateway for testing or as a starting point for your API.

{
  "swagger": "2.0",
  "info": {
    "version": "2016-04-21T23:52:49Z",
    "title": "Best practices for API error responses with API Gateway and Lambda"
  },
  "schemes": [
    "https"
  ],
  "paths": {
    "/lambda": {
      "get": {
        "consumes": [
          "application/json"
        ],
        "produces": [
          "application/json"
        ],
        "parameters": [
          {
            "name": "status",
            "in": "query",
            "required": true,
            "type": "string"
          }
        ],
        "responses": {
          "200": {
            "description": "200 response",
            "schema": {
              "$ref": "#/definitions/Empty"
            }
          },
          "400": {
            "description": "400 response",
            "schema": {
              "$ref": "#/definitions/Error"
            }
          },
          "403": {
            "description": "403 response",
            "schema": {
              "$ref": "#/definitions/Error"
            }
          },
          "404": {
            "description": "404 response",
            "schema": {
              "$ref": "#/definitions/Error"
            }
          },
          "500": {
            "description": "500 response",
            "schema": {
              "$ref": "#/definitions/Error"
            }
          }
        },
        "x-amazon-apigateway-integration": {
          "responses": {
            "default": {
              "statusCode": "200"
            },
            ".\*httpStatus\\\":404.\*": {
              "statusCode": "404",
              "responseTemplates": {
                "application/json": "#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))\n#set ($bodyObj = $util.parseJson($input.body))\n{\n  \"type\" : \"$errorMessageObj.errorType\",\n  \"message\" : \"$errorMessageObj.message\",\n  \"request-id\" : \"$errorMessageObj.requestId\"\n}"
              }
            },
            ".\*httpStatus\\\":403.\*": {
              "statusCode": "403",
              "responseTemplates": {
                "application/json": "#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))\n#set ($bodyObj = $util.parseJson($input.body))\n{\n  \"type\" : \"$errorMessageObj.errorType\",\n  \"message\" : \"$errorMessageObj.message\",\n  \"request-id\" : \"$errorMessageObj.requestId\"\n}"
              }
            },
            ".\*httpStatus\\\":400.\*": {
              "statusCode": "400",
              "responseTemplates": {
                "application/json": "#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))\n#set ($bodyObj = $util.parseJson($input.body))\n{\n  \"type\" : \"$errorMessageObj.errorType\",\n  \"message\" : \"$errorMessageObj.message\",\n  \"request-id\" : \"$errorMessageObj.requestId\"\n}"
              }
            },
            ".\*httpStatus\\\":500.\*": {
              "statusCode": "500",
              "responseTemplates": {
                "application/json": "#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))\n#set ($bodyObj = $util.parseJson($input.body))\n{\n  \"type\" : \"$errorMessageObj.errorType\",\n  \"message\" : \"$errorMessageObj.message\",\n  \"request-id\" : \"$errorMessageObj.requestId\"\n}"
              }
            }
          },
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "{\"failureStatus\" : $input.params('status')\n}"
          },
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/[MY_FUNCTION_ARN]/invocations",
          "type": "aws"
        }
      }
    }
  },
  "definitions": {
    "Empty": {
      "type": "object"
    },
    "Error": {
      "type": "object",
      "properties": {
        "message": {
          "type": "string"
        },
        "type": {
          "type": "string"
        },
        "request-id": {
          "type": "string"
        }
      }
    }
  }
}

Conclusion

There are many ways to represent errors in your API. While API Gateway and Lambda provide the basic building blocks, it is helpful to follow some best practices when designing your API. This post highlights a few successful patterns that we have identified but we look forward to seeing other patterns emerge from our serverless API users.

Airtime Relaunches with Major Overhaul using ECS

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/airtime-relaunches-with-major-overhaul-using-ecs/

Guest post by Abby Fuller (@abbyfuller) of Airtime

A major part of the recent Airtime relaunch was an architecture overhaul changing our application from monolith to microservices.

This switch was intended to solve some long-standing issues (bottlenecked deployments, limited redundancy, and inconsistent testing and staging/production environments), while keeping in mind our major priorities: user experience above all, developer efficiency, automation is king, and setting ourselves up to scale properly. We rebuilt our entire platform on Docker and Amazon ECS, which allowed us to rebuild relatively quickly.

To learn more, continue reading the full blog article @ Airtime

Sumo Logic App for AWS Lambda Released

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/sumo-logic-app-for-aws-lambda/

Today, Sumo Logic made its App for AWS Lambda generally available and introduced a purpose-built Lambda function, which is immediately available for launch in the AWS Blueprint library.

Now, customers using Lambda functions can better visualize things like metering for code execution, predict compute usage, and monitor memory limits through Sumo Logic’s predictive analytics capabilities, thus helping to optimize performance and cost.

Sumo Logic App for AWS Lambda

Lambda handles the scaling of the service behind the scenes, while sending operational logs to CloudWatch. To gain a better handle on costs, customers have requested greater visibility and predictability around Lambda usage. Sumo Logic is purpose-built for this, allowing you to see statistics like compute usage and billed duration, and drill down into items like compute usage by function. The Sumo Logic App also calculates your unused memory. This is important because when you allocate memory and are billed based on memory size, unused memory can be an indicator that you aren’t allocating appropriately.

Other out-of-the-box insights that users get with the Sumo Logic App include:

  • Compute usage by function. Sumo Logic converts MB to GB and milliseconds to seconds to get the real billing numbers used, and reports over the last 12 hours.
  • Predicted compute usage by function (GB-sec) based on the behavior of the last 12 hours.
  • Billed duration of functions by the hour as a line chart juxtaposed with a trendline for the predicted billed duration on a timeline for the last 24 hours.
  • Maximum memory used by functions as a line on an outlier chart with the threshold displayed as the area, on a timeline for the last 24 hours.

The Sumo Logic App also allows users to drill down to analyze the number of requests, billed duration and memory utilization of an individual function, as well as function version. This allows users to assign a hard number to assess the impact of a code change across different versions of the same Lambda function, and easily make cost/benefit trade-offs.

In other situations, where different departments own and manage their own set of Lambda functions, the Sumo Logic App for AWS Lambda allows the admin to identify the billing contribution of each function and charge that portion of the bill back to a specific department.

How it Works

AWS Lambda generates operation logs which are exposed via CloudWatch. Every time a function is triggered, it generates useful troubleshooting logs. When a new CloudWatch log becomes available, the Sumo Logic Lambda function for CloudWatch routes that log message to a Sumo Logic webhook. This in turn, acts as the data source for the dashboards and searches enclosed within the Sumo Logic App.

Getting Started

Setting up the Lambda function is easy and can be done under 15 minutes. If you don’t already have a Sumo Logic account, you can try it free. Go to www.sumologic.com/signup-free.

Step 1. Create a Lambda function that routes CloudWatch logs to Sumo Logic.

Step 2. Install the Sumo Logic App for AWS Lambda from the library.

Select a source category and choose Install. Choose Go when asked to confirm.

Step 3. Customize the dashboard and add alerts. Modify the parameters of the search queries. Admins and other users can create custom alerts for themselves so they can be warned of potential spikes.

Sumo Logic Lambda functions

Sumo Logic Lambda functions are open source and available on GitHub. There you’ll find source code for Lambda functions for Amazon Kinesis, Amazon S3 and Amazon CloudWatch Logs. The latter is what powers the Sumo Logic App. As CloudWatch logs become available, the Lambda function processes the data and passes it to Sumo Logic. Assuming you have the Sumo Logic App installed, you can then use our pre-built dashboards and searches to get visualizations or troubleshoot and monitor your applications and systems hosted on AWS. We encourage Lambda developers to contribute to the repository.

Summary

As AWS users increasingly find interesting and unique ways to leverage Lambda, their usage of the service and the number of functions they deploy in their accounts will rapidly grow. The simplicity and the power of Lambda makes it easy to create a function for every application and even maintain different versions of each function.

Learn more

Simply Serverless: Use constant values in Cloudwatch event triggered Lambda functions

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/simply-serverless-use-constant-values-in-cloudwatch-event-triggered-lambda-functions/

Welcome to the second post in our new series: Simply Serverless, today we will show you a quick tip for using Cloudwatch Events with AWS Lambda.

When using an AWS Cloudwatch rule to trigger a Lambda event, one of the multiple options you have to pass data onto your Lamba function is “Constant (JSON Text)”. This handy feature allows you to send static content to your function instead of the matched event.

constant values

Keep in mind that you can also create the input targets via the CLI:

aws events put-targets --rule testjson --targets file://targetfile

Hope you’ve enjoyed, and as always I’m listening! @listonb

Bryan Liston

Building, Testing and Deploying Java applications on AWS Lambda using Maven and Jenkins

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/building-testing-and-deploying-java-applications-on-aws-lambda-using-maven-and-jenkins/

Jeff Nunn
Solutions Architect

With continuous integration (the practice of continually integrating code into a shared code repository) and continuous deployment (the practice of building, testing, and deploying code often), developers can release software faster and more frequently.

This post shows how the principles of code testing, continuous integration, and continuous deployment can be applied to your AWS Lambda workflows. Using Git, Maven, and Jenkins, you can integrate, test, build and deploy your Lambda functions using these same paradigms.

As a side note, we will be having a webinar on Continuous Delivery to AWS Lambda which covers more methods of Continuous Delivery on Thursday, April 28th. Register for the webinar.

Prerequisites

  • Jenkins
    Many of our customers use Jenkins as an automation server to perform continuous integration. While the setup of Jenkins is out of scope for this post, you can still learn about unit testing and pushing your code to Lambda by working through this example.
  • A Git repository
    This method uses Git commit hooks to perform builds against code to be checked into Git. You can use your existing Git repository, or create a new Git repository with AWS CodeCommit or other popular Git source control managers.

Getting started

In this example, you are building a simple document analysis system, in which metadata is extracted from PDF documents and written to a database, allowing indexing and searching based on that metadata.

You use a Maven-based Java project to accomplish the PDF processing. To explain concepts, we show snippets of code throughout the post.

Overview of event-driven code

To accomplish the document analysis, an Amazon S3 bucket is created to hold PDF documents. When a new PDF document is uploaded to the bucket, a Lambda function analyzes the document for metadata (the title, author, and number of pages), and adds that data to a table in Amazon DynamoDB, allowing other users to search on those fields.

Lambda executes code in response to events. One such event would be the creation of an object in Amazon S3. When an object is created (or even updated or deleted), Lambda can run code using that object as a source.

image

Create an Amazon DynamoDB table

To hold the document metadata, create a table in DynamoDB, using the Title value of the document as the primary key.

image

For this example, you can set your provisioned throughput to 1 write capacity unit and 1 read capacity unit.

image

Write the Java code for Lambda

The Java function takes the S3 event as a parameter, extracting the PDF object and analyzing the document for metadata using Apache PDFBox, and writing the results to DynamoDB.

// Get metadata from the document
PDDocument document = PDDocument.load(objectData);
PDDocumentInformation metadata = document.getDocumentInformation();
...
String title = metadata.getTitle();

if (title == null) {
    title = "Unknown Title";
}
...
Item item = new Item()
    .withPrimaryKey("Title", title)
    .withString("Author", author)
    .withString("Pages", Integer.toString(document.getNumberOfPages()));

The Maven project comes with a sample S3 event (/src/test/resources/s3-event.put.json) from which you can build your tests.

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "79104EXAMPLEB723",
        "x-amz-id-2": "IOWQ4fDEXAMPLEQM+ey7N9WgVhSnQ6JEXAMPLEZb7hSQDASK+Jd1vEXAMPLEa3Km"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "builtonaws",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::builtonaws"
        },
        "object": {
          "key": "blogs/lambdapdf/aws-overview.pdf",
          "size": 558985,
          "eTag": "ac265da08a702b03699c4739c5a8269e"
        }
      }
    }
  ]
}

Take care to to replace the awsRegion, arn, and key to match your specific region, Amazon Resource Name, and key of the PDF document that you’ve uploaded.

Test your code

The sample code you’ve downloaded contains some basic unit tests. One test gets an item from the DynamoDB table and verifies that the expected metadata exists:

@Test
public void checkMetadataResult() {
    DynamoDB dynamoDB = new DynamoDB(new AmazonDynamoDBClient());
    Table table = dynamoDB.getTable("PDFMetadata");
    Item item = table.getItem("Title", "Overview of Amazon Web Services");

    assertEquals(31, item.getInt("Pages"));
    assertEquals("[email protected]", item.getString("Author"));
    assertEquals("Overview of Amazon Web Services", item.getString("Title"));
}

Before continuing, test your code to ensure that everything is working:

mvn test

After ensuring there are no errors, check your DynamoDB table to see the metadata now added to your table.

DynamoDB

The code executes because of the sample event in your Maven project, but how does it work when a new PDF is added to your bucket? To test this, complete the connection between Amazon S3 and Lambda.

Create a Lambda function

Use mvn package to package your working code, then upload the resulting JAR file to a Lambda function.

  1. In the Lambda console, create a new function and set runtime to Java 8.
  2. Set function package to the project JAR file Maven created in the target folder.
  3. Set handler to “example.S3EventProcessorExtractMetadata”.
  4. Create a new value for role based on the Basic With DynamoDB option. A role gives your function access to interact with other services from AWS. In this case, your function to interacts with both Amazon S3 and Amazon DynamoDB. In the window that opens, choose View Policy Document, then choose Edit to edit your policy document.

Policy

While this document gives your function access to your DynamoDB resources, you need to add access to your S3 resources as well. Use the policy document below to replace the original.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    },
    {
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

It is important to note that this policy document allows your Lambda function access to all S3 and DynamoDB resources. You should lock your roles down to interact only with those specific resources that you wish the function to have access to.

After completing your policy document and reviewing the function settings, choose Create Function.

Create Amazon S3 bucket

  1. Create a bucket in the Amazon S3 console. Note that buckets created in S3 are in a global namespace: you need to create a unique name.
  2. After the bucket is created, upload the Overview of Amazon Web Services PDF to your bucket. We’ve included this white paper to use in unit tests for debugging your Lambda function.
  3. Manage events for your S3 bucket by going to the root bucket properties and choosing Events:S3
  4. Give your event a name, such as “PDFUploaded”.
  5. For Events, choose Object Created (all).
  6. For Prefix, list the key prefix for the subdirectory that holds your PDFs, if any. If you want to upload PDF documents to the root bucket, you can leave this blank. If you made a “subdirectory” called “pdf”, then the prefix would be “pdf”.
  7. Leave Suffix blank, and choosing Lambda function as the Send To option, choosing the Lambda function you created.
  8. Choose Save to save your S3 event.

Test everything

Test the entire process by uploading a new PDF to your bucket. Verify that a new entry was added to your DynamoDB table.

S3

(To troubleshoot any errors, choose Monitoring in your Lambda function to view logs generated by Amazon CloudWatch.)

Enter Jenkins

At this point, you have created a testable Java function for Lambda that uses an S3 event to analyze metadata from a PDF document and stores that information in a DynamoDB table.

In a CI/CD environment, changes to the code might be made and uploaded to a code repository on a frequent basis. You can bring those principles into this project now by configuring Jenkins to perform builds, package a JAR file, and ultimately push the JAR file to Lambda. This process can be based off of a Git repo, by polling the repo for changes, or using Git’s built-in hooks for post-receive or post-commit actions.

Build hooks

Use the post-commit hook to trigger a Jenkins build when a commit is made to the repo. (For the purposes of this post, the repo was cloned to the Jenkins master, allowing you to use the post-commit hook.)

To enable Jenkins to build off Git commits, create a Jenkins project for your repo with the Git plugin, set Build Trigger to “Poll SCM”, and leave Schedule blank.

In your project folder, find .git/hooks/post-commit and add the following:

#!/bin/sh

curl http://<jenkins-master>:8080/job/<your-project-name>/build?delay=0sec

This ensures that when a commit is made in this project, a request is made to your project’s build endpoint on Jenkins. You can try it by adding or modifying a file, committing it to your repo, and examining the build history and console output in your Jenkins dashboard for the status update.

S3

(For more information about implementing a post-receive hook, see the Integrating AWS CodeCommit with Jenkins AWS DevOps Blog post.)

Deploy code to Lambda

You may notice in the console output a command for aws sns publish --topic-arn .... In this project, we’ve added a post-build step to publish a message via Amazon Simple Notification Service (Amazon SNS) as an SMS message. You can add a similar build step to do the same, or take advantage of SNS to HTTP(S) endpoints to post status messages to team chat applications or a distributed list.

However, to be able to push the code to AWS Lambda after a successful commit and build, look at adding a post build step.

  1. In the configuration settings for your project, choose Add built step and Invoke top-level Maven targets, setting Goal to “package”. This packages up your project as a JAR file and places it into the target directory.
  2. Add a second build step by choosing Add built step and the Execute shell option.
  3. For Command, add the following Lambda CLI command (substitute the function-name variable and zip-file variable as necessary):
    aws lambda update-function-code --function-name extractPDFMeta --zip-file fileb://target/lambda-java-example-1.0.jar

You have now added the necessary build steps for Jenkins to test and package your code, then upload it to Lambda. Test the entire process start to finish by adding a new PDF document into your S3 bucket and checking the DynamoDB table for changes.

Take it a step further

The code and architecture described here are meant to serve as illustrations for testing your Lambda functions and building out a continuous deployment process using Maven, Jenkins, and AWS Lambda. If you’re running this in a production environment, there may be additional steps you would want to take. Here are a few:

  • Add additional unit tests
  • Build in additional features and sanity checks, for example, to make sure documents to be analyzed are actually PDF documents
  • Adjust the Write Capacity Units (WCU) of your DynamoDB table to accommodate higher levels of traffic
  • Add an additional Jenkins post-build step to integrate Amazon SNS to send a notification about a successful Lambda deployment

Simply Serverless: Using AWS Lambda to Expose Custom Cookies with API Gateway

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/simply-serverless-using-aws-lambda-to-expose-custom-cookies-with-api-gateway/

Simply Serverless

Welcome to a new series on quick and simple hacks/tips/tricks and common use cases to using AWS Lambda and AWS API Gateway. As always, I’m listening to readers (@listonb), so if you have any questions, comments or tips you’d like to see, let me know!

This is a guest post by Jim Warner from Survata.

This first tip describes how Survata uses Lambda to drop a new cookie on API Gateway requests. Learn more about how Survata is using this during Serverless Day at the San Francisco Loft on April 28th. Register for Serverless Day now.

Step 1: Return a cookie ID from Lambda

This walkthrough assumes you have gone through the Hello World API Gateway Getting Started Guide code.

Expand upon the “Hello World” example and update it as follows:

'use strict';
exports.handler = function(event, context) {
  console.log("{'Cookie':event['Cookie']}");
  var date = new Date();

  // Get Unix milliseconds at current time plus 365 days
  date.setTime(+ date + (365 * 86400000)); //24 * 60 * 60 * 100
  var cookieVal = Math.random().toString(36).substring(7); // Generate a random cookie string
  
  var cookieString = "myCookie="+cookieVal+"; domain=my.domain; expires="+date.toGMTString()+";";
  
  context.done(null, {"Cookie": cookieString}); 

};

This makes a random string and returns it in JSON format as a proper HTTP cookie string. The result from the Lambda function is as follows:

{"Cookie": "myCookie=t81e70kke29; domain=my.domain; expires=Wed, 19 Apr 2017 20:41:27 GMT;"}

Step 2: Set the cookie in API Gateway

In the API Gateway console, go to the GET Method page and choose Method Response. Expand the default 200 HTTP status, and choose Add Header. Add a new header called “Set-Cookie.”

On the GET Method page, choose Integration Response. Under the Header Mappings section of the default 200 HTTP status, choose the pencil icon to edit the “Set-Cookie” header. In the mapping value section, put:

integration.response.body.Cookie

Make sure to save the header by choosing the check icon!

For a real production deployment, use a body mapping template to return only the parts of the JSON that you want to expose (so the cookie data wouldn’t show here).

Deploying both the Lambda function and API Gateway gets you up and cookie-ing.

How to turn Node.js projects into AWS Lambda microservices easily with ClaudiaJS

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/how-to-turn-node-js-projects-into-aws-lambda-microservices-easily-with-claudiajs/

This is a guest post by Gojko Adzic, creator of ClaudiaJS

 

While working on MindMup 2.0, we started moving parts of our API and back-end infrastructure from Heroku to AWS Lambda. The first Lambda function we created required a shell script of about 120 lines of AWS command-line calls to properly set up, and the second one had a similar number with just minor tweaks. Instead of duplicating this work for each service, we decided to create an open-source tool that can handle the deployment process for us.

Enter Claudia.js: an open-source deployment tool for Node.js microservices that makes getting started with AWS Lambda and Amazon API Gateway very easy for JavaScript developers.

Claudia takes care of AWS deployment workflows, simplifying and automating many error-prone tasks, so that you can focus on solving important business problems rather than worrying about infrastructure code. Claudia sets everything up the way JavaScript developers expect out of the box, and significantly shortens the learning curve required to get Node.js projects running inside Lambda.

Hello World

Here’s a quick ‘hello world’ example.

Create a directory, and initialize a new NPM project

npm init

Next, create app.js with the following code:

var ApiBuilder = require('claudia-api-builder'),
  api = new ApiBuilder();
module.exports = api;

api.get('/hello', function () {
  return 'hello world';
});

Add the Claudia API Builder as a project dependency:

npm install claudia-api-builder --save

Finally, install Claudia.js in your global path:

npm install -g claudia

That’s pretty much it. You can now install your new microservice in AWS by running the following command:

claudia create --region us-east-1 --api-module app

In a few moments, Claudia will respond with the details of the newly-installed Lambda function and REST API.

{
  "lambda": {
    "role": "test-executor",
    "name": "test",
    "region": "us-east-1"
  },
  "api": {
    "id": "8x7uh8ho5k",
    "module": "app",
    "url": "https://8x7uh8ho5k.execute-api.us-east-1.amazonaws.com/latest"
  }
}

The result contains the root URL of your new API Gateway resource. Claudia automatically created an endpoint resource for /hello, so just add /hello to the URL, and try it out in a browser or from the console. You should see the ‘hello world’ response.

That’s it! Your first Claudia-deployed Lambda function and API Gateway endpoint is now live on AWS!

What happened in the background?

In the background, Claudia.js executed the following steps:

  • Created a copy of the project.
  • Packaged all the NPM dependencies.
  • Tested that the API is deployable.
  • Zipped up your application and deployed it to Lambda.
  • Created the correct IAM access privileges.
  • Configured an API Gateway endpoint with the /hello resource.
  • Linked the new resource to the previously-deployed Lambda function.
  • Installed the correct API Gateway transformation templates.

Finally, it saved the resulting configuration into a local file (claudia.json), so that you can easily update the function without remembering any of those details.

Try this next:

Install the superb module as a project dependency:

npm install superb --save

Add a new endpoint to the API by appending these lines to api.js:

api.get('/greet', function (request) {
  var superb = require('superb');
  return request.queryString.name + ' is ' + superb();
});

You can now update your existing deployed application by executing the following command:

claudia update

When the deployment completes, try out the new endpoint by adding `/greet?name=’ followed by your name.

Benefits of using Claudia

Claudia significantly reduces the learning curve for deploying and managing serverless style applications, REST API, and event-driven microservices. Developers can use Lambda and API Gateway in a way that is similar to popular lightweight JavaScript web frameworks.

All the query string arguments are immediately available to your function in the request.queryString object. HTTP Form POST variables are in request.post, and any JSON, XML, or text content posted in as raw body text are in request.body.

Asynchronous processes are also easy; just return a Promise from the API endpoint handler, and Claudia waits until the promise resolves before responding to the caller. You can use any A+ Promise-compliant library, including the promises supported out of the box by the new AWS Lambda 4.3.2 runtime.

To make serverless-style applications easier to set up, Claudia automatically enables cross-origin resource sharing (CORS), so a client browser can call your new API directly even from a different domain. All errors are, by default, triggering the 500 HTTP code, so your API works well with most AJAX libraries. You can, of course, easily customize the API endpoints to return a different content type or HTTP response code, or include additional headers. For more information, see the Claudia API Builder documentation.

Conclusion

Claudia helps people get started quickly, and easily migrate existing, self-hosted or third-party-hosted APIs into Lambda. Because it’s not opinionated and does not require a particular structure or way of working, teams can easily start chopping pieces of existing infrastructure and gradually moving it over. For more information visit the git repository for Sample ClaudiaJS Projects.

Building Enterprise Level Web Applications on AWS Lambda with the DEEP Framework

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/building-enterprise-level-web-applications-on-aws-lambda-with-deep/

This is a guest post by Eugene Istrati, the co-creator of the DEEP Framework, a full-stack web framework that enables developers to build cloud-native applications using microservices architecture.

 

From the beginning, Mitoc Group has been building web applications for enterprise customers. We are a small group of developers who are helping customers with their entire web development process, from conception through execution and down to maintenance. Being in the business of doing everything is very hard, and it would be impossible without using AWS foundational services, but we incrementally needed more. That is why we became early adopters of the serverless computing approach and developed an ecosystem called Digital Enterprise End-to-end Platform (DEEP) with AWS Lambda at the core.

In this post, we dive deeper into how DEEP is using AWS Lambda to empower developers to build cloud-native applications or platforms using microservices architecture. We will walk you through the process of identifying the front-end, back-end and data tiers required to build web applications with AWS Lambda at the core. We will focus on the structure of the AWS Lambda functions we use, as well as security, performance and benchmarking steps that we take to build enterprise-level web applications.

Enterprise-level web applications

Our approach to web development is full-stack and user-driven, focused on UI (aka the user interface) and UX (aka user eXperience). Before going into the details, we’d like to emphasize the strategical (biased and opinionated) decisions we made early on:

  • We don’t say “no” to customers; every problem is seriously evaluated and sometimes we offer options that involve our direct competitors.
  • We are developers and we focus only on the application level; everything else (platform level and infrastructure level) must be managed by AWS.
  • We focus 20% of effort to solve 80% of work load; everything must be automated and pushed on the service side rather than the client side.

To be honest and fair, it doesn’t work all the time as expected, but it does help us to learn fast and move quickly, sustainably and incrementally solving business problems through technical solutions that really matter. However, the definition of “really matter” differs from customer to customer, quite uniquely in some cases.

Nevertheless, what we have learned from our customers is that enterprise-level web applications must provide the following common expectations:

Architecture

This post describes how we transformed a self-managed task management application (aka todo app) in minutes. The original version can be seen on www.todomvc.com and the original code can be downloaded from https://github.com/tastejs/todomvc/tree/master/examples/angularjs.

The architecture of every web application we build or transform, including the one described above, is similar to the reference architecture of the realtime voting application published recently by AWS on GitHub.

The todo app is written in AngularJS and deployed on Amazon S3, behind Amazon CloudFront (front-end). Task management is processed by AWS Lambda, optionally behind Amazon API Gateway (back-end). Task metadata is stored in Amazon DynamoDB (data tier). The transformed todo app, along with instructions on how to install and deploy this web application, is described in the Building Scalable Web Apps with AWS Lambda and Home-Grown Serverless blog post and the todo code is available on GitHub.

Let’s look at AWS Lambda functions and the value proposition they offer to us and our customers.

AWS Lambda functions

The goal of the todo app is to manage tasks in a self-service mode. End users can view tasks, create new tasks, mark or unmark a task as done, and clear completed tasks. From the UI point of view, that leads to four user interactions that require different back-end calls:

  • web service that retrieves tasks
  • web service that creates tasks
  • web service that deletes tasks
  • web service that updates tasks

A simple reordering of the above identified back-end services calls leads to basic CRUD (create, retrieve, update, delete) operations on the Task data object. These are the simple logical steps that we take to identify the front-end, back-end, and data tiers of (drums beating, trumpets playing) our approach to microservices, which we prefer to call microapplications.

Therefore, coming back to AWS Lambda, we have written four small Node.js functions that are context-bounded and self-sustained (each microservice corresponds to the above identified back-end web service):

Microservice that retrieves tasks
'use strict';

import DeepFramework from 'deep-framework';

export default class Handler extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let taskId = request.getParam('Id');

    if (taskId) {
      this.retrieveTask(taskId, (task) => {
        return this.createResponse(task).send();
      });
    } else {
      this.retrieveAllTasks((result) => {
        return this.createResponse(result).send();
      });
    }
  }

  /**
   * @param {Function} callback
   */
  retrieveAllTasks(callback) {
    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.findAll((err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return callback(task.Items);
    });
  }

  /**
   * @param {String} taskId
   * @param {Function} callback
   */
  retrieveTask(taskId, callback) {
    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.findOneById(taskId, (err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return callback(task ? task.get() : null);
    });
  }
}
Microservice that creates a task
'use strict';

import DeepFramework from 'deep-framework';

export default class extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.createItem(request.data, (err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return this.createResponse(task.get()).send();
    });
  }
}
Microservice that updates a task
'use strict';

import DeepFramework from 'deep-framework';

export default class Handler extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let taskId = request.getParam('Id');

    if (typeof taskId !== 'string') {
      throw new InvalidArgumentException(taskId, 'string');
    }

    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.updateItem(taskId, request.data, (err, task) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return this.createResponse(task.get()).send();
    });
  }
}
Microservice that deletes a task
'use strict';

import DeepFramework from 'deep-framework';

export default class extends DeepFramework.Core.AWS.Lambda.Runtime {
  /**
   * @param {Array} args
   */
  constructor(...args) {
    super(...args);
  }

  /**
   * @param request
   */
  handle(request) {
    let taskId = request.getParam('Id');

    if (typeof taskId !== 'string') {
      throw new DeepFramework.Core.Exception.InvalidArgumentException(taskId, 'string');
    }

    let TaskModel = this.kernel.get('db').get('Task');

    TaskModel.deleteById(taskId, (err) => {
      if (err) {
        throw new DeepFramework.Core.Exception.DatabaseOperationException(err);
      }

      return this.createResponse({}).send();
    });
  }
}

Each above file with related dependencies is compressed into .zip file and uploaded to AWS Lambda. If you’re new to this process, we strongly recommend following the How to Create, Upload and Invoke an AWS Lambda function tutorial.

Back to the four small Node.js functions, you can see that we have adopted ES6 (aka ES2015) as our coding standard. And we are importing deep-framework in every function. What is this framework anyway and why are we using it everywhere?

Full-stack web framework

Step back for a minute. Building and uploading AWS Lambda functions to the service is very simple and straight-forward, but now imagine that you need to manage 100–150 web services to access a web page, multiplied by hundreds or thousands of web pages.

We believe that the only way to achieve this kind of flexibility and scale is automation and code reuse. These principles led us to build and open source DEEP Framework — a full-stack web framework that abstracts web services and web applications from specific cloud services — and DEEP CLI (aka deepify) — a development tool-chain that abstracts package management and associated development operations.

Therefore, to make sure that the process of managing AWS Lambda functions is streamlined and automated, we consistently include two more files in each uploaded .zip:

DEEP microservice bootstrap
'use strict';

import DeepFramework from 'deep-framework';
import Handler from './Handler';

export default DeepFramework.LambdaHandler(Handler);
DEEP microservice package metadata (for npm) 
{
  "name": "deep-todo-task-create",
  "version": "0.0.1",
  "description": "Create a new todo task",
  "scripts": {
    "postinstall": "npm run compile",
    "compile": "deepify compile-es6 `pwd`"
  },
  "dependencies": {
    "deep-framework": "^1.8.x"
  },
  "preferGlobal": false,
  "private": true,
  "analyze": true
}

Having these three files (Handler.es6, bootstrap.es6, and package.json) in each Lambda function doesn’t mean that your final .zip file will be that small. Actually, a lot of additional operations happen before the .zip file is created. To name a few:

  • AWS Lambda performs better when the uploaded codebase is smaller. Because we provide both local development capabilities and one-step push to production, our process optimizes resources before deploying to AWS.
  • ES6 is not supported by the node.js v0.10.x runtime that we use in AWS Lambda, it is however available in the Node 4.3 runtime, so we compile .es6 files into ES5-compliant .js files using Babel.
  • Dependencies that are defined in package.json are automatically pulled and fine-tuned for node.js v0.10.x to provide the best performance possible.

Putting everything together

First, you need the following pre-requisites:

  1. AWS account (Create an Amazon Web Services Account)
  2. AWS CLI (Configure AWS Command Line Interface)
  3. Git v2+ (Get Started — Installing Git)
  4. Java / JRE v6+ (JDK 8 and JRE 8 Installation Start Here)
  5. js v4+ (Install nvm and Use latest node v4)

Note: Don’t use sudo to install nvm. Otherwise, you’ll have to fix npm permissions.

Second, install the DEEP CLI with the following command:

npm install deepify -g

Next, deploy the todo app using deepify:

deepify install github://MitocGroup/deep-microservices-todo-app ~/deep-todo-app

deepify server ~/deep-todo-app

deepify deploy ~/deep-todo-app

Note: When the deepify server command is finished, you can open http://localhost:8000 in your browser and enjoy the todo app running locally.

Cleaning up

There are at least half a dozen services and several dozen of resources created during deepify deploy. If only there was a simple command that would clean up everything when we’re done. We thought of that and created deepify undeploy to address this need. When you are done using todo app and want to remove web app related resources, execute the following:

deepify undeploy ~/deep-todo-app

As you can see, we empower developers to build hassle-free, cloud-native applications or platforms using microservices architecture and serverless computing.

And what about security?

Security

One of the biggest value propositions on AWS is out-of-the-box security and compliance. The beauty of the cloud-native approach is that security comes by design (in other words, it won’t work otherwise). We take full advantage of that shared responsibility model and enforce security in every layer.

End users benefit from IAM best practices through streamlined implementations of least privilege access, delegated roles instead of credentials, and integration with logging and monitoring services (e.g., AWS CloudTrail, Amazon CloudWatch, and Amazon Elasticsearch Service + Kibana). For example, developers and end users of the todo app didn’t need to explicitly define any security roles (it was done by deepify deploy), but they can rest assured that only their instance of todo app will be using their infrastructure, platform, and application resources.

The following are two security roles (back-end and front-end) that have been seamlessly generated and enforced in each layer:

IAM role that allows back-end invocation of AWS Lambda function (e.g. DeepProdTodoCreate1234abcd) in web application AWS account (e.g. 123456789000)
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["lambda:InvokeFunction"],
            "Resource": ["arn:aws:lambda:us-east-1:123456789000:function:DeepProdTodoCreate1234abcd*"]
        }
    ]
}
DEEP role that allows front-end resource (e.g deep.todo:task) to execute action (e.g. deep.todo:task:create)
{
  "Version": "2015-10-07",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["deep.todo:task:create"],
      "Resource": ["deep.todo:task"]
    }
  ]
}

Benchmarking

We have been continuously benchmarking AWS Lambda for various use cases in our microapplications. After a couple of repetitive situations doing similar analysis, we decided to build the benchmarking as another microapplication and re-use the ecosystem to include it automatically where we needed it. You can find the open-source code for the benchmarking microapplication on GitHub:

Particularly, for todo app, we performed various benchmarking analysis on AWS Lambda by tweaking different components in a specific function (e.g. function size, memory size, billable cost, etc.). Next, we would like to share results with you:

Benchmarking for todo app

Req NoFunction Size (MB)Memory Size (MB)Max Memory Used (MB)Start timeStop timeFront-end Call (ms)Back-end Call (ms)Billed Time (ms)Billed Time ($)
11.11283420:15.820:16.2359200.473000.000000624
21.11283420:17.820:18.2381202.453000.000000624
31.11283420:19.920:20.3406192.522000.000000416
41.11283420:21.920:22.2306152.192000.000000416
51.11283420:23.920:24.2333175.012000.000000416
61.11283420:25.920:26.3431278.033000.000000624
71.11283420:27.920:28.2323170.972000.000000416
81.11283420:29.920:30.2327160.242000.000000416
91.11283420:31.920:32.4556225.253000.000000624
101.11283520:33.920:34.2333179.592000.000000416
Average375.50193.67Total0.000004992

Performance

Speaking of performance, we find AWS Lambda mature enough to power large-scale web applications. The key is to build the functions as small as possible, focusing on a simple rule of one function to achieve only one task. Over time, these functions might grow in size; therefore, we always keep an eye on them and re-factor / split into the lowest possible logical denominator (smallest task).

Using the benchmarking tool, we ran multiple scenarios on the same function from todo app

Function Size (MB)Memory Size (MB)Max Memory Used (MB)Avg Front-end (ms)Avg Back-end (ms)Total Calls (#)Total Billed (ms)Total Billed ($/1B)*
1.112834-35375.50193.67102,4004,992
1.125634-37399.40153.25102,0008,340
1.151233-35341.60134.32101,80015,012
1.112834-49405.57223.8210027,30056,784
1.125628-48354.75177.9110023,80099,246
1.151232-47345.92163.1710023,100192,654
55.812849-50543.00284.03103,4007,072
55.825649-50339.80153.13102,1008,757
55.851249-50342.60141.02102,00016,680
55.812883-87416.10220.9110026,90055,952
55.825650-71377.69194.2210025,600106,752
55.851257-81353.46174.6510023,300194,322

Based on performance data, we have learned some pretty cool stuff:

  • The smaller the function is, the better it performs; On the other hand, if more memory is allocated, the size of the function matters less and less.
  • Memory size is not directly proportional to billable costs; developers can decide the memory size based on performance requirements combined with associated costs.
  • The key to better performance is continuous load, thanks to container reuse in AWS Lambda.

Conclusion

In this post, we presented a small web application that is built with AWS Lambda at the core. We walked you through the process of identifying the front-end, back-end, and data tiers required to build the todo app. You can fork the example code repository as a starting point for your own web applications.

If you have questions or suggestions, please leave a comment below.

Node.js 4.3.2 Runtime Now Available on Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-now-available-on-lambda/

We are happy to announce that you may now develop your AWS Lambda functions using the Node.js 4.3.2 runtime. You can start using this new runtime version today by specifying a runtime parameter value of “nodejs4.3” when creating or updating functions. We will continue to support creating new Lambda functions on Node.js 0.10. However starting October 2016 you will no longer be able to create functions using Node.js 0.10, given the upcoming end of life for the runtime. Here’s a quick primer on what’s changed between the two versions:

 

New Node features

You can now leverage features in the V8 Javascript Engine such as ES6 Support, block scoping, Promises, and new arrow functions, to name a few. For more information, see the Expressive ES6 features that shine in Node.js 4.0 post by Ryan Paul @ RethinkDB.

 

Backward compatible

Nothing in regards to your existing functions running under Node.js 0.10 will change, and they will continue to operate and function as expected. You may also port your existing Node.js 0.10 functions over to Node.js 4.3.2 by simply updating the runtime, and they will continue to work as written. You will however need to take into account any static modules you may have compiled for 0.10 before making this move. Be sure to review the API changes between Node.js 0.10 and Node.js 4 to see if there are other changes that affect your code.

 

Node callbacks

The programming model for Node.js 0.10, Lambda required an explicit Context method call (done(), suceeed(), fail()) to exit the function. Context.succeed, context.done, and context.fail however, are more than just bookkeeping – they cause the request to return after the current task completes and freeze the process immediately, even if other tasks remain in the Node.js event loop. Generally that’s not what you want if those tasks represent incomplete callbacks.

This programming model for Node.js 4.3.2 improves on this by adding an optional callback parameter to the method. The callback parameter can be used to specify error or return values for the function execution. You specify the optional callback parameter when defining your function handler as below:

exports.myHandler = (event, context, callback) => {null, "I'm running Node4!”};

By default, the callback waits for all the tasks in the Node.js event loop to complete, just as it would if you ran the function locally. If you chose to not use the callback parameter in your code, then AWS Lambda implicitly calls it with a return value of null. You can still use the Context methods to terminate the function, but the callback approach of waiting for all tasks to complete is more idiomatic to how Node.js behaves in general. The context parameter will also continue to exist and provides your handler with the runtime information of the Lambda function that is executing.

If you want to simulate the same behavior as a context method, you now have the ability to access the callbackWaitsForEmptyEventLoop setting via the context object. This property is useful to modify the default behavior of the callback. You can set this property to false to request AWS Lambda to freeze the process after the callback is called. For more information about this new functionality, see Lambda Function Handler.

Remember, the existing Node.js 0.10 programming model does not support the new callback functionality that specifically exists in the new 4.3.2 runtime. If you continue to use 0.10, you will still need to take advantage of the context object to specify return values of your function.

For more information, see the AWS Lambda Developer Guide.

Hope you enjoy,

-Bryan

Have feedback? I’m always listening @listonb

Indexing Amazon DynamoDB Content with Amazon Elasticsearch Service Using AWS Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/indexing-amazon-dynamodb-content-with-amazon-elasticsearch-service-using-aws-lambda/

Stephan Hadinger
Sr Mgr, Solutions Architecture

Mathieu Cadet Account Representative

A lot of AWS customers have adopted Amazon DynamoDB for its predictable performance and seamless scalability. The main querying capabilities of DynamoDB are centered around lookups using a primary key. However, there are certain times where richer querying capabilities are required. Indexing the content of your DynamoDB tables with a search engine such as Elasticsearch would allow for full-text search.

In this post, we show how you can send changes to the content of your DynamoDB tables to an Amazon Elasticsearch Service (Amazon ES) cluster for indexing, using the DynamoDB Streams feature combined with AWS Lambda.

 

Architectural overview

Here’s a high-level overview of the architecture:

DynamoDB Streams to Elasticsearch bridge

We’ll cover the main steps required to put this bridge in place:

  1. Choosing the DynamoDB tables to index and enabling DynamoDB Streams on them.
  2. Creating an IAM role for accessing the Amazon ES cluster.
  3. Configuring and enabling the Lambda blueprint.

 

Choosing the DynamoDB table to index

In this post, you look at indexing the content of a product catalog in order to provide full-text search capabilities. You’ll index the content of a DynamoDB table called all_products, which is acting as the catalog of all products.

Here’s an example of an item stored in that table:

{
  "product_id": "B016JOMAEE",
  "name": "Serverless Single Page Apps: Fast, Scalable, and Available",
  "category": "ebook",
  "description": "AWS Lambda - A Guide to Serverless Microservices
                  takes a comprehensive look at developing 
                  serverless workloads using the new
                  Amazon Web Services Lambda service.",
  "author": "Matthew Fuller",
  "price": 15.0,
  "rating": 4.8
}

Enabling DynamoDB Streams

In the DynamoDB console, enable the DynamoDB Streams functionality on the all_products table by selecting the table and choosing Manage Stream.

Enabling DynamoDB Streams

Multiple options are available for the stream. For this use case, you need new items to appear in the stream; choose either New image or New and old images. For more information, see Capturing Table Activity with DynamoDB Streams.

DynamoDB Streams Options

After the stream is set up, make a good note of the stream ARN. You’ll need that information later, when configuring the access permissions.

Finding a DynamoDB Stream ARN

Creating a new IAM role

The Lambda function needs read access to the DynamoDB stream just created. In addition, the function also requires access to the Amazon ES cluster to submit new records for indexing.

In the AWS Identity and Access Management (IAM) console, create a new role for the Lambda function and call it ddb-elasticsearch-bridge.

Creating new IAM role

As this role will be used by the Lambda function, choose AWS Lambda from the AWS Service Roles list.

Attaching policy to the role

On the following screens, choose the AWSLambdaBasicExecutionRole managed policy, which allows the Lambda function to send logs to Amazon CloudWatch Logs.

Configuring access to the Amazon ES cluster

First, you need a running Amazon ES cluster. In this example, create a search domain called inventory. After the domain has been created, note its ARN:

Attaching policy to the role

In the IAM console, select the ddb-elasticsearch-bridge role created earlier and add two inline policies to that role:

Attaching policy to the role

Here’s the policy to add to allow the Lambda code to push new documents to Amazon ES (replace the resource ARN with the ARN of your Amazon ES cluster):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:ESHttpPost"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:es:us-east-1:0123456789:domain/inventory/*"
        }
    ]
}

Important: you need to add /* to the resource ARN as depicted above.

Next, add a second policy for read access to the DynamoDB stream (replace the resource ARN with the ARN of your DynamoDB stream):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:DescribeStream",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:ListStreams"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:dynamodb:us-east-1:0123456789:table/all_products/stream/2016-02-16T23:13:07.600"
            ]
        }
    ]
}

Enabling the Lambda blueprint

When you log into the Lambda console and choose Create a Lambda Function, you are presented with a list of blueprints to use. Select the blueprint called dynamodb-to-elasticsearch.

dynamodb-to-elasticsearch blueprint

Next, select the DynamoDB table all_products as the event source:

Lambda event source

Then, customize the Lambda code to specify the Elasticsearch endpoint:

Customizing the blueprint

Finally, select the ddb-elasticsearch-bridge role created earlier to give the Lambda function the permissions required to interact with DynamoDB and the Amazon ES cluster:

Choosing a role

Testing the result

You’re all set!

After a few records have been added to your DynamoDB table, you can go back to the Amazon ES console and validate that a new index for your items has been automatically created:

Amazon ES indices

Playing with Kibana (Optional)

Elasticsearch is commonly used with Kibana for visual exploration of data.

To start querying the indexed data, create an index pattern in Kibana. Use the name of the DynamoDB table as an index pattern:

Kibana Index pattern

Kibana automatically determines the best type for each field:

Kibana Index pattern

Use a simple query to search the product catalog for all items in the category book containing the word aws in any field:

Kibana Index pattern

Other considerations

Indexing pre-existing content

The solution presented earlier is ideal to ensure that new data is indexed as soon it is added to the DynamoDB table. But what about pre-existing data stored in the table?

Luckily, the Lambda function used earlier can also be used to process data from an Amazon Kinesis stream, as long as the format of the data is similar to the DynamoDB Streams records.

Provided that you have an Amazon Kinesis stream set up as an additional input source for the Lambda code above, you can use the (very naive) sample Python3 code below to read the entire content of a DynamoDB table and push it to an Amazon Kinesis stream called ddb-all-products for indexing in Amazon ES.

import json
import boto3
import boto3.dynamodb.types

# Load the service resources in the desired region.
# Note: AWS credentials should be passed as environment variables
# or through IAM roles.
dynamodb = boto3.resource('dynamodb', region_name="us-east-1")
kinesis = boto3.client('kinesis', region_name="us-east-1")

# Load the DynamoDB table.
ddb_table_name = "all_products"
ks_stream_name = "ddb-all-products"
table = dynamodb.Table(ddb_table_name)

# Get the primary keys.
ddb_keys_name = [a['AttributeName'] for a in table.attribute_definitions]

# Scan operations are limited to 1 MB at a time.
# Iterate until all records have been scanned.
response = None
while True:
    if not response:
        # Scan from the start.
        response = table.scan()
    else:
        # Scan from where you stopped previously.
        response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])

    for i in response["Items"]:
        # Get a dict of primary key(s).
        ddb_keys = {k: i[k] for k in i if k in ddb_keys_name}
        # Serialize Python Dictionaries into DynamoDB notation.
        ddb_data = boto3.dynamodb.types.TypeSerializer().serialize(i)["M"]
        ddb_keys = boto3.dynamodb.types.TypeSerializer().serialize(ddb_keys)["M"]
        # The record must contain "Keys" and "NewImage" attributes to be similar
        # to a DynamoDB Streams record. Additionally, you inject the name of
        # the source DynamoDB table in the record so you can use it as an index
        # for Amazon ES.
        record = {"Keys": ddb_keys, "NewImage": ddb_data, "SourceTable": ddb_table_name}
        # Convert the record to JSON.
        record = json.dumps(record)
        # Push the record to Amazon Kinesis.
        res = kinesis.put_record(
            StreamName=ks_stream_name,
            Data=record,
            PartitionKey=i["product_id"])
        print(res)

    # Stop the loop if no additional records are
    # available.
    if 'LastEvaluatedKey' not in response:
        break

Note: In the code example above, you are passing the name of the source DynamoDB table as an extra record attribute SourceTable. The Lambda function uses that attribute to build the Amazon ES index name. Another approach for passing that information is tagging the Amazon Kinesis stream.

Now, create the Amazon Kinesis stream ddb-all-productsand then add permissions to the ddb-elasticsearch-bridge role in IAM to allow the Lambda function to read from the stream:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:Get*",
                "kinesis:DescribeStream"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-1:0123456789:stream/ddb-all-products"
            ]
        }
    ]
}

Finally, set the Amazon Kinesis stream as an additional input source to the Lambda function:

Amazon Kinesis input source

Neat tip: Doing a full re-index of the content this way will not create duplicate entries in Amazon ES.

Paying attention to attribute types

With DynamoDB, you can use different types for the same attribute on different records, but Amazon ES expects a given attribute to be of only one type. Similarly, changing the type of an existing attribute after it has been indexed in Amazon ES causes problems and some searches won’t work as expected.

In these cases, you must rebuild the Amazon ES index. For more information, see Reindexing Your Data in the Elasticsearch documentation.

Conclusion

In this post, you have seen how you can use AWS Lambda with DynamoDB to index your table content in Amazon ES as changes happen.

Because you are relying entirely on Lambda for the business logic, you don’t have to deal with servers at any point: everything is managed by the AWS platform in a highly available and scalable fashion. To learn more about Lambda and serverless infrastructures, see the Microservices without the Servers blog post.

Now that you have added full-text search to your DynamoDB table, you might be interested in exposing its content through a small REST API. For more information, see Using Amazon API Gateway as a proxy for DynamoDB.

Building a Dynamic DNS for Route 53 using CloudWatch Events and Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/building-a-dynamic-dns-for-route-53-using-cloudwatch-events-and-lambda/

Jeremy Cowan, AWS Solutions Architect
Jeremy Cowan, AWS Solutions Architect

Efrain Fuentes, Enterprise Solutions Architect
Efrain Fuentes, Enterprise Solutions Architect

Introduction
 
Dynamic registration of resource records is useful when you have instances that are not behind a load balancer that you would like to address by a host name and domain suffix of your choosing, rather than the default <region>.compute.internal or ec2.internal.
In this post, we explore how you can use CloudWatch Events and Lambda to create a Dynamic DNS for Route 53. Besides creating A records, this solution allows you to create alias, i.e. CNAME records, for when you want to address a server by a “friendly” or alternate name. Although this is antithetical to treating instances as disposable resources, there are still a lot of shops that find this useful.
 
Using CloudWatch and Lambda to respond to infrastructure changes in real-time
 
With the advent of CloudWatch Events in January 2016, you can now get near real-time information when an AWS resource changes its state, including when instances are launched or terminated.  When you combine this with the power of Amazon Route 53 and AWS Lambda, you can create a system that closely mimics the behavior of Dynamic DNS.
For example, when a newly-launched instance changes its state from pending to running, an event can be sent to a Lambda function that creates a resource record in the appropriate Route 53 hosted zone.  Similarly, when instances are stopped or terminated, Lambda can automatically remove resource records from Route 53.
The example provided in this post works precisely this way.  It uses information from a CloudWatch event to gather information about the instance, such as its public and private DNS name, its public and private IP address, the VPC ID of the VPC that the instance was launched in, its tags, and so on.  It then uses this information to create A, PTR, and CNAME records in the appropriate Route 53 public or private hosted zone.  The solution persists data about the instances in an Amazon DynamoDB table so it can remove resource records when instances are stopped or terminated.
 
Route 53 Hosted zones
 
Route 53 offers the convenience of domain name services without having to build a globally distributed highly reliable DNS infrastructure.  It allows instances within your VPC to resolve the names of resources that run within your AWS environment.  It also lets clients on the Internet resolve names of your public-facing resources.  This is accomplished by querying resource record sets that reside within a Route 53 public or private hosted zone.
A private hosted zone is basically a container that holds information about how you want to route traffic for a domain and its subdomains within one or more VPCs, whereas a public hosted zone is a container that holds information about how you want to route traffic from the Internet.
 
Choosing between VPC DNS or Route 53 Private Hosted Zones
 
Admittedly, you can use VPC DNS for internal name resolution instead of Route 53 private hosted zones.  Although it doesn’t dynamically create resource records, VPC DNS will provide name resolution for all the hosts within a VPC’s CIDR range.
Unless you create a DHCP option set with a custom domain name and disable hostnames at the VPC, you can’t change the domain suffix; all instances are either assigned the ec2.internal or <region>.compute.internal domain suffix.  You can’t create aliases or other resource record types with VPC DNS either.
Private hosted zones help you overcome these challenges by allowing you to create different resource record types with a custom domain suffix.  Moreover, with Route 53 you can create a subdomain for your current DNS namespace or you can migrate an existing subdomain to Route 53.  By using these options, you can create a contiguous DNS namespace between your on-premises environment and AWS.
So, while VPC DNS can provide basic name resolution for your VPC, Route 53 private hosted zones offer richer functionality by comparison.  It also has a programmable API that can be used to automate the creation/removal of records sets and hosted zones which we’re going leverage later in this post.
Route 53 doesn’t offer support for dynamic registration of resource record sets for public or private hosted zones.  This can pose challenges when an Auto Scaling event occurs and the instances are not behind a load balancer.  A common workaround is to use an automation framework like Chef, Puppet, Ansible, or Salt to create resource records, or by adding instance user data to the launch profile of the Auto Scaling group.  The drawbacks to these approaches are that:
1)   automation frameworks typically require you to manage additional infrastructure.
2)   instance user data doesn’t handle the removal of resource records when the instance is terminated.
This was the motivation for creating a serverless architecture that dynamically creates and removes resource records from Route 53 as EC2 instances are created and destroyed.
 
DDNS/Lambda example
 
Make sure that you have the latest version of the AWS CLI installed locally.   For more information, see Getting Set Up with the AWS Command Line Interface.
For this example, create a new VPC configured with a private and public subnet, using Scenario 2: VPC with Public and Private Subnets (NAT) from the Amazon VPC User Guide.  Ensure that the VPC has the DNS resolution and DNS hostnames options set to yes.
After the VPC is created, you can proceed to the next steps.
 
Step 1 – Create an IAM role for the Lambda function
 
In this step, you use the AWS Command Line Interface (AWS CLI) to create the Identity and Access Management (IAM) role that the Lambda function assumes when the function is invoked.  You need to create an IAM policy with the required permissions and then attach this policy to the role.
Download the ddns-policy.json and ddns-trust.json files from the AWS Labs GitHub repo.
 
ddns-policy.json
 
The policy includes ec2:Describe permission, required for the function to obtain the EC2 instance’s attributes, including the private IP address, public IP address, and DNS hostname.   The policy also includes DynamoDB and Route 53 full access, required for the function to create the DynamoDB table and to update the Route 53 DNS records.  The policy also allows the function to create log groups and log events.
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ec2:Describe*",
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "dynamodb:*"
    ],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "route53:*"
    ],
    "Resource": [
      "*"
    ]
  }]
}

ddns-trust.json
 
The ddns-trust.json file contains the trust policy that grants the Lambda service permission to assume the role.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create the policy using the policy document in the ddns-pol.json file.  You need to replace <LOCAL PATH> with your local path to the ddns-pol.json file.   The output of the aws iam create-policy command includes the Amazon Resource Locator (ARN).  Save the ARN, since you will need it for future steps.
aws iam create-policy –policy-name ddns-lambda-policy –policy-document file://<LOCAL PATH>/ddns-pol.json
Create the ddns-lambda-role IAM role using the trust policy in the ddns-trust.json file.  You need to replace <LOCAL PATH> with your local path to the ddns-trust.json file.  The output of the aws iam create-role command includes the ARN associated with the role that you created.  Save this ARN,  since you will need it when you create the Lambda function in the next section.
aws iam create-role –role-name ddns-lambda-role –assume-role-policy-document file://<LOCAL PATH>/ddns-trust.json
Attach the policy to the role.  Use the ARN returned in step 2 for the –policy-arn input parameter.
aws iam attach-role-policy –role-name ddns-lambda-role –policy-arn <enter-your-policy-arn-here>
Step 2 – Create the Lambda function
 
The Lambda function uses modules included in the Python 2.7 Standard Library and the AWS SDK for Python module (boto3), which is preinstalled as part of the Lambda service.  As such, you do not need to create a deployment package for this example.
The function code performs the following:

Checks to see whether the “DDNS” table exists in DynamoDB and creates the table if it does not. This table is used to keep a record of instances that have been created along with their attributes. It’s necessary to persist the instance attributes in a table because once an EC2 instance is terminated, its attributes are no longer available to be queried via the EC2 API. Instead, they must be fetched from the table.

Queries the event data to determine the instance’s state. If the state is “running”, the function queries the EC2 API for the data it will need to update DNS. If the state is anything else, e.g. “stopped” or “terminated”, it will retrieve the necessary information from the “DDNS” DynamoDB table.

Verifies that “DNS resolution” and “DNS hostnames” are enabled for the VPC, as these are required in order to use Route 53 for private name resolution. The function then checks whether a reverse lookup zone for the instance already exists. If it does, it checks to see whether the reverse lookup zone is associated with the instance’s VPC. If it isn’t, it creates the association. This association is necessary in order for the VPC to use Route 53 zone for private name resolution.

Checks the EC2 instance’s tags for the CNAME and ZONE tags. If the ZONE tag is found, the function creates A and PTR records in the specified zone. If the CNAME tag is found, the function creates a CNAME record in the specified zone.

Verifies whether there’s a DHCP option set assigned to the VPC. If there is, it uses the value of the domain name to create resource records in the appropriate Route 53 private hosted zone. The function also checks to see whether there’s an association between the instance’s VPC and the private hosted zone. If there isn’t, it creates it.

Deletes the required DNS resource records if the state of the EC2 instance changes to “shutting down” or “stopped”.

 
Use the AWS CLI to create the Lambda function:

Download the union.py.zip file from the AWS Labs GitHub repo.
Execute the following command to create the function.  Note that you need to update the command to use the ARN of the role that you created earlier, as well as the local path to the union.py.zip file containing the Python code for the Lambda function.

aws lambda create-function –function-name ddns_lambda –runtime python2.7 –role <enter-your-role-arn-here> –handler union.lambda_handler –timeout 30 –zip-file fileb://<LOCAL PATH>/union.py.zip

The output of the command returns the ARN of the newly-created function. Save this ARN, as you will need it in the next section

 
Step 3 – Create the CloudWatch Events Rule
 
In this step, you create the CloudWatch Events rule that triggers the Lambda function whenever CloudWatch detects a change to the state of an EC2 instance.  You configure the rule to fire when any EC2 instance state changes to “running”, “shutting down”, or “stopped”.  Use the aws events put-rule command to create the rule and set the Lambda function as the execution target:
aws events put-rule –event-pattern "{"source":["aws.ec2"],"detail-type":["EC2 Instance State-change Notification"],"detail":{"state":["running","shutting-down","stopped"]}}" –state ENABLED –name ec2_lambda_ddns_rule
The output of the command returns the ARN to the newly created CloudWatch Events rule, named ec2_lambda_ddns_rule. Save the ARN, as you will need it to associate the rule with the Lambda function and to set the appropriate Lambda permissions.
Next, set the target of the rule to be the Lambda function.  Note that the —targets input parameter requires that you include a unique identifier for the Id target.  You also need to update the command to use the ARN of the Lambda function that you created previously.
aws events put-targets –rule ec2_lambda_ddns_rule –targets Id=id123456789012,Arn=<enter-your-lambda-function-arn-here>
Next, you add the permissions required for the CloudWatch Events rule to execute the Lambda function.   Note that you need to provide a unique value for the –statement-id input parameter.  You also need to provide the ARN of the CloudWatch Events rule you created earlier.
aws lambda add-permission –function-name ddns_lambda –statement-id 45 –action lambda:InvokeFunction –principal events.amazonaws.com –source-arn <enter-your-cloudwatch-events-rule-arn-here>
Step 4 – Create the private hosted zone in Route 53
 
To create the private hosted zone in Route 53, follow the steps outlined in Creating a Private Hosted Zone.
 
Step 5 – Create a DHCP options set and associate it with the VPC
 
In this step, you create a new DHCP options set and set the domain to be that of your private hosted zone.

Follow the steps outlined in Creating a DHCP Options Set to create a new set of DHCP options.
In the Create DHCP options set dialog box, give the new options set a name, set Domain name to the name of the private hosted zone that you created in Route 53, and set Domain name servers to “AmazonProvidedDNS”.  Choose Yes, Create.DHCP Options Set
Next, follow the steps outlined in Changing the Set of DHCP Options a VPC Uses to update the VPC to use the newly-created DHCP options set.

Step 6 – Launching the EC2 instance and validating results
 
In this step, you launch an EC2 instance and verify that the function executed successfully.
As mentioned previously, the Lambda function looks for the ZONE or CNAME tags associated with the EC2 instance.  If you specify these tags when you launch the instance, you should include the trailing dot.  In this example, the ZONE tag would be set to “ddnslambda.com.” and the CNAME tag could be set to “test.ddnslambda.com.”.
Because you updated the DHCP options set in this example, the Lambda function uses the specified zone when it creates the Route 53 DNS resource records.  You can use the ZONE tag to override this behavior if you want the function to update a different hosted zone.
In this example, you launch an EC2 instance into the private subnet of the VPC.  Because you updated the domain value of the DHCP options set to be that of the private hosted zone, the Lambda function creates the DNS resource records in the Route 53 zone file.
 
Launching the EC2 instance
 

Follow the steps to launch an EC2 instance outlined in Launching an Instance.
In Step 3: Configure Instance Details, for Network, select the VPC.  For Subnet, select the private subnet.  Choose Review and Launch.
(Optional) If you would like to update a different private hosted zone than the one you associated with the VPC, specify the ZONE tag in this step.  You can also specify the CNAME tag if you would like the function to create a CNAME resource record in the associated zone.

Choose Edit tags in the Step 7: Review Instance Launch.
Enter the key and value for Step 5: Tag Instance then choose Review and Launch.
Tag Instance
Complete the launch of the instance and wait until the instance state changes to “running”.  Then, continue to the next step.
 
Validating results
 
In this step, you verify that your Lambda function successfully updated the Route 53 resource records.

Log in to the Route 53 console.
In the left navigation pane, choose Hosted Zones to view the list of private and public zones currently configured in Route 53.
Select the hosted zone that you created in step 4 to view the zone file.Verify Hosted Zone
Verify that the resource records were created.Verify Resource Records
Now that you’ve verified that the Lambda function successfully updated the Route 53 resource records in the zone file, stop the EC2 instance and verify that the records are removed by the function.
Log in to the EC2 console.
Choose Instances in the left navigation pane.Select Instance
Select the EC2 instance you launched earlier and choose Stop.Stop Instance
Follow Steps 1 – 3 to view the DNS resource records in the Route 53 zone.
Verify that the records have been removed from the zone file by the Lambda function.

 
Conclusion
 
Now that you’ve seen how you can combine various AWS services to automate the creation and removal of Route 53 resource records, we hope you are inspired to create your own solutions.  CloudWatch Events is a powerful tool because it allows you to respond to events in real-time, such as when an instance changes its state.  When used with Lambda, you can create highly scalable serverless infrastructures that react instantly to infrastructure changes.
To learn more about CloudWatch Events, see Using CloudWatch Events in the Amazon CloudWatch Developer Guide.  To learn more about Lambda and serverless infrastructures, see the AWS Lambda Developer Guide and the “Microservices without the Servers” blog post.
We’ve open-sourced the code in this example in the AWS Labs GitHub repo and can’t wait to see your feedback and your ideas about how to improve the solution

Building a Dynamic DNS for Route 53 using CloudWatch Events and Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/building-a-dynamic-dns-for-route-53-using-cloudwatch-events-and-lambda/

Jeremy Cowan, AWS Solutions Architect
Jeremy Cowan, AWS Solutions Architect

Efrain Fuentes, Enterprise Solutions Architect
Efrain Fuentes, Enterprise Solutions Architect

Introduction
 
Dynamic registration of resource records is useful when you have instances that are not behind a load balancer that you would like to address by a host name and domain suffix of your choosing, rather than the default <region>.compute.internal or ec2.internal.
In this post, we explore how you can use CloudWatch Events and Lambda to create a Dynamic DNS for Route 53. Besides creating A records, this solution allows you to create alias, i.e. CNAME records, for when you want to address a server by a “friendly” or alternate name. Although this is antithetical to treating instances as disposable resources, there are still a lot of shops that find this useful.
 
Using CloudWatch and Lambda to respond to infrastructure changes in real-time
 
With the advent of CloudWatch Events in January 2016, you can now get near real-time information when an AWS resource changes its state, including when instances are launched or terminated.  When you combine this with the power of Amazon Route 53 and AWS Lambda, you can create a system that closely mimics the behavior of Dynamic DNS.
For example, when a newly-launched instance changes its state from pending to running, an event can be sent to a Lambda function that creates a resource record in the appropriate Route 53 hosted zone.  Similarly, when instances are stopped or terminated, Lambda can automatically remove resource records from Route 53.
The example provided in this post works precisely this way.  It uses information from a CloudWatch event to gather information about the instance, such as its public and private DNS name, its public and private IP address, the VPC ID of the VPC that the instance was launched in, its tags, and so on.  It then uses this information to create A, PTR, and CNAME records in the appropriate Route 53 public or private hosted zone.  The solution persists data about the instances in an Amazon DynamoDB table so it can remove resource records when instances are stopped or terminated.
 
Route 53 Hosted zones
 
Route 53 offers the convenience of domain name services without having to build a globally distributed highly reliable DNS infrastructure.  It allows instances within your VPC to resolve the names of resources that run within your AWS environment.  It also lets clients on the Internet resolve names of your public-facing resources.  This is accomplished by querying resource record sets that reside within a Route 53 public or private hosted zone.
A private hosted zone is basically a container that holds information about how you want to route traffic for a domain and its subdomains within one or more VPCs, whereas a public hosted zone is a container that holds information about how you want to route traffic from the Internet.
 
Choosing between VPC DNS or Route 53 Private Hosted Zones
 
Admittedly, you can use VPC DNS for internal name resolution instead of Route 53 private hosted zones.  Although it doesn’t dynamically create resource records, VPC DNS will provide name resolution for all the hosts within a VPC’s CIDR range.
Unless you create a DHCP option set with a custom domain name and disable hostnames at the VPC, you can’t change the domain suffix; all instances are either assigned the ec2.internal or <region>.compute.internal domain suffix.  You can’t create aliases or other resource record types with VPC DNS either.
Private hosted zones help you overcome these challenges by allowing you to create different resource record types with a custom domain suffix.  Moreover, with Route 53 you can create a subdomain for your current DNS namespace or you can migrate an existing subdomain to Route 53.  By using these options, you can create a contiguous DNS namespace between your on-premises environment and AWS.
So, while VPC DNS can provide basic name resolution for your VPC, Route 53 private hosted zones offer richer functionality by comparison.  It also has a programmable API that can be used to automate the creation/removal of records sets and hosted zones which we’re going leverage later in this post.
Route 53 doesn’t offer support for dynamic registration of resource record sets for public or private hosted zones.  This can pose challenges when an Auto Scaling event occurs and the instances are not behind a load balancer.  A common workaround is to use an automation framework like Chef, Puppet, Ansible, or Salt to create resource records, or by adding instance user data to the launch profile of the Auto Scaling group.  The drawbacks to these approaches are that:
1)   automation frameworks typically require you to manage additional infrastructure.
2)   instance user data doesn’t handle the removal of resource records when the instance is terminated.
This was the motivation for creating a serverless architecture that dynamically creates and removes resource records from Route 53 as EC2 instances are created and destroyed.
 
DDNS/Lambda example
 
Make sure that you have the latest version of the AWS CLI installed locally.   For more information, see Getting Set Up with the AWS Command Line Interface.
For this example, create a new VPC configured with a private and public subnet, using Scenario 2: VPC with Public and Private Subnets (NAT) from the Amazon VPC User Guide.  Ensure that the VPC has the DNS resolution and DNS hostnames options set to yes.
After the VPC is created, you can proceed to the next steps.
 
Step 1 – Create an IAM role for the Lambda function
 
In this step, you use the AWS Command Line Interface (AWS CLI) to create the Identity and Access Management (IAM) role that the Lambda function assumes when the function is invoked.  You need to create an IAM policy with the required permissions and then attach this policy to the role.
Download the ddns-policy.json and ddns-trust.json files from the AWS Labs GitHub repo.
 
ddns-policy.json
 
The policy includes ec2:Describe permission, required for the function to obtain the EC2 instance’s attributes, including the private IP address, public IP address, and DNS hostname.   The policy also includes DynamoDB and Route 53 full access, required for the function to create the DynamoDB table and to update the Route 53 DNS records.  The policy also allows the function to create log groups and log events.
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ec2:Describe*",
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "dynamodb:*"
    ],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "route53:*"
    ],
    "Resource": [
      "*"
    ]
  }]
}

ddns-trust.json
 
The ddns-trust.json file contains the trust policy that grants the Lambda service permission to assume the role.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create the policy using the policy document in the ddns-pol.json file.  You need to replace <LOCAL PATH> with your local path to the ddns-pol.json file.   The output of the aws iam create-policy command includes the Amazon Resource Locator (ARN).  Save the ARN, since you will need it for future steps.
aws iam create-policy –policy-name ddns-lambda-policy –policy-document file://<LOCAL PATH>/ddns-pol.json
Create the ddns-lambda-role IAM role using the trust policy in the ddns-trust.json file.  You need to replace <LOCAL PATH> with your local path to the ddns-trust.json file.  The output of the aws iam create-role command includes the ARN associated with the role that you created.  Save this ARN,  since you will need it when you create the Lambda function in the next section.
aws iam create-role –role-name ddns-lambda-role –assume-role-policy-document file://<LOCAL PATH>/ddns-trust.json
Attach the policy to the role.  Use the ARN returned in step 2 for the –policy-arn input parameter.
aws iam attach-role-policy –role-name ddns-lambda-role –policy-arn <enter-your-policy-arn-here>
Step 2 – Create the Lambda function
 
The Lambda function uses modules included in the Python 2.7 Standard Library and the AWS SDK for Python module (boto3), which is preinstalled as part of the Lambda service.  As such, you do not need to create a deployment package for this example.
The function code performs the following:

Checks to see whether the “DDNS” table exists in DynamoDB and creates the table if it does not. This table is used to keep a record of instances that have been created along with their attributes. It’s necessary to persist the instance attributes in a table because once an EC2 instance is terminated, its attributes are no longer available to be queried via the EC2 API. Instead, they must be fetched from the table.

Queries the event data to determine the instance’s state. If the state is “running”, the function queries the EC2 API for the data it will need to update DNS. If the state is anything else, e.g. “stopped” or “terminated”, it will retrieve the necessary information from the “DDNS” DynamoDB table.

Verifies that “DNS resolution” and “DNS hostnames” are enabled for the VPC, as these are required in order to use Route 53 for private name resolution. The function then checks whether a reverse lookup zone for the instance already exists. If it does, it checks to see whether the reverse lookup zone is associated with the instance’s VPC. If it isn’t, it creates the association. This association is necessary in order for the VPC to use Route 53 zone for private name resolution.

Checks the EC2 instance’s tags for the CNAME and ZONE tags. If the ZONE tag is found, the function creates A and PTR records in the specified zone. If the CNAME tag is found, the function creates a CNAME record in the specified zone.

Verifies whether there’s a DHCP option set assigned to the VPC. If there is, it uses the value of the domain name to create resource records in the appropriate Route 53 private hosted zone. The function also checks to see whether there’s an association between the instance’s VPC and the private hosted zone. If there isn’t, it creates it.

Deletes the required DNS resource records if the state of the EC2 instance changes to “shutting down” or “stopped”.

 
Use the AWS CLI to create the Lambda function:

Download the union.py.zip file from the AWS Labs GitHub repo.
Execute the following command to create the function.  Note that you need to update the command to use the ARN of the role that you created earlier, as well as the local path to the union.py.zip file containing the Python code for the Lambda function.

aws lambda create-function –function-name ddns_lambda –runtime python2.7 –role <enter-your-role-arn-here> –handler union.lambda_handler –timeout 30 –zip-file fileb://<LOCAL PATH>/union.py.zip

The output of the command returns the ARN of the newly-created function. Save this ARN, as you will need it in the next section

 
Step 3 – Create the CloudWatch Events Rule
 
In this step, you create the CloudWatch Events rule that triggers the Lambda function whenever CloudWatch detects a change to the state of an EC2 instance.  You configure the rule to fire when any EC2 instance state changes to “running”, “shutting down”, or “stopped”.  Use the aws events put-rule command to create the rule and set the Lambda function as the execution target:
aws events put-rule –event-pattern "{"source":["aws.ec2"],"detail-type":["EC2 Instance State-change Notification"],"detail":{"state":["running","shutting-down","stopped"]}}" –state ENABLED –name ec2_lambda_ddns_rule
The output of the command returns the ARN to the newly created CloudWatch Events rule, named ec2_lambda_ddns_rule. Save the ARN, as you will need it to associate the rule with the Lambda function and to set the appropriate Lambda permissions.
Next, set the target of the rule to be the Lambda function.  Note that the —targets input parameter requires that you include a unique identifier for the Id target.  You also need to update the command to use the ARN of the Lambda function that you created previously.
aws events put-targets –rule ec2_lambda_ddns_rule –targets Id=id123456789012,Arn=<enter-your-lambda-function-arn-here>
Next, you add the permissions required for the CloudWatch Events rule to execute the Lambda function.   Note that you need to provide a unique value for the –statement-id input parameter.  You also need to provide the ARN of the CloudWatch Events rule you created earlier.
aws lambda add-permission –function-name ddns_lambda –statement-id 45 –action lambda:InvokeFunction –principal events.amazonaws.com –source-arn <enter-your-cloudwatch-events-rule-arn-here>
Step 4 – Create the private hosted zone in Route 53
 
To create the private hosted zone in Route 53, follow the steps outlined in Creating a Private Hosted Zone.
 
Step 5 – Create a DHCP options set and associate it with the VPC
 
In this step, you create a new DHCP options set and set the domain to be that of your private hosted zone.

Follow the steps outlined in Creating a DHCP Options Set to create a new set of DHCP options.
In the Create DHCP options set dialog box, give the new options set a name, set Domain name to the name of the private hosted zone that you created in Route 53, and set Domain name servers to “AmazonProvidedDNS”.  Choose Yes, Create.DHCP Options Set
Next, follow the steps outlined in Changing the Set of DHCP Options a VPC Uses to update the VPC to use the newly-created DHCP options set.

Step 6 – Launching the EC2 instance and validating results
 
In this step, you launch an EC2 instance and verify that the function executed successfully.
As mentioned previously, the Lambda function looks for the ZONE or CNAME tags associated with the EC2 instance.  If you specify these tags when you launch the instance, you should include the trailing dot.  In this example, the ZONE tag would be set to “ddnslambda.com.” and the CNAME tag could be set to “test.ddnslambda.com.”.
Because you updated the DHCP options set in this example, the Lambda function uses the specified zone when it creates the Route 53 DNS resource records.  You can use the ZONE tag to override this behavior if you want the function to update a different hosted zone.
In this example, you launch an EC2 instance into the private subnet of the VPC.  Because you updated the domain value of the DHCP options set to be that of the private hosted zone, the Lambda function creates the DNS resource records in the Route 53 zone file.
 
Launching the EC2 instance
 

Follow the steps to launch an EC2 instance outlined in Launching an Instance.
In Step 3: Configure Instance Details, for Network, select the VPC.  For Subnet, select the private subnet.  Choose Review and Launch.
(Optional) If you would like to update a different private hosted zone than the one you associated with the VPC, specify the ZONE tag in this step.  You can also specify the CNAME tag if you would like the function to create a CNAME resource record in the associated zone.

Choose Edit tags in the Step 7: Review Instance Launch.
Enter the key and value for Step 5: Tag Instance then choose Review and Launch.
Tag Instance
Complete the launch of the instance and wait until the instance state changes to “running”.  Then, continue to the next step.
 
Validating results
 
In this step, you verify that your Lambda function successfully updated the Route 53 resource records.

Log in to the Route 53 console.
In the left navigation pane, choose Hosted Zones to view the list of private and public zones currently configured in Route 53.
Select the hosted zone that you created in step 4 to view the zone file.Verify Hosted Zone
Verify that the resource records were created.Verify Resource Records
Now that you’ve verified that the Lambda function successfully updated the Route 53 resource records in the zone file, stop the EC2 instance and verify that the records are removed by the function.
Log in to the EC2 console.
Choose Instances in the left navigation pane.Select Instance
Select the EC2 instance you launched earlier and choose Stop.Stop Instance
Follow Steps 1 – 3 to view the DNS resource records in the Route 53 zone.
Verify that the records have been removed from the zone file by the Lambda function.

 
Conclusion
 
Now that you’ve seen how you can combine various AWS services to automate the creation and removal of Route 53 resource records, we hope you are inspired to create your own solutions.  CloudWatch Events is a powerful tool because it allows you to respond to events in real-time, such as when an instance changes its state.  When used with Lambda, you can create highly scalable serverless infrastructures that react instantly to infrastructure changes.
To learn more about CloudWatch Events, see Using CloudWatch Events in the Amazon CloudWatch Developer Guide.  To learn more about Lambda and serverless infrastructures, see the AWS Lambda Developer Guide and the “Microservices without the Servers” blog post.
We’ve open-sourced the code in this example in the AWS Labs GitHub repo and can’t wait to see your feedback and your ideas about how to improve the solution