All posts by Bryan Liston

Announcing C# Support for AWS Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/announcing-c-sharp-support-for-aws-lambda/

Today, we’re excited to announce C# as a supported language for AWS Lambda! Using the new, open source .NET Core 1.0 runtime, you can easily publish C# code to AWS Lambda from a variety of popular .NET tools. .NET developers can now build Lambda functions and serverless applications with the C# language and .NET tools that they know and love. With tooling support in Visual Studio, Yeoman, and the dotnet CLI, you can easily deploy individual Lambda functions or entire serverless applications written in C# to Lambda and Amazon API Gateway.

Lambda is the core of the AWS serverless platform. Originally launched in 2015, Lambda enables customers to deploy Node.js, Python, and Java code to AWS without needing to worry about infrastructure or scaling. This allows developers to focus on the business logic for their application and not spend time maintaining and scaling infrastructure. Until today, .NET developers were not able to take advantage of this model. We’re excited to add C# to the list of supported languages and enable a new category of developers to take advantage of Lambda and API Gateway to create serverless applications.

C# in Lambda

Look at a simple C# Lambda function. If you’ve already used Lambda with Node.js, Python, or Java, this should look familiar:

using System;
using System.IO;
using System.Text;
using Amazon.Lambda.Core;
using Amazon.Lambda.DynamoDBEvents;

using Amazon.Lambda.Serialization.Json;
 

namespace DynamoDBStreams
{
    public class DdbSample
    {
        private static readonly JsonSerializer _jsonSerializer = new JsonSerializer();

        public void ProcessDynamoEvent(DynamoDBEvent dynamoEvent)
        {
            Console.WriteLine($"Beginning to process {dynamoEvent.Records.Count} records...");

            foreach (var record in dynamoEvent.Records)
            {
                Console.WriteLine($"Event ID: {record.EventID}");
                Console.WriteLine($"Event Name: {record.EventName}");
 
                string streamRecordJson = SerializeObject(record.Dynamodb);
                Console.WriteLine($"DynamoDB Record:");
                Console.WriteLine(streamRecordJson);
            }

            Console.WriteLine("Stream processing complete.");
        }


        private string SerializeObject(object streamRecord)
        {
            using (var ms = new MemoryStream())
            {
                _jsonSerializer.Serialize(streamRecord, ms);
                return Encoding.UTF8.GetString(ms.ToArray());
            }
        }
    }
}

As you can see, this is straightforward code, but there are a few important details to call out. Unlike other languages supported on Lambda, you don’t need to implement a specific interface to mark your code as a Lambda function. Instead, just provide a handler string when uploading your code to tell Lambda where to start execution.

Similar to other languages supported on Lambda, you have a few choices for handling input and return types in your function. The most basic choice is to use a low-level stream interface of System.IO.Stream. Alternatively you can also apply the default serializer at the assembly or method level of your application, or you can define your own serialization logic by implementing the ILambdaSerializer interface, which is also provided by the Amazon.Lambda.Core library.

Look at the function signature for the ProcessDynamoEvent function and notice DynamoDBEvent in the signature. This comes from the Amazon.Lambda.Core library provided by Lambda, along with many more classes for other AWS event types. You can add a project dependency on this NuGet package to get access to a static Lambda logger, serialization interfaces, and a C# implementation of the Lambda context object.

For logging, you can use the static Write or WriteLine methods provided by the C# Console class, the Log method on the Amazon.Lambda.Core.LambdaLogger class, or the Logger property in the context object. You can get more information about the C# programming model in the AWS Lambda Developer Guide.

AWS Toolkit for Visual Studio

The AWS Toolkit for Visual Studio supports developing, testing, and deploying .NET Core Lambda functions and serverless applications. The toolkit has two new project templates to help you get started:

  • The AWS Lambda Project template creates a simple project with a single C# Lambda function.
  • The AWS Serverless Application template creates a small AWS serverless application, following the AWS Serverless Application Model (AWS SAM). This template shows how to develop a complete serverless application composed of multiple Lambda functions exposed through an API Gateway REST endpoint. Also, AWS SAM allows you to model the AWS resources that your application uses as part of your project’s template.

dotnet_1.png

After your code is ready, you can deploy directly from Visual Studio by right-clicking your project and choosing Publish to AWS Lambda… in the Solution Explorer. From there, the deployment wizard guides you through the deployment process.

dotnet_2.png

Cross-platform development using the .NET Core CLI

One of the great features of .NET Core is cross-platform support. With the traditional .NET framework, developers are required to build and run their applications on Windows. However, .NET Core enables you to develop your C# code on any platform of your choice and deploy it to any platform as well.

If you’re not developing on Windows and don’t have access to the AWS Toolkit for Visual Studio, you can still use .NET tools to easily publish your C# Lambda functions and serverless applications to AWS. Even if you are using the AWS Toolkit for Visual Studio, knowing how to use the dotnet CLI can be helpful in automating your build and deployment process.

After you create a .NET Core project, enable the Lambda tools in the dotnet CLI using tools like Yeoman; just add a tools dependency on the Amazon.Lambda.Tools NuGet package to your new project.

dotnet_3.png

The Amazon.Lambda.Tools NuGet package adds commands to the new dotnet CLI that allow you to deploy your Lambda functions and serverless applications to AWS, no matter what platform you’re on. Even if you are developing in Visual Studio on Windows, the AWS Lambda tools in the dotnet CLI are helpful for setting up a CI/CD pipeline for your application.

To learn more about the new Lambda commands in the dotnet CLI, type dotnet lambda help in your project directory.

dotnet_4.png

Summary

We’re excited to open up AWS Lambda for C# applications through the .NET Core runtime. You can find more information on writing C# Lambda functions in the AWS Lambda Developer Guide. Download the AWS Toolkit for Visual Studio to get started or check out the Lambda extensions to the dotnet CLI.

Monetize your APIs in AWS Marketplace using API Gateway

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/monetize-your-apis-in-aws-marketplace-using-api-gateway/


Shiva Krishnamurthy, Sr. Product Manager

Amazon API Gateway helps you quickly build highly scalable, secure, and robust APIs. Today, we are announcing an integration of API Gateway with AWS Marketplace. You can now easily monetize your APIs built with API Gateway, market them directly to AWS customers, and reuse AWS bill calculation and collection mechanisms.

AWS Marketplace lists over 3,500 software listings across 35 product categories with over 100K active customers. With the recent announcement of SaaS Subscriptions, API sellers can, for the first time, take advantage of the full suite of Marketplace features, including customer acquisition, unified billing, and reporting. For AWS customers, this means that they can now subscribe to API products through AWS Marketplace and pay on an existing AWS bill. This gives you direct access to the AWS customer base.

To get started, identify the API on API Gateway that you want to sell on AWS Marketplace. Next, package that API into usage plans. Usage plans allow you to set throttling limits and quotas to your APIs and allow you to control third-party usage of your API. You can create multiple usage plans with different limits (e.g., Silver, Gold, Platinum) and offer them as different API products on AWS Marketplace.

Let’s suppose that you offer a Pet Store API managed by API Gateway and you want to start selling it through AWS Marketplace: you must offer a developer portal, a website that you must maintain that allows new customers to register an account using AWS-provided billing identifiers. Also, the portal needs to provide registered customers with access to your APIs during and after purchase.

To help you get started, we have created a reference implementation of a developer portal application. You can use this implementation to create a developer portal from scratch, or use it as a reference guide to integrate API Gateway into an existing developer portal that you already operate. For a detailed walkthrough on setting up a developer portal using our reference application, see (Generate Your Own API Gateway Developer Portal).

After you have set up your developer portal, register as a seller with the AWS Marketplace. After registration, submit a product load form to list your product for sale. In this step, you describe your API product, define pricing, and submit AWS account IDs to be used to test the subscription. You also submit the URL of your developer portal. Currently, API Gateway only supports “per request” pricing models.

After you have registered as a seller, you are given an AWS Marketplace product code. Log in to the API Gateway console to associate this product code with the corresponding usage plan on API Gateway. This tells API Gateway to send telemetry records to AWS Marketplace when your API is used.

marketplaceintegration_1.png

After the product code is associated, test the “end user” flow by subscribing to your API products using the AWS IDs that you submitted via the Marketplace; verify the proper functionality. When you’ve finished verifying, submit your product for final approval using instructions provided in the Seller Guide.

Visit here to learn more about this feature.

Generate Your Own API Gateway Developer Portal

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/generate-your-own-api-gateway-developer-portal/


Shiva Krishnamurthy, Sr. Product Manager

Amazon API Gateway helps you quickly build highly scalable, secure, and robust APIs. Developers who want to consume your API to build web, mobile, or other types of apps need a site where they can learn about the API, acquire access, and manage their consumption. You can do this by hosting a developer portal: a web application that lists your APIs in catalog form, displays API documentation to help developers understand your API, and allows them to sign up for access. The application also needs to let developers test your APIs, and provide feedback to grow a successful developer ecosystem. Finally, you may also want to monetize your APIs and grow API product revenue.

Today, we published aws-api-gateway-developer-portal, an open source serverless web application that you can use to get started building your own developer portal. In just a few easy steps, you can generate a serverless web application that lists your APIs on API Gateway in catalog form, and allows for developer signups. The application is built using aws-serverless-express, an open source library that we published recently, that makes it incredibly easy to build serverless web applications using API Gateway and AWS Lambda. The application uses a SAM (Serverless Application Model) template to deploy its serverless resources, and is very easy to deploy and operate.

Walkthrough

In this post, I walk through the steps for creating a sample API and developer portal for third-party developer consumption:

  • Create an API or import a Swagger definition
  • Deploy the API for developer access
  • Document the API for easier developer consumption
  • Structure it into a usage plan to set access control policies, and set throttling limits.
  • Use aws-api-gateway-developer-portal to generate a developer portal, and then set up and configure the portal
  • Log in as a developer and verify the process of consuming an API.

You can also see how to create a AWS Marketplace listing and monetize your API.

Create an API

To get started, log in to the Amazon API Gateway console and import the example API (PetStore). You can also create your own APIs using Swagger, the console, or APIs.

Deploy the API

After creation, deploy the API to a stage to make it accessible to developers, and ready for testing. For this example, use “Marketplace” as the name for the API stage.

Document the API

API Gateway recently announced support for API documentation. You can now document various parts of your API by either importing Swagger or using the API Gateway console to edit API documentation directly.

Navigate to the Documentation tab to add missing documentation and make it easier for developers to understand your API. After you are satisfied with the new documentation, choose Publish Documentation and choose the Marketplace stage created earlier to update the previously deployed PetStore API with the modified documentation.

Package the API into usage plans

Usage plans in API Gateway help you structure your API for developer access. You can associate API keys to usage plans, and set throttling limits and quotas on a per-API key basis. Choose Usage Plans in the console, create a new usage plan, and set throttling limits and quotas as shown below.

When your customers subscribe to this usage plan, their requests are throttled at 200 RPS, and they can each make only 200,000 requests per month. You can change these limits at any time.

Choose Next to create the usage plan. (Skip the API Key screen, and add the Marketplace stage we created earlier)

Generate and configure a developer portal

Now, generate a developer portal in order to list the usage plan that you created earlier. To do this, clone the aws-api-gateway-developer-portal into a local folder. The README includes instructions on how to get set up, but let’s run through it step-by-step. First, ensure that you have the latest version of Node.js installed. For this walkthrough, you use the latest version of the AWS CLI, so make sure that you have your AWS credentials handy. and that you have configured the AWS CLI with the access-key and secret-key using the `aws –configure’ command.

Set up the developer portal

Then, open up a terminal window and run `npm run setup’. This step modifies several project files in-line with the information that you supply—choose a name for the S3 bucket that store webpages for the developer portal, as well as a name for the Lambda function that wraps the Express.js web application. The S3 bucket names that you specify in this step must be region-unique, so use a prefix (eg. ‘my-org-dev-portal-artifacts’).

After entering those details, it then creates the artifacts bucket on S3 (if the S3 bucket name you provided in the previous step doesn’t already exist). It also runs “package and deploy” on the SAM template using the new

command. It takes several minutes for CloudFormation to create the stack.

This step creates all the resources that you need for the developer portal, after setting up the IAM roles that are needed for the operation of this application. Navigate to the AWS CloudFormation console, and choose the Resources tab to see the full list of resources that have been created.

The core components that were created for your portal are listed below:

  • API Gateway API: Proxies HTTP requests to the back-end Lambda function (Express.js web server that serves website content)
  • Developer portal S3 website URL: Runs the developer portal site (publically accessible bucket)
  • Lambda function: Acts as the primary back end or API server
  • Cognito user and identity pools: Creates Cognito user pool and identity pool, and links them.
  • DynamoDB table – Stores a map of API key IDs to customer IDs, which lets you track the API key and customer association.
  • Lambda function Acts as a listener that subscribes to a particular SNS topic, which is useful when customers cancel or subscribe to an API.

Configure the developer portal post-setup

Additional files are then modified with values from your new resources, such as Cognito pool IDs, etc., and your web app is then uploaded to Amazon S3. After this step completes, it opens up your developer portal website in your default browser.

At this point, you have a complete serverless website ready. The application has been integrated with Cognito User Pools, which makes it easy for you to sign up developers and manage their access. For more information, see the Amazon Cognito Your User Pools – Now Generally Available post.

Edit the dev-portal/src/catalog.json file to list your APIs that appear in the developer portal. This can be a subset of all the APIs you have in API Gateway. Replace the contents of this file with your API definitions. For accuracy, export your APIs from API Gateway as Swagger, and copy them into the catalog.json file. Then, run `npm run upload-site’ to make these APIs available in the developer portal.

Verify the process

Test the end-user flow by signing up for access.

After you sign in as a developer, you can subscribe to an API, to receive an API key that you as a developer would use in your API requests.

You can also browse the API documentation that the API owner published (you, as of 5 minutes ago…), and learn more about the API.

You can even test an API with your API Key. Click the “Show API Key” button on the top right corner of the page, and copy your API Key. Next click the red alert icon, enter your API Key, and click Authorize. Finally, click the “Try it out!” button on any of your resources to make a request to your live API.

Your static content for the site is packaged into the folder ‘/dev-portal’. Modify the case studies, and getting started content in this folder; when you’re satisfied, run ‘npm run upload-site’ to push the changes to your portal.

Monetize your APIs

Amazon API Gateway now integrates with the AWS Marketplace to help API owners monetize their API and meter usage for their API products, without writing any code. Now, any API built using API Gateway can easily be published on the AWS Marketplace using the API Gateway console, AWS CLI, or AWS SDK.

API Gateway automatically and accurately meters API consumption for any such API published to the AWS Marketplace and send it to the AWS Marketplace Metering Service, enabling sellers to focus on adding value to their API products rather than building integrations with AWS Marketplace. For more information, see the Monetize your APIs in AWS Marketplace using API Gateway post.

To leverage this feature, you must offer a developer portal that accepts signups from AWS customers. You can use this developer portal implementation to either build your own from scratch, or use it to add functionality to your existing site.

Conclusion

To summarize, you started with an API on API Gateway and structured it for developer consumption. Then in just a few easy steps, you used aws-api-gateway-developer-portal, to spin up a developer portal on serverless architecture, ready to accept developer signups. This application needs no infrastructure management, scales out-of-the-box, and directly integrates with AWS Marketplace to help you monetize your APIs.

If you have any questions or suggestions, please comment below.

Binary Support for API Integrations with Amazon API Gateway

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/

A year ago, the Microservices without the Servers post showed how Lambda can be used for creating image thumbnails. This required the client to Base64 encode the binary image file before calling the image conversion API as well as Base64 decode the response before it could be rendered. With the recent Amazon API Gateway launch of Binary Support for API Integrations, you can now specify media types that you want API Gateway to treat as binary and send and receive binary data through API endpoints hosted on Amazon API Gateway.

After this feature is configured, you can specify if you would like API Gateway to either pass the Integration Request and Response bodies through, convert them to text (Base64 encoding), or convert them to binary (Base64 decoding). These options are available for HTTP, AWS Service, and HTTP Proxy integrations. In the case of Lambda Function and Lambda Function Proxy Integrations, which currently only support JSON, the request body is always converted to JSON.

In this post, I show how you can use the new binary support in API Gateway to turn this Lambda function into a public API, which you can use to include a binary image file in a POST request and get a thumbnail version of that same image. I also show how you can now use API Gateway and Lambda to create a thumbnail service, which you can use to include a binary image file in a POST request and get a thumbnail version of the same image.

Walkthrough

To get started, log in to the AWS Management Console to set up a Lambda integration, using the image-processing-service blueprint.

Create the Lambda function

In the Lambda console, choose Create a Lambda Function.

In the blueprint filter step, for Select runtime , type in ‘image’ and then choose image-processing-service.

Do not set up a trigger. Choose Next.

In the Configure function step, specify the function name, such as ‘thumbnail’.

In the Lambda function handler and role step, for Role , choose Create new role from template(s), and specify the role name (e.g., ‘myMicroserviceRole’). Finally, choose Next. For more details, see AWS Lambda Permissions Model.

Review your Lambda function configuration and choose Create Function.

You have now successfully created the Lambda function that will create a thumbnail.

Create an API and POST method

In this section, you set up an API Gateway thumbnail API to expose a publically accessible RESTful endpoint.

In the API Gateway console, choose Create API.

For API name , enter ‘Thumbnail’, add a description, and choose Create API.

In the created API, choose Resources , Actions , and Create Method.

To create the method, choose POST and select the checkmark.

To set up the POST method, for Integration type , select Lambda Function , select the appropriate Lambda region, and enter ‘thumbnail’ for Lambda Function. Choose Save.

In the Add Permission to Lambda Function dialog box, choose OK to enable API Gateway to invoke the ‘thumbnail’ Lambda function.

Set up the integration

Now, you are ready to set up the integration. In the main page, open Integration Request.

On the Integration Request page, expand Body Mapping Templates.

For Request body passthrough , choose When there are no templates defined (recommended). For Content-Type , enter "image/png".

Choose Add mapping template and add the following template. The thumbnail Lambda function requires that you pass an operation to execute, in this case "thumbnail", and the image payload "base64Image" you are passing in, which is "$input.body". Review the following JSON and choose Save.

Specify which media types need to be handled as binary. Choose [API name], Binary Support.

Choose Edit , specify the media type (such as "image/png") to be handled as binary, and then choose Save.

Deployment

Now that the API is configured, you need to deploy it. On the thumbnail Resources page, choose Action , Deploy API.

For Deployment stage , select [New Stage], specify a stage name, and then choose Deploy.

A stage has been created for you; you receive the Invoke URL value to be used for your thumbnail API.

Testing

Now, you are ready to test the newly created API. Download your favorite .png image (such as apigateway.png), and issue the following curl command. Update the .png image file name and the Invoke URL value accordingly.

$ curl --request POST -H "Accept: image/png" -H "Content-Type: image/png" --data-binary "@apigateway.png" https://XXXXX.execute-api.us-east-1.amazonaws.com/prod > apigateway-thumb.png

You should now be able to open the created images in your favorite image viewer to confirm that resizing has occurred.

Summary

This is just one example of how you can leverage the new binary capabilities of Binary Support in API Gateway. For more examples, see the API Gateway Payload Encodings topic in the Amazon API Gateway Developer Guide.

If you have questions or suggestions, please comment below.

Powering Mobile Backend Services with AWS Lambda and Amazon API Gateway

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/powering-mobile-backend-services-with-aws-lambda-and-amazon-api-gateway/

Daniel Austin
Solutions Architect

Asif Khan
Solutions Architect

Have you ever wanted to create a mobile REST API quickly and easily to make database calls and manipulate data sources? The Node.js and Amazon DynamoDB tutorial shows how to perform CRUD operations (Create, Read, Update, and Delete) easily on DynamoDB tables using Node.js.

In this post, I extend that concept by adding a REST API that calls an AWS Lambda function from Amazon API Gateway. The API allows you to perform the same operations on DynamoDB, from any HTTP-enabled device, such as a browser or mobile phone. The client device doesn’t need to load any libraries, and with serverless architecture and API Gateway, you don’t need to maintain any servers at all!

Walkthrough

In this post, I show you how to write a Lambda function so that it can handle all of the API calls in a single code function, and then add a RESTful API on top of it. API Gateway tells you what function was called.

The problem to solve: how to use API Gateway, AWS Lambda, and DynamoDB to simplify DynamoDB access? Our approach involves using a single Lambda function to provide a CRUD façade on DynamoDB. This required solving two additional problems:

  1. Sending the date from API Gateway about which API method was called, along with POST information about the DynamoDB operations. This is solved by using a generic mapping template for each API call, sending all HTTP data to the Lambda function in a standard JSON format, including the path of the API call, i.e., ‘/movies/add-movie’.

  2. Providing a generic means in Node.js to use multiple function calls and properly use callbacks to send the function results back to the API, again in a standard JSON format. This required writing a generic callback mechanism (a very simple one) that is invoked by each function, and gathers the data for the response.

This is a very cool and easy way to implement basic DynamoDB functions to HTTP(S) calls from API Gateway. It works from any browser or mobile device that understands HTTP.

Mobile developers can write backend code in Java, Node.js, or Python and deploy on Lambda.

In this post, I continue the demonstration with a sample mobile movies database backend, written in Node.js, using DynamoDB. The API is hosted on API Gateway.

Optionally, you can use AWS Mobile Hub to develop and test the mobile client app.

The steps to deploy a mobile backend in Lambda are:

  1. Set up IAM users and roles to allow access to Lambda and DynamoDB.
  2. Download the sample application and edit to include your configuration.
  3. Create a table in DynamoDB using the console or the AWS CLI.Create a new Lambda function and upload the sample app.
  4. Create endpoints in API Gateway
  5. Test the API and Lambda function.

Set up IAM roles to allow access to Lambda and DynamoDB

To set up your API, you need an IAM user and role that has permissions to access DynamoDB and Lambda.

In the IAM console, choose Roles , Create role. Choose AWS Lambda from the list of service roles, then choose AmazonDynamoDBFullAccess and attach another policy, AWSLambdaFullAccess. You need to add this role to an IAM user: you can create a new user for this role, or use an existing one.

Download the sample application and edit to include your configuration

Now download the sample application and edit its configuration file.

The archive can be downloaded from GitHub: https://github.com/awslabs/aws-serverless-crud-sample

git clone https://github.com/awslabs/aws-serverless-crud-sample

After you download the archive, unzip it to an easily-found location and look for the file app_config.json. This file contains set up information for your Lambda function. Edit the file to include your access key ID and secret access key. If you created a new user in step 1, use those credentials. You also need to add your AWS region to the file – this is the region where you will create the DynamoDB table.

Create a table in DynamoDB using the console or the AWS CLI.

To create a table in DynamoDB, use the instructions in the Node.js and DynamoDB tutorial, in the Amazon DynamoDB Getting Started Guide. Next, run the file createMoviesTable.js from the downloaded code in the previous step. You could also use the AWS CLI with this input:

aws DynamoDB create-table  --cli-input-json file://movies-table.json  --region us-west-2  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

The file movies-table.json is in the code archive linked below. If you use the CLI, then the user must have sufficient permissions.

IMPORTANT : The table must be created before completing the rest of the walkthrough.

Create a new Lambda function and upload the sample app.

It can be a little tricky creating the archive for the Lambda function. Make sure you are not zipping the folder, but its contents. This is important; it should look like the following:

This creates a file called "Archive.zip", which is the file to be uploaded to Lambda.

In the Lambda console, choose Create a Lambda function and skip the Blueprints and Configure triggers sections. In the Configure function section, for Name , enter ‘movies-db’. For Runtime , choose ‘Node.js4.3’. For Code entry type, choose ‘Upload a zip file’. Choose Upload and select the archive file that you created in the previous step. For Handler , choose ‘movies-dynamodb.handler’, which is the name of the JavaScript function inside the archive that will be called via Lambda from API Gateway. For Role , choose Choose an existing role and select the role that you created in the first step.

You can leave the other options unchanged and then review and create your Lambda function. You can test the function using the following bit of JSON (this mimics the data that will be sent to the Lambda function from API Gateway):

{
  "method": "POST",
  "body" : { "title": "Godzilla vs. Dynamo", "year": "2016", "info": "New from Acme Films, starring John Smith."},
  "headers": {
      },
  "queryParams": {
      },
  "pathParams": {
      },
  "resourcePath": "/add-movie"
}

Create endpoints in API Gateway

Now, you create five API methods in API Gateway. First, navigate to the API Gateway console and choose Create API , New API. Give the API a name, such as ‘MoviesDP-API’.

In API Gateway, you first create resources and then the methods associated with those resources. The steps for each API call are basically the same:

  1. Create the resource (/movies or /movies/add-movie).
  2. Create a method for the resource – GET for /movies, POST for all others
  3. Choose Integration request , Lambda and select the node-movies Lambda function created earlier. All the API calls use the same Lambda function.
  4. Under Integration request , choose Body mapping templates , create a new template with type application/json , and copy the template file shown below into the form input.
  5. Choose Save.

Use these steps for the following API resources and methods:

  • /movies – lists all movies in the DynamoDB table
  • /movies/add-movie – add an item to DynamoDB
  • /movies/delete-movie – deletes a movie from DynamoDB
  • /movies/findbytitleandyear – finds a movie with a specific title and year
  • /movies/update-movie – modifies and existing movie item in DynamoDB

This body mapping template is a standard JSON template for passing information from API Gateway to Lambda. It provides the calling Lambda function with all of the HTTP input data – including any path variables, query strings, and most importantly for this purpose, the resourcePath variable, which contains the information from API Gateway about which function was called.

Here’s the template:

{
  "method": "$context.httpMethod",
  "body" : $input.json('$'),
  "headers": {
    #foreach($param in $input.params().header.keySet())
    "$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
    #end
  },
  "queryParams": {
    #foreach($param in $input.params().querystring.keySet())
    "$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
    #end
  },
  "pathParams": {
    #foreach($param in $input.params().path.keySet())
    "$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
    #end
  },
  "resourcePath": "$context.resourcePath"
}

Notice the last line where the API Gateway variable $context.resourcePath is sent to the Lambda function as the JSON value of a field called (appropriately enough) resourcePath. This value is used by the Lambda function to perform the required action on the DynamoDB table.

(I originally found this template online, and modified it to add variables like resourcePath. Thanks to Kenn Brodhagen!)

As you create each API method, copy the requestTemplate.vel file from the original code archive and paste it into the Body mapping template field, using application/json as the type. Do this for each API call (using the same file). The file is the same one shown above, but it’s easier to copy the file than cut and paste from a web page!

After the five API calls are created, you are almost done. You need to test your API, and then deploy it before it can be used from the public web.

Test your API and Lambda function

To test the API, add a movie to the DB. The API Gateway calls expect a JSON payload that is sent via POST to your Lambda function. Here’s an example, it’s the same one used above to test the Lambda function:

{ "title": "Love and Friendship", "year": "2016", "info": "New from Amazon Studios, starring Kate Beckinsale."}

To add this movie to your DB, test the /movies/add-movie method, as shown here:

Check logs on the test page of your Lambda function and in API Gateway

[picture10.png]

Conclusion

In this post, I demonstrated a quick way to get started on AWS Lambda for your mobile backend. You uploaded a movie database app which performed CRUD operations on a movies DB table in DynamoDB. The API was hosted on API Gateway for scale and manageability. With the above deployment, you can focus on building your API and mobile clients with serverless architecture. You can reuse the mapping template in future API Gateway projects.

Interested in reading more? See Access Resources in a VPC from Your Lambda Functions and

AWS Mobile Hub – Build, Test, and Monitor Mobile Applications.

If you have questions or suggestions, please comment below.

Going Serverless: Migrating an Express Application to Amazon API Gateway and AWS Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/going-serverless-migrating-an-express-application-to-amazon-api-gateway-and-aws-lambda/

Brett Andrews
Brett Andrews
Software Development Engineer

Amazon API Gateway recently released three new features that simplify the process of forwarding HTTP requests to your integration endpoint: greedy path variables, the ANY method, and proxy integration types. With this new functionality, it becomes incredibly easy to run HTTP applications in a serverless environment by leveraging the aws-serverless-express library.

In this post, I go through the process of porting an "existing" Node.js Express application onto API Gateway and AWS Lambda, and discuss some of the advantages, disadvantages, and current limitations. While I use Express in this post, the steps are similar for other Node.js frameworks, such as Koa, Hapi, vanilla, etc.

Modifying an existing Express application

Express is commonly used for both web applications as well as REST APIs. While the primary API Gateway function is to deliver APIs, it can certainly be used for delivering web apps/sites (HTML) as well. To cover both use cases, the Express app below exposes a web app on the root / resource, and a REST API on the /pets resource.

The goal of this walkthrough is for it to be complex enough to cover many of the limitations of this approach today (as comments in the code below), yet simple enough to follow along. To this end, you implement just the entry point of the Express application (commonly named app.js) and assume standard implementations of views and controllers (which are more insulated and thus less affected). You also use MongoDB, due to it being a popular choice in the Node.js community as well as providing a time-out edge case. For a greater AWS serverless experience, consider adopting Amazon DynamoDB.

//app.js
'use strict'
const path = require('path')
const express = require('express')
const bodyParser = require('body-parser')
const cors = require('cors')
const mongoose = require('mongoose')
// const session = require('express-session')
// const compress = require('compression')
// const sass = require('node-sass-middleware')

// Lambda does not allow you to configure environment variables, but dotenv is an
// excellent and simple solution, with the added benefit of allowing you to easily
// manage different environment variables per stage, developer, environment, etc.
require('dotenv').config()

const app = express()
const homeController = require('./controllers/home')
const petsController = require('./controllers/pets')

// MongoDB has a default timeout of 30s, which is the same timeout as API Gateway.
// Because API Gateway was initiated first, it also times out first. Reduce the
// timeout and kill the process so that the next request attempts to connect.
mongoose.connect(process.env.MONGODB_URI, { server: { socketOptions: { connectTimeoutMS: 10000 } } })
mongoose.connection.on('error', () => {
   console.error('Error connecting to MongoDB.')
   process.exit(1)
})

app.set('views', path.join(__dirname, 'views'))
app.set('view engine', 'pug')
app.use(cors())
app.use(bodyParser.json())
app.use(bodyParser.urlencoded({ extended: true }))

/*
* GZIP support is currently not available to API Gateway.
app.use(compress())

* node-sass is a native binary/library (aka Addon in Node.js) and thus must be
* compiled in the same environment (operating system) in which it will be run.
* If you absolutely need to use a native library, you can set up an Amazon EC2 instance
* running Amazon Linux for packaging your Lambda function.
* In the case of SASS, I recommend to build your CSS locally instead and
* deploy all static assets to Amazon S3 for improved performance.
const publicPath = path.join(__dirname, 'public')
app.use(sass({ src: publicPath, dest: publicPath, sourceMap: true}))
app.use(express.static(publicPath, { maxAge: 31557600000 }))

* Storing local state is unreliable due to automatic scaling. Consider going stateless (using REST),
* or use an external state store (for MongoDB, you can use the connect-mongo package)
app.use(session({ secret: process.env.SESSION_SECRET }))
*/

app.get('/', homeController.index)
app.get('/pets', petsController.listPets)
app.post('/pets', petsController.createPet)
app.get('/pets/:petId', petsController.getPet)
app.put('/pets/:petId', petsController.updatePet)
app.delete('/pets/:petId', petsController.deletePet)

/*
* aws-serverless-express communicates over a Unix domain socket. While it's not required
* to remove this line, I recommend doing so as it just sits idle.
app.listen(3000)
*/

// Export your Express configuration so that it can be consumed by the Lambda handler
module.exports = app

Assuming that you had the relevant code implemented in your views and controllers directories and a MongoDB server available, you could uncomment the listen line, run node app.js and have an Express application running at http://localhost:3000. The following "changes" made above were specific to API Gateway and Lambda:

  • Used dotenv to set environment variables.
  • Reduced the timeout for connecting to DynamoDB so that API Gateway does not time out first.
  • Removed the compression middleware as API Gateway does not (currently) support GZIP.
  • Removed node-sass-middleware (I opted for serving static assets through S3, but if there is a particular native library your application absolutely needs, you can build/package your Lambda function on an EC2 instance).
  • Served static assets through S3/CloudFront. Not only is S3 a better option for static assets for performance reasons, API Gateway does not currently support binary data (e.g., images).
  • Removed session state for scalability (alternatively, you could have stored session state in MongoDB using connect-mongo.
  • Removed app.listen() as HTTP requests are not being sent over ports (not strictly required).
  • Exported the Express configuration so that you can consume it in your Lambda handler (more on this soon).

Going serverless with aws-serverless-express

In order for users to be able to hit the app (or for developers to consume the API), you first need to get it online. Because this app is going to be immensely popular, you obviously need to consider scalability, resiliency, and many other factors. Previously, you could provision some servers, launch them in multiple Availability Zones, configure Auto Scaling policies, ensure that the servers were healthy (and replace them if they weren’t), keep up-to-date with the latest security updates, and so on…. As a developer, you just care that your users are able to use the product; everything else is a distraction.

Enter serverless. By leveraging AWS services such as API Gateway and Lambda, you have zero servers to manage, automatic scaling out-of-the-box, and true pay-as-you-go: the promise of the cloud, finally delivered.

The example included in the aws-serverless-express library library includes a good starting point for deploying and managing your serverless resources.

  1. Clone the library into a local directory git clone https://github.com/awslabs/aws-serverless-express.git
  2. From within the example directory, run npm run config <accountId> <bucketName> [region] (this modifies some of the files with your own settings).
  3. Edit the package-function command in package.json by removing "index.html" and adding "views" and "controllers" (the additional directories required for running your app).
  4. Copy the following files in the example directory into your existing project’s directory:

    • simple-proxy-api.yaml – A Swagger file that describes your API.
    • cloudformation.json – A CloudFormation template for creating the Lambda function and API.
    • package.json – You may already have a version of this file, in which case just copy the scripts and config sections. This includes some helpful npm scripts for managing your AWS resources, and testing and updating your Lambda function.
    • api-gateway-event.json – Used for testing your Lambda function locally.
    • lambda.js – The Lambda function, a thin wrapper around your Express application.

    Take a quick look at lambda.js so that you understand exactly what’s going on there. The aws-serverless-express library transforms the request from the client (via API Gateway) into a standard Node.js HTTP request object; sends this request to a special listener (a Unix domain socket); and then transforms it back for the response to API Gateway. It also starts your Express server listening on the Unix domain socket on the initial invocation of the Lambda function. Here it is in its entirety:

// lambda.js
'use strict'
const awsServerlessExpress = require('aws-serverless-express')
const app = require('./app')
const server = awsServerlessExpress.createServer(app)

exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context)

TIP: Everything outside of the handler function is executed only one time per container: that is, the first time your app receives a request (or the first request after several minutes of inactivity), and when it scales up additional containers.

Deploying

Now that you have more of an understanding of how API Gateway and Lambda communicate with your Express server, it’s time to release your app to the world.

From the project’s directory, run:

npm run setup

This command creates the Amazon S3 bucket specified earlier (if it does not yet exist); zips the necessary files and directories for your Lambda function and uploads it to S3; uploads simple-proxy-api.yaml to S3; creates the CloudFormation stack; and finally opens your browser to the AWS CloudFormation console where you can monitor the creation of your resources. To clean up the AWS resources created by this command, simply run npm run delete-stack. Additionally, if you specified a new S3 bucket, run npm run delete-bucket.

After the status changes to CREATE_COMPLETE (usually after a couple of minutes), you see three links in the Outputs section: one to the API Gateway console, another to the Lambda console, and most importantly one for your web app/REST API. Clicking the link to your API displays the web app; appending /pets in the browser address bar displays your list of pets. Your Express application is available online with automatic scalability and pay-per-request without having to manage a single server!

Additional features

Now that you have your REST API available to your users, take a quick look at some of the additional features made available by API Gateway and Lambda:

  • Usage plans for monetizing the API
  • Caching to improve performance
  • Authorizers for authentication and authorization microservices that determine access to your Express application
  • Stages and versioning and aliases when you need additional stages or environments (dev, beta, prod, etc.)
  • SDK generation to provide SDKs to consumers of your API (available in JavaScript, iOS, Android Java, and Android Swift)
  • API monitoring for logs and insights into usage

After running your Express application in a serverless environment for a while and learning more about the best practices, you may start to want more: more performance, more control, more microservices!

So how do you take your existing serverless Express application (a single Lambda function) and refactor it into microservices? You strangle it. Take a route, move the logic to a new Lambda function, and add a new resource or method to your API Gateway API. However, you’ll find that the tools provided to you by the aws-serverless-express example just don’t cut it for managing these additional functions. For that, you should check out Claudia; it even has support for aws-serverless-express.

Conclusion

To sum it all up, you took an existing application of moderate complexity, applied some minimal changes, and deployed it in just a couple of commands. You now have no servers to manage, automatic scaling out-of-the-box, true pay-as-you-go, loads of features provided by API Gateway, and as a bonus, a great path forward for refactoring into microservices.

If that’s not enough, or this server-side stuff doesn’t interest you and the front end is where you live, consider using aws-serverless-express for server-side rendering of your single page application.

If you have questions or suggestions, please comment below.

Easier integration with AWS Lambda and Amazon API Gateway

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/easier-integration-with-aws-lambda-and-amazon-api-gateway/

This week, Amazon API Gateway announced three new features that make it easier for you to leverage API Gateway and AWS Lambda to build your serverless applications.

First, we now support catch-all path variables. You can define routes such as /store/{proxy+}, where the + symbol tells API Gateway to intercept all requests to the /store/* path. Second, we now support a new method type called ANY. You can use the catch-all ANY method to define the same integration behavior for all requests (GET, POST, etc). Third, you can now use a new proxy integration type for Lambda functions and HTTP endpoints. Lambda function proxy integrations apply a default mapping template to send the entire request to your functions, and it automatically maps Lambda output to HTTP responses. HTTP proxy integrations simply pass the entire request and response directly through to your HTTP endpoint.

One way to use these new features is to migrate Express applications to Lambda and API Gateway. Previously, in order to preserve your API routes in Express, you had to redefine each API method and its corresponding Express integration endpoint on API Gateway. Now, you can simply define one catch-all resource in API Gateway and configure it as a proxy integration with a Lambda function that wraps your Express application.

Head on over to Jeff Barr’s blog to read more about these new features!

Techniques and Tools for Better Serverless API Logging with Amazon API Gateway and AWS Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/techniques-and-tools-for-better-serverless-api-logging-with-amazon-api-gateway-and-aws-lambda/


Ryan Green @ryangtweets
Software Development Engineer, API Gateway

Developing, testing, and operating Serverless APIs using Amazon API Gateway and AWS Lambda can be made much easier with built-in support for Amazon CloudWatch Logs.

In Lambda functions, you can use log statements to send log events to CloudWatch log streams, and API Gateway automatically submits log events for requests to APIs with logging enabled.

However, it can be difficult to reconcile log events for a serverless API sent across multiple CloudWatch log groups and log streams. Tracking down logs for a specific request or tailing request logs for that operation can sometimes be a cumbersome experience.

Here are some simple logging techniques and tools that can help greatly while developing, testing, and operating your serverless API operatios.

Technique: Capture the request ID for a particular API request

The request ID for an individual API request is always returned by API Gateway in the " x-amzn-RequestId" response header. When API Gateway logs an individual API request, the request ID is always included in the first event in the request log:

"Starting execution for request: [REQUEST_ID]"

Logs for an API Gateway API are always sent to a log group in the following format:

"API-Gateway-Execution-Logs_[API_ID]/[STAGE_NAME]"

To troubleshoot an individual API request, search for the request ID in the CloudWatch Logs console, or using the Cloudwatch API or an AWS SDK (more on tooling later).

Technique: Correlate your API Gateway request IDs with Lambda request IDs

Individual API requests are tracked independently across AWS services. Thus, an individual request to your API actually generates at least two request identifiers ("request IDs") – one for the API Gateway request and one for the Lambda invocation request.

To reduce time-to-diagnosis when analyzing logs, it is helpful to correlate both request IDs together in the logs. Log the API Gateway request ID from your Lambda function and send the API Gateway request ID ($context.requestId) to your Lambda function via a mapping template:

{
   "context" : {
      "request-id" : "$context.requestId"
   }
   …
}

Then, in your Lambda function, log the API Gateway request ID along with the Lambda request ID. The Lambda request ID is automatically included in the log message.

exports.handler = function(event, context) {
    var apiRequestId = event.context['request-id'];
    var lambdaRequestId = context.awsRequestId;
    console.log("API Gateway Request ID: " + apiRequestId + " Lambda Request ID: " + context.awsRequestId);
    var logprefix = "APIG: " + apiRequestId + " -  ";
    console.log(logprefix + "hello world!");
    ...
}

Invoking this Lambda function produces these log messages in the "/aws/lambda/[FUNCTION_NAME]" log group:

/aws/lambda/echo                           2016/09/07/[$LATEST]6ccf17d298b64b5fac8c41b1a65e0831 2016-09-07T21:39:39.145Z        943ad105-7543-11e6-a9ac-65e093327849        API Gateway Request ID: 9439989f-7543-11e6-8dda-150c09a55dc2 Lambda Request ID: 943ad105-7543-11e6-a9ac-65e093327849

/aws/lambda/echo                           2016/09/07/[$LATEST]6ccf17d298b64b5fac8c41b1a65e0831 2016-09-07T21:39:39.145Z        943ad105-7543-11e6-a9ac-65e093327849        APIG: 9439989f-7543-11e6-8dda-150c09a55dc2 -   hello world!

Using this technique allows you to quickly locate the Lambda function logs for an individual API request. Search for the request ID returned in the "x-amzn-RequestId" header in the log group for the Lambda function (by default, named "/aws/lambda/[FUNCTION_NAME]").

Tooling

While the AWS SDK and CLI provide excellent building blocks for working with API Gateway, Lambda, and CloudWatch Logs, there is still room for specialized tooling for logs when developing, testing, and operating your serverless API.

To that end, I’ve created apilogs—a fork of the excellent awslogs project—to include native support for API Gateway/Lambda serverless APIs.

Given an API Gateway REST API ID and Stage name, this command-line tool produces an aggregated stream of time-ordered, ANSI-colored log events emitted by API Gateway and all Lambda functions attached to your API. It automatically aggregates events from all log streams for the API and Lambda functions.

For example:

Stream all log events emitted from API Gateway as well as from all Lambda functions attached to the API:

$ apilogs get --api-id xyz123 --stage prod –-watch

Search APIG/Lambda logs for events from a specific request ID in the past hour:

$ apilogs get --api-id xyz123 --stage prod --start='1h ago' | grep "6605b081-6f04-11e6-97ac-c34deb0b3dd9"

The log events can then be further filtered and processed by standard command-line tools. Credentials are passed to apilogs via the same mechanism as the AWS CLI.

Conclusion

I hope apilogs can become part of your standard dev, test, or ops workflows. Check out apilogs on Github. Feedback and contributions are always welcome!

Migrating a Native JAVA REST API to a Serverless Architecture with the Lambada Framework for AWS

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/migrating-a-native-java-rest-api-to-a-serverless-architecture-with-the-lambada-framework-for-aws/

This is a guest post by Çağatay Gürtürk, the creator of the Lambada framework

Serverless computing has become a hot topics since AWS Lambda and Amazon API Gateway started to offer an elegant way to build and deploy REST APIs without needing to maintain 24/7 running servers and infrastructure, with attractive pricing models.

Being the first language offered by Lambda, Node.JS seems to have the most online resources and tools but it is also possible to write Lambda functions natively with Java and Python. Java is especially interesting as a language because of its maturity, large community, and available codebase. With Lambda and Java, it is even possible to apply enterprise patterns and frameworks such as Spring, as well as all the best practices we used to apply in the Java world.

In order to make development for Lambda in Java easier, I started Lambada Framework as an open source project. It is a little, but powerful, open source project in beta stage that lets developers create a new serverless API in AWS infrastructure or migrate an existing one.

Lambada Framework accomplishes this target by implementing the most common JAX-RS annotations and providing a Maven plugin to deploy easily to the AWS cloud. Briefly, JAX-RS is a standard annotation set which can be used to map regular Java methods to HTTP paths and methods. For instance, you can look at the following method:

@GET
@Path("/helloworld/{id}")
public Response indexEndpoint(@PathParam int id) {
    return Response.status(200).entity("Hello world: " + id).build();
}

This is a very lean method marked with @GET and @Path annotations, which mean that this method is called when a GET request comes to URLs in "/helloworld/{id}" format, with theid parameter as an argument. Finally, it returns a Response object within this method with a 200 response code and text content. As you can see, these annotations offer a seamless way to define a REST API and map different resources to Java methods.

JAX-RS annotations on their own do not mean so much and they do not have any effect out-of-the-box. To make these annotations work, a JAX-RS implementation framework should be added to the project. This framework would scan all the JAX-RS annotations in the project and create a server and routing table to respond to HTTP requests correctly. While Jersey is one such reference implementation, and the most popular one, there are also other implementations of JAX-RS, such as RESTEasy and Apache CXF. You are free to choose any of them and your controller methods always stay same, thanks to standard annotations.

Lambada Framework is a JAX-RS implementation but different from the others: instead of running a web server, it scans the JAX-RS annotations at build time and populates Lambda functions and the API Gateway definitions using them.

This means that if you already marked your controller methods with JAX-RS annotations and used a framework like Jersey or RestEasy, you can easily switch to serverless architecture with very little modifications in your code. You would have to change only your build mechanism and replace your preferred JAX-RS implementation with Lambada Framework.

In the following example, you see how to deploy a very basic REST API to Lambda.

  1. First, clone the example project to your local directory:
    git clone https://github.com/lambadaframework/lambadaframework-boilerplate
  1. This project has a pom.xml file with some configuration options. You must change the deployment.bucket option; other changes are up to you. The Lambada Framework creates this bucket in your account if it does not exists and it uses that bucket during your project’s lifetime. S3 bucket names are global and must be unique, so you must pick a name which is not taken by any one else.

  2. Make sure that the default AWS profile installed in your system has administrator privileges, or at least the following IAM policy:

     {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "cloudformation:*",
                        "s3:*",
                        "lambda:*",
                        "execute-api:*",
                        "apigateway:*",
                        "iam:*",
                        "ec2:DescribeSecurityGroups",
                        "ec2:DescribeVpcs",
                        "ec2:DescribeSubnets"
                    ],
                    "Resource": [
                        "*"
                    ]
                }
            ]
        }
  1. Now you are all set. In the root directory of your project, fire the following command:

    mvn deploy

Your project compiles to a fat JAR with all its dependencies and is deployed to Lambda. After the JAR file is on the S3 bucket, Lambada scans that for supported JAX-RS annotations in your code and creates the necessary API Gateway endpoints. At the end of the process, the URL of your API is printed on the screen. You can navigate to this URL and explore your API using the AWS Management Console to see which resources and methods are created.

Lambada Framework is under development and support for missing JAX-RS annotations are being added. Follow the Lambada Framework GitHub page for the newest features and feel free to submit any issue or contributions.

Happy serverless computing!

Maintaining a Healthy Email Database with AWS Lambda, Amazon SNS, and Amazon DynamoDB

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/maintaining-a-healthy-email-database-with-aws-lambda-amazon-sns-and-amazon-dynamodb/

Carlos Sanchiz
Sr. Solutions Architect

Mike Deck
Partner Solutions Architect

Reputation in the email world is critical to achieve reasonable deliverability rates (the percentage of emails that arrive to inboxes); if you fall under certain levels, your emails end up in the spam folder or rejected by the email servers. To keep these numbers high, you have to constantly improve your email quality, but most importantly, you have to take action when a delivery fails or a recipient doesn’t want to receive your email.

Back in 2012, we showed you how to automate the process of handling bounces and complaints with an scheduled task, using Amazon SNS, Amazon SQS, Amazon EC2, and some C# code. We have released many AWS services since then, so this post shows a different approach towards the same goal of a clean and healthy email database.

To set a little bit of context about bounces and complaints processing, I’m reusing some of the previous post:

Amazon SES assigns a unique message ID to each email that you successfully submit to send. When Amazon SES receives a bounce or complaint message from an ISP, we forward the feedback message to you. The format of bounce and complaint messages varies between ISPs, but Amazon SES interprets these messages and, if you choose to set up Amazon SNS topics for them, categorizes them into JSON objects.

Amazon SES will categorize your hard bounces into two types: permanent and transient. A permanent bounce indicates that you should never send to that recipient again. A transient bounce indicates that the recipient’s ISP is not accepting messages for that particular recipient at that time and you can retry delivery in the future. The amount of time you should wait before resending to the address that generated the transient bounce depends on the transient bounce type. Certain transient bounces require manual intervention before the message can be delivered (e.g., message too large or content error). If the bounce type is undetermined, you should manually review the bounce and act accordingly.

A complaint indicates the recipient does not want the email that you sent them. When we receive a complaint, we want to remove the recipient addresses from our list.

In this post, we show you how to use AWS Lambda functions to receive SES notifications from the feedback loop from ISPs email servers via Amazon SNS and update an Amazon DynamoDB table with your email database.

Here is a high-level overview of the architecture:

Using the combination of Lambda, SNS and DynamoDB frees you from the operational overhead of having to run servers and maintain them. You focus on your application logic and AWS handles the undifferentiating heavy lifting behind the operations, scalability, and high availability.

Workflow

  1. Create the SNS topic to receive the SES bounces, deliveries and complaints.
  2. Create the DynamoDB table to use for our email database.
  3. Create the Lambda function to process the bounces, deliveries and complaints and subscribe it to the SNS topic
  4. Test & start emailing!

Create an SNS topic

First, create an SNS topic named "ses-notifications". You subscribe your Lambda function to the topic later.

Create a DynamoDB table

Create a simple DynamoDB table called "mailing" to store the email database. Use the UserId (email address) as the partition key.

Create the Lambda function

Set up your Lambda function that will process all the notifications coming from SES through your SNS topic.

Note: This post uses Node.js 4.3 as the Lambda runtime but at the time of publication, you can also use Python 2.7, Java 8 or Node.js 0.10.

For the Lambda function code, I used the recently published blueprint (ses-notification-nodejs) and adapted it to work with the DynamoDB table. The following code has the modifications highlighted:

'use strict';
console.log('Loading function');

let doc = require('dynamodb-doc');
let dynamo = new doc.DynamoDB();
let tableName = 'mailing';

exports.handler = (event, context, callback) => {
    //console.log('Received event:', JSON.stringify(event, null, 2));
    const message = JSON.parse(event.Records[0].Sns.Message);

    switch(message.notificationType) {
        case "Bounce":
            handleBounce(message);
            break;
        case "Complaint":
            handleComplaint(message);
            break;
        case "Delivery":
            handleDelivery(message);
            break;
        default:
            callback("Unknown notification type: " + message.notificationType);
    }
};

function handleBounce(message) {
    const messageId = message.mail.messageId;
    const addresses = message.bounce.bouncedRecipients.map(function(recipient){
        return recipient.emailAddress;
    });
    const bounceType = message.bounce.bounceType;

    console.log("Message " + messageId + " bounced when sending to " + addresses.join(", ") + ". Bounce type: " + bounceType);

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "disable");
    }
}

function handleComplaint(message) {
    const messageId = message.mail.messageId;
    const addresses = message.complaint.complainedRecipients.map(function(recipient){
        return recipient.emailAddress;
    });

    console.log("A complaint was reported by " + addresses.join(", ") + " for message " + messageId + ".");

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "disable");
    }
}

function handleDelivery(message) {
    const messageId = message.mail.messageId;
    const deliveryTimestamp = message.delivery.timestamp;
    const addresses = message.delivery.recipients;

    console.log("Message " + messageId + " was delivered successfully at " + deliveryTimestamp + ".");

    for (var i=0; i<addresses.length; i++){
        writeDDB(addresses[i], message, tableName, "enable");
    }
}

function writeDDB(id, payload, tableName, status) {
    const item = {
            UserId: id,
            notificationType: payload.notificationType,
            from: payload.mail.source,
            timestamp: payload.mail.timestamp,
            state: status
        };
    const params = {
            TableName:tableName,
            Item: item
        };
    dynamo.putItem(params,function(err,data){
            if (err) console.log(err);
            else console.log(data);
    });
}

Assign the function a role with execute and DynamoDB permissions so it can run and update the DynamoDB table accordingly and use index.handler as the function Handler.

This is the lambda_dynamo IAM role policy to use for the three functions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1428341300017",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:UpdateItem"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:dynamodb:us-east-1:ACCOUNT-ID:table/mailing"
        },
        {
            "Sid": "",
            "Resource": "*",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow"
        }
    ]
}

You also set the corresponding SNS topic as an event source so that the Lambda function is triggered when notifications arrive.

Test the Lambda function

After everything is in place, it’s time to test. Publish notifications to your SNS topics using Amazon SNS notification examples for Amazon SES, and see how your DynamoDB table is updated by your Lambda functions.

Here’s an example of publishing a complaint notification to the ses-complaints-topic SNS topic using the CLI:

$ aws sns publish --topic-arn "arn:aws:sns:us-east-1:xxxxxxxxxxx:ses-notifications" --message file://message_complaints.txt --subject Test --region us-east-1

{
    "MessageId": "f7f5ad2d-a268-548d-a45c-e28e7624a64d"
}
$ cat message_complaints.txt

{
      "notificationType":"Complaint",
      "complaint":{
         "userAgent":"Comcast Feedback Loop (V0.01)",
         "complainedRecipients":[
            {
               "emailAddress":"[email protected]"
            }
         ],
         "complaintFeedbackType":"abuse",
         "arrivalDate":"2009-12-03T04:24:21.000-05:00",
         "timestamp":"2012-05-25T14:59:38.623-07:00",
         "feedbackId":"000001378603177f-18c07c78-fa81-4a58-9dd1-fedc3cb8f49a-000000"
      },
      "mail":{
         "timestamp":"2012-05-25T14:59:38.623-07:00",
     "messageId":"000001378603177f-7a5433e7-8edb-42ae-af10-f0181f34d6ee-000000",
         "source":"[email protected]",
         "sourceArn": "arn:aws:sns:us-east-1:XXXXXXXXXXXX:ses-notifications",
         "sendingAccountId":"XXXXXXXXXXXX",
         "destination":[
            "[email protected]",
            "[email protected]",
            "[email protected]",
            "[email protected]"
         ]
      }
   }

And here is what you’d start seeing coming in your DynamoDB items list:

After you are done with your tests, you can point your SES notifications to the SNS topic you created and start sending emails.

Conclusion

In this post, we showed how you can use AWS Lambda, Amazon SNS, and Amazon DynamoDB to keep a healthy email database, have a good email sending score, and of course, do all of it without servers to maintain or scale.

While you’re at it, why not expose your DynamoDB table with your email database using Amazon API Gateway? For more information, see Using Amazon API Gateway as a proxy for DynamoDB.

If you have questions or suggestions, please comment below.

A Data Sharing Platform Based on AWS Lambda

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/a-data-sharing-platform-based-on-aws-lambda/

Julien Lepine

Julien Lepine
Solutions Architect

As developers, one of our top priorities is to build reliable systems; this is a core pillar of the AWS Well Architected Framework. A common pattern to fulfill this goal is to have an architecture built around loosely coupled components.

Amazon Kinesis Streams offers an excellent answer for this, as the events generated can be consumed independently by multiple consumers and remain available for 1 to 7 days. Building an Amazon Kinesis consumer application is done by leveraging the Amazon Kinesis Client Library (KCL) or native integration with AWS Lambda.

As I was speaking with other developers and customers about their use of Amazon Kinesis, there are a few patterns that came up. This post addresses those common patterns.

Protecting streams

Amazon Kinesis has made the implementation of event buses easy and inexpensive, so that applications can send meaningful information to their surrounding ecosystem. As your applications grow and get more usage within your company, more teams will want to consume the data generated, even probably external parties such as business partners or customers.

When the applications get more usage, some concerns may arise:

  • When a new consumer starts (or re-starts after some maintenance), it needs to read a lot of data from the stream (its backlog) in a short amount of time in order to get up to speed
  • A customer may start many consumers at the same time, reading a lot of events in parallel or having a high call rate to Amazon Kinesis
  • A consumer may have an issue (such as infinite loop, retry error) that causes it to call Amazon Kinesis at an extremely high rate

These cases may lead to a depletion of the resources available in your stream, and that could potentially impact all your consumers.

Managing the increased load can be done by leveraging the scale-out model of Amazon Kinesis through the addition of shards to an existing stream. Each shard adds both input (ingestion) and output (consumption) capacity to your stream:

  • 1000 write records and up to 1 megabyte per second for ingesting events
  • 5 read transactions and up to 2 megabytes per second for consuming events

Avoiding these scenarios could be done by scaling-out your streams, and provisioning for peak, but that would create inefficiencies and may not even fully protect your consumers from the behavior of others.

What becomes apparent in these cases is the impact that a single failing consumer may have on all other consumers, a symptom described as the “noisy neighbor”, or managing the blast radius of your system. The key point is to limit the impact that a single consumer can have on others.

A solution is to compartmentalize your platform: this method consists of creating multiple streams and then creating groups of consumers that share the same stream. This gives you the possibility to limit the impact a single consumer can have on its neighbors, and potentially to propose a model where some customers have a dedicated stream.

You can build an Amazon Kinesis consumer application (via the KCL or Lambda) that reads a source stream and sends the messages to the “contained” streams that the actual consumers will use.

Transforming streams

Another use case I see from customers is the need to transfer the data in their stream to other services:

  • Some applications may have limitations in their ability to receive or process the events
  • They may not have connectors to Amazon Kinesis, and only support Amazon SQS
  • They may only support a push model, where their APIs need to be called directly when a message arrives
  • Some analytics/caching/search may be needed on the events generated
  • Data may need to be archived or sent to a data warehouse engine

There are many other cases, but the core need is having the ability to get the data from Amazon Kinesis into other platforms.

The solution for these use cases is to build an Amazon Kinesis consumer application that reads a stream and prepares these messages for other services.

Sharing data with external parties

The final request I have seen is the possibility to process a stream from a different AWS account or region. While you can give access to your resources to an external AWS account through cross-account IAM roles, that feature requires development and is not supported natively by some services. For example, you cannot subscribe a Lambda function to a stream in a different AWS account or region.

The solution is to replicate the Amazon Kinesis stream or the events to another environment (AWS account, region, or service).

This can be done one time through an Amazon Kinesis consumer application that reads a stream and forwards the events to the remote environment.

Solution: A Lambda-based fan-out function

These three major needs have a common solution: the deployment of an Amazon Kinesis consumer application that listens to a stream and is able to send messages to other instances of Amazon Kinesis, services, or environments (AWS accounts or regions).

In the aws-lambda-fanout GitHub repository, you’ll find a Lambda function that specifically supports this scenario. This function is made to forward incoming messages from Amazon Kinesis or DynamoDB Streams.

The architecture of the function is made to be simple and extensible, with one core file fanout.js that loads modules for the different providers. The currently supported providers are as follows:

  • Amazon SNS
  • Amazon SQS
  • Amazon Elasticsearch Service
  • Amazon Kinesis Streams
  • Amazon Kinesis Firehose
  • AWS IoT
  • AWS Lambda
  • Amazon ElastiCache for Memcached
  • Amazon ElastiCache for Redis

The function is built to support multiple inputs:

  • Amazon Kinesis streams
  • Amazon Kinesis streams containing Amazon Kinesis Producer Library (KPL) records
  • DynamoDB Streams records

It relies on Lambda for a fully-managed environment where scaling, logging, and monitoring are automated by the platform. It also supports Lambda functions in a VPC for Amazon ElastiCache.

The configuration is stored in a DynamoDB table, and associates the output configuration with each function. This table has a simple schema:

  • sourceArn (Partition Key): The Amazon Resource Name (ARN) of the input Amazon Kinesis stream
  • id [String]: The name of the mapping
  • type [String]: The destination type
  • destination [String]: The ARN or name of the destination
  • active [Boolean]: Whether that mapping is active

Depending on the target, some other properties are also stored.

The function can also group records together for services that don’t initially support it, such as Amazon SQS, Amazon SNS, or AWS IoT. Amazon DynamoDB Streams records can also be transformed to plain JSON objects to simplify management in later stages. The function comes with a Bash-based command line Interface to make the deployment and management easier.

As an example, the following lines deploy the function, which registers a mapping from one stream (inputStream) to another (outputStream).

./fanout deploy --function fanout

./fanout register kinesis --function fanout --source-type kinesis --source inputStream --id target1 --destination outputStream --active true

./fanout hook --function fanout --source-type kinesis --source inputStream

Summary

There are many options available for you to forward your events from one service or environment to another. For more information about this topic, see Using AWS Lambda with Amazon Kinesis. Happy eventing!

If you have questions or suggestions, please comment below.

Implementing a Serverless AWS IoT Backend with AWS Lambda and Amazon DynamoDB

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/implementing-a-serverless-aws-iot-backend-with-aws-lambda-and-amazon-dynamodb/

Ed Lima

Ed Lima
Cloud Support Engineer

Does your IoT device fleet scale to hundreds or thousands of devices? Do you find it somewhat challenging to retrieve the details for multiple devices? AWS IoT provides a platform to connect those devices and build a scalable solution for your Internet of Things workloads.

Out of the box, the AWS IoT console gives you your own searchable device registry with access to the device state and information about device shadows. You can enhance and customize the service using AWS Lambda and Amazon DynamoDB to build a serverless backend with a customizable device database that can be used to store useful information about the devices as well as helping to track what devices are activated with an activation code, if required.

You can use DynamoDB to extend the AWS IoT internal device registry to help manage the device fleet, as well as storing specific additional data about each device. Lambda provides the link between AWS IoT and DynamoDB allowing you to add, update, and query your new device database backend.

In this post, you learn how to use AWS IoT rules to trigger specific device registration logic using Lamba in order to populate a DynamoDB table. You then use a second Lambda function to search the database for a specific device serial number and a randomly generated activation code to activate the device and register the email of the device owner in the same table. After you’re done, you’ll have a fully functional serverless IoT backend, allowing you to focus on your own IoT solution and logic instead of managing the infrastructure to do so.

Prerequisites

You must have the following before you can create and deploy this framework:

  • An AWS account
  • An IAM user with permissions to create AWS resources (AWS IoT things and rules, Lambda functions, DynamoDB tables, IAM policies and roles, etc.)
  • JS and the AWS SDK for JavaScript installed locally to test the deployment

Building a backend

In this post, I assume that you have some basic knowledge about the services involved. If not, you can review the documentation:

For this use case, imagine that you have a fleet of devices called “myThing”. These devices can be anything: a smart lightbulb, smart hub, Internet-connected robot, music player, smart thermostat, or anything with specific sensors that can be managed using AWS IoT.

When you create a myThing device, there is some specific information that you want to be available in your database, namely:

  • Client ID
  • Serial number
  • Activation code
  • Activation status
  • Device name
  • Device type
  • Owner email
  • AWS IoT endpoint

The following is a sample payload with details of a single myThing device to be sent to a specific MQTT topic, which triggers an IoT rule. The data is in a format that AWS IoT can understand, good old JSON. For example:

{
  "clientId": "ID-91B2F06B3F05",
  "serialNumber": "SN-D7F3C8947867",
  "activationCode": "AC-9BE75CD0F1543D44C9AB",
  "activated": "false",
  "device": "myThing1",
  "type": "MySmartIoTDevice",
  "email": "[email protected]",
  "endpoint": "<endpoint prefix>.iot.<region>.amazonaws.com"
}

The rule then invokes the first Lambda function, which you create now. Open the Lambda console, choose Create a Lambda function , and follow the steps. Here’s the code:

console.log('Loading function');
var AWS = require('aws-sdk');
var dynamo = new AWS.DynamoDB.DocumentClient();
var table = "iotCatalog";

exports.handler = function(event, context) {
    //console.log('Received event:', JSON.stringify(event, null, 2));
   var params = {
    TableName:table,
    Item:{
        "serialNumber": event.serialNumber,
        "clientId": event.clientId,
        "device": event.device,
        "endpoint": event.endpoint,
        "type": event.type,
        "certificateId": event.certificateId,
        "activationCode": event.activationCode,
        "activated": event.activated,
        "email": event.email
        }
    };

    console.log("Adding a new IoT device...");
    dynamo.put(params, function(err, data) {
        if (err) {
            console.error("Unable to add device. Error JSON:", JSON.stringify(err, null, 2));
            context.fail();
        } else {
            console.log("Added device:", JSON.stringify(data, null, 2));
            context.succeed();
        }
    });
}

The function adds an item to a DynamoDB database called iotCatalog based on events like the JSON data provided earlier. You now need to create the database as well as making sure the Lambda function has permissions to add items to the DynamoDB table, by configuring it with the appropriate execution role.

Open the DynamoDB console, choose Create table and follow the steps. For this table, use the following details.

The serial number uniquely identifies your device; if, for instance, it is a smart hub that has different client devices connecting to it, use the client ID as the sort key.

The backend is good to go! You just need to make the new resources work together; for that, you configure an IoT rule to do so.

On the AWS IoT console, choose Create a resource and Create a rule , and use the following settings to point the rule to your newly-created Lambda function, also called iotCatalog.

After creating the rule, AWS IoT adds permissions on the background to allow it to trigger the Lambda function whenever a message is published to the MQTT topic called registration. You can use the following Node.js deployment code to test:

var AWS = require('aws-sdk');
AWS.config.region = 'ap-northeast-1';

var crypto = require('crypto');
var endpoint = "<endpoint prefix>.iot.<region>.amazonaws.com";
var iot = new AWS.Iot();
var iotdata = new AWS.IotData({endpoint: endpoint});
var topic = "registration";
var type = "MySmartIoTDevice"

//Create 50 AWS IoT Things
for(var i = 1; i < 51; i++) {
  var serialNumber = "SN-"+crypto.randomBytes(Math.ceil(12/2)).toString('hex').slice(0,15).toUpperCase();
  var clientId = "ID-"+crypto.randomBytes(Math.ceil(12/2)).toString('hex').slice(0,12).toUpperCase();
  var activationCode = "AC-"+crypto.randomBytes(Math.ceil(20/2)).toString('hex').slice(0,20).toUpperCase();
  var thing = "myThing"+i.toString();
  var thingParams = {
    thingName: thing
  };
  
  iot.createThing(thingParams).on('success', function(response) {
    //Thing Created!
  }).on('error', function(response) {
    console.log(response);
  }).send();

  //Publish JSON to Registration Topic

  var registrationData = '{\n \"serialNumber\": \"'+serialNumber+'\",\n \"clientId\": \"'+clientId+'\",\n \"device\": \"'+thing+'\",\n \"endpoint\": \"'+endpoint+'\",\n\"type\": \"'+type+'\",\n \"activationCode\": \"'+activationCode+'\",\n \"activated\": \"false\",\n \"email\": \"[email protected]\" \n}';

  var registrationParams = {
    topic: topic,
    payload: registrationData,
    qos: 0
  };

  iotdata.publish(registrationParams, function(err, data) {
    if (err) console.log(err, err.stack); // an error occurred
    // else Published Successfully!
  });
  setTimeout(function(){},50);
}

//Checking all devices were created

iot.listThings().on('success', function(response) {
  var things = response.data.things;
  var myThings = [];
  for(var i = 0; i < things.length; i++) {
    if (things[i].thingName.includes("myThing")){
      myThings[i]=things[i].thingName;
    }
  }

  if (myThings.length = 50){
    console.log("myThing1 to 50 created and registered!");
  }
}).on('error', function(response) {
  console.log(response);
}).send();

console.log("Registration data on the way to Lambda and DynamoDB");

The code above creates 50 IoT things in AWS IoT and generate random client IDs, serial numbers, and activation codes for each device. It then publishes the device data as a JSON payload to the IoT topic accordingly, which in turn triggers the Lambda function:

And here it is! The function was triggered successfully by your IoT rule and created your database of IoT devices with all the custom information you need. You can query the database to find your things and any other details related to them.

In the AWS IoT console, the newly-created things are also available in the thing registry.

Now you can create certificates, policies, attach them to each “myThing” AWS IoT Thing then install each certificate as you provision the physical devices.

Activation and registration logic

However, you’re not done yet…. What if you want to activate a device in the field with the pre-generated activation code as well as register the email details of whoever activated the device?

You need a second Lambda function for that, with the same execution role from the first function (Basic with DynamoDB). Here’s the code:

console.log('Loading function');

var AWS = require('aws-sdk');
var dynamo = new AWS.DynamoDB.DocumentClient();
var table = "iotCatalog";

exports.handler = function(event, context) {
    //console.log('Received event:', JSON.stringify(event, null, 2));

   var params = {
    TableName:table,
    Key:{
        "serialNumber": event.serialNumber,
        "clientId": event.clientId,
        }
    };

    console.log("Gettings IoT device details...");
    dynamo.get(params, function(err, data) {
    if (err) {
        console.error("Unable to get device details. Error JSON:", JSON.stringify(err, null, 2));
        context.fail();
    } else {
        console.log("Device data:", JSON.stringify(data, null, 2));
        console.log(data.Item.activationCode);
        if (data.Item.activationCode == event.activationCode){
            console.log("Valid Activation Code! Proceed to register owner e-mail and update activation status");
            var params = {
                TableName:table,
                Key:{
                    "serialNumber": event.serialNumber,
                    "clientId": event.clientId,
                },
                UpdateExpression: "set email = :val1, activated = :val2",
                ExpressionAttributeValues:{
                    ":val1": event.email,
                    ":val2": "true"
                },
                ReturnValues:"UPDATED\_NEW"
            };
            dynamo.update(params, function(err, data) {
                if (err) {
                    console.error("Unable to update item. Error JSON:", JSON.stringify(err, null, 2));
                    context.fail();
                } else {
                    console.log("Device now active!", JSON.stringify(data, null, 2));
                    context.succeed("Device now active! Your e-mail is now registered as device owner, thank you for activating your Smart IoT Device!");
                }
            });
        } else {
            context.fail("Activation Code Invalid");
        }
    }
});
}

The function needs just a small subset of the data used earlier:

{
  "clientId": "ID-91B2F06B3F05",
  "serialNumber": "SN-D7F3C8947867",
  "activationCode": "AC-9BE75CD0F1543D44C9AB",
  "email": "[email protected]"
}

Lambda uses the hash and range keys (serialNumber and clientId) to query the database and compare the database current pre-generated activation code to a code that is supplied by the device owner along with their email address. If the activation code matches the one from the database, the activation status and email details are updated in DynamoDB accordingly. If not, the user gets an error message stating that the code is invalid.

You can turn it into an API with Amazon API Gateway. In order to do so, go to the Lambda function and add an API endpoint, as follows.

Now test the access to the newly-created API endpoint, using a tool such as Postman.

If an invalid code is provided, the requester gets an error message accordingly.

Back in the database, you can confirm the record was updated as required.

Cleanup

After you finish the tutorial, delete all the newly created resources (IoT things, Lambda functions, and DynamoDB table). Alternatively, you can keep the Lambda function code for future reference, as you won’t incur charges unless the functions are invoked.

Conclusion

As you can see, by leveraging the power of the AWS IoT Rules Engine, you can take advantage of the seamless integration with AWS Lambda to create a flexible and scalable IoT backend powered by Amazon DynamoDB that can be used to manage your growing Internet of Things fleet.

You can also configure an activation API to make use of the newly-created backend and activate devices as well as register email contact details from the device owner; this information could be used to get in touch with your users regarding marketing campaigns or newsletters about new products or new versions of your IoT products.

If you have questions or suggestions, please comment below.

Redirection in a Serverless API with AWS Lambda and Amazon API Gateway

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/redirection-in-a-serverless-api-with-aws-lambda-and-amazon-api-gateway/

Ronald Widha

Ronald Widha @ronaldwidha
Partner Solutions Architect

Redirection is neither a success nor an error response. You return redirection when the requested resource resides either temporarily or permanently under a different URI. The client needs to issue subsequent calls to the new location in order to retrieve the requested resource. Even though you typically see 302 and 301 redirects when requesting text/html, these response types also apply to REST JSON endpoints.

Many of you are already familiar how to return success and error responses. Amazon API Gateway and AWS Lambda help development teams to build scalable web endpoints very quickly. In a previous post (Error Handling Patterns in Amazon API Gateway and AWS Lambda), we discussed several error handling patterns to implement an explicit contract for the types of error responses that an API can produce and how they translate into HTTP status codes. However, so far this blog has not discussed how to handle redirection.

This post shows the recommended patterns for handling redirection in your serverless API built on API Gateway and Lambda.

Routing Lambda errors to API Gateway HTTP 30x responses

In HTTP, there are several types of redirection codes. You return these status codes as part of an HTTP response to the client.

TypeHTTP Status CodeDescription
Multiple types available300The requested resource is available in multiple representations with its own specific location (not commonly used).
Moved permanently301The requested resource has been moved a new permanent URI.
Found302The requested resource is temporarily available under a different URI.

For more information about HTTP server status codes, see RFC2616 section 10.5 on the W3C website.

In this specific scenario, you would like your API Gateway method to return the following HTTP response. Note that there are two key parameters to return:

  1. The status code, i.e., 301 or 302
  2. The new URI of the resource
HTTP/1.1 302 Found
Content-Type: text/html
Content-Length: 503
Connection: keep-alive
Date: Fri, 08 Jul 2016 16:16:44 GMT
Location: http://www.amazon.com/

In API Gateway, AWS recommends that you model the various HTTP response types that your API method may produce, and define a mapping from the return value of your Lambda function to these HTTP responses. To do that, you need to do two things:

  1. API Gateway can only map responses to different method responses on error. So even though redirection isn’t strictly a failure, you still throw an exception from Lambda to communicate back to API Gateway when it needs to issue a 30x response.
  2. Unlike HTTP 200 Success or any of the HTTP Error status codes, a redirection requires two values: the status code and the location. API Gateway do not support VTL for JSON serialization in the header mappings; thus, in this case, you need to take advantage of how the Lambda runtime handles exceptions to pass these values in two separate fields back to API Gateway.

Node.JS: Using the exception name to store the redirection URI

API Gateway > Lambda high level diagram

The mapping from a Lambda function error to an API Gateway method response is defined by an integration response, which defines a selection pattern used to match the Lambda function errorMessage and routes it to an associated method response. In this example, you use the prefix-based error handling pattern:

API Gateway SettingsValue
Integration Response: Lambda Error Regex^HandlerDemo.ResponseFound.*
Method Response: HTTP Status302

Because you don’t have access to VTL in the API Gateway header mappings, retrieve your redirection URI from errorType.

API Gateway SettingsValue
Integration Response: Header MappingsLocation: integration.response.body.errorType

In the Lambda function Node.JS runtime, errorType can be assigned to any value. In this case, use it to store the redirection URI. In the handler.js, you have the following:

// Returns 302 or 301
var err = new Error("HandlerDemo.ResponseFound Redirection: Resource found elsewhere");
err.name = "http://a-different-uri";
context.done(err, {});

The same technique applies to 301 permanent redirects except you replace the API Gateway regex above to detect HandlerDemo.MovedPermanently.

Node.JS: Handling other response types

In order to keep it consistent, you can leverage the same pattern for other error codes. You can leverage errorType for any end user–visible messages.

//404 error
var err = new Error(“HandlerDemo.ResponseNotFound: Resource not found.”);
err.name = “Sorry, we can't find what you're looking for. Are you sure the address is correct?";
context.done(err, {});

Just as an example, upon failure, you return a “Fail Whale” HTML webpage to the user.

API Gateway SettingsValue
Integration Response: Lambda Error Regex^HandlerDemo.ResponseNotFound.*
Method Response: HTTP Status404
Integration Response: Content-Type (Body Mapping Templates)text/html
Integration Response: Template<html>

   <img src=”fail-whale.gif” />

   $input.path(‘$.errorType’)

</html>

HTTP 200 Success can be handled as you normally would in your handler.js:

//200 OK
context.done(null, response);

Java: Using the Java inner exception to store the redirection URI

API Gateway > Lambda high level diagram

Because Java is a strongly typed language, the errorType Lambda function return value contains the Java exception type which you cannot override (nor should you). Thus, you will be using a different property to retrieve the redirection URI:

API Gateway SettingsValue
Method Response: HTTP Status302
Integration Response: Header MappingsLocation: integration.response.body.cause.errorMessage

The cause.errorMessage parameter is accessible in the Lambda function Java runtime as an inner exception.

throw new Exception(new ResponseFound("http://www.amazon.com"));

ResponseFound is a class that extends Throwable.

package HandlerDemo;

public class ResponseFound extends Throwable {
  public ResponseFound(String uri) { super(uri); }
}

The full response from Lambda received by API Gateway is the following:

{
  "errorMessage": "HandlerDemo.ResponseFound: http://www.amazon.com",
  "errorType": "java.lang.Exception",
  "stackTrace": [ ... ],
  "cause": {
    "errorMessage": "http://www.amazon.com",
    "errorType": "HandlerDemo.ResponseFound",
    "stackTrace": [ ... ]
  }
}

Because Lambda serializes the inner exception type to the outer errorMessage. you can still use the same Lambda error regex in API Gateway to detect a redirect from errorMessage.

API Gateway SettingsValue
Integration Response: Lambda Error Regex^HandlerDemo.ResponseFound.*

Conclusion

In this post, I showed how to emit different HTTP responses, including a redirect response, on Amazon API Gateway and AWS Lambda for Node.JS and Java runtimes. I encourage developers to wrap these patterns into a helper class where it handles the inner working of returning a success, error, or redirection response.

One thing you may notice missing is an example of handling redirection in Python. At the time of writing, the Chalice community is considering support for this use case as part of the framework.

If you have questions or suggestions, please comment below.

Powering Secondary DNS in a VPC using AWS Lambda and Amazon Route 53 Private Hosted Zones

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/powering-secondary-dns-in-a-vpc-using-aws-lambda-and-amazon-route-53-private-hosted-zones/

Mark Statham, Senior Cloud Architect

When you implement hybrid connectivity between existing on-premises environments and AWS, there are a number of approaches to provide DNS resolution of both on-premises and VPC resources. In a hybrid scenario, you likely require resolution of on-premises resources, AWS services deployed in VPCs, AWS service endpoints, and your own resources created in your VPCs.

You can leverage Amazon Route 53 private hosted zones to provide private DNS zones for your VPC resources and dynamically register resources, as shown in a previous post, Building a Dynamic DNS for Route 53 using CloudWatch Events and Lambda.

Ultimately, this complex DNS resolution scenario requires that you deploy and manage additional DNS infrastructure, running on EC2 resources, into your VPC to handle DNS requests either from VPCs or on-premises. Whilst this is a familiar approach it adds additional cost and operational complexity, where a solution using AWS managed services can be used instead.

In this post, we explore how you can use leverage Route 53 private hosted zones with AWS Lambda and Amazon CloudWatch Events to mirror on-premises DNS zones which can then be natively resolved from within your VPCs, without the need for additional DNS forwarding resources.

Route 53 private hosted zones

Route 53 offers the convenience of domain name services without having to build a globally distributed highly reliable DNS infrastructure. It allows instances within your VPC to resolve the names of resources that run within your AWS environment. It also lets clients on the Internet resolve names of your public-facing resources. This is accomplished by querying resource record sets that reside within a Route 53 public or private hosted zone.

A private hosted zone is basically a container that holds information about how you want to route traffic for a domain and its subdomains within one or more VPCs and is only resolvable from the VPCs you specify; whereas a public hosted zone is a container that holds information about how you want to route traffic from the Internet.

Route 53 has a programmable API that can be used to automate the creation/removal of records sets which we’re going leverage later in this post.

Using Lambda with VPC support and scheduled events

AWS Lambda is a compute service where you can upload your code and the service runs the code on your behalf using AWS infrastructure. You can create a Lambda function and execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a cron expression. This functionality is underpinned by CloudWatch Events.

Lambda runs your function code securely within a VPC by default. However, to enable your Lambda function to access resources inside your private VPC, you must provide additional VPC-specific configuration information that includes VPC subnet IDs and security group IDs. Lambda uses this information to set up elastic network interfaces (ENIs) that enable your function to connect securely to other resources within your private VPC or reach back into your own network via AWS Direct Connect or VPN.

Each ENI is assigned a private IP address from the IP address range within the subnets that you specify, but is not assigned any public IP addresses. You cannot use an Internet gateway attached to your VPC, as that requires the ENI to have public IP addresses. Therefore, if your Lambda function requires Internet access, for example to access AWS APIs, you can use the Amazon VPC NAT gateway. Alternatively, you can leverage a proxy server to handle HTTPS calls, such as those used by the AWS SDK or CLI.

Building an example system

When you combine the power of Route 53 private hosted zones and Lambda, you can create a system that closely mimics the behavior of a stealth DNS to provide resolution of on-premises domains via VPC DNS.

For example, it is possible to schedule a Lambda function that executes every 15 minutes to perform a zone transfer from an on-premises DNS server, using a full zone transfer query (AXFR). The function can check the retrieved zone for differences from a previous version. Changes can then be populated into a Route 53 private hosted zone, which is only resolvable from within your VPCs, effectively mirroring the on-premises master to Route 53.

This then allows your resources deployed in VPC to use just VPC DNS to resolve on-premises, VPC and Internet resources records without the need for any additional forwarding infrastructure to on-premises DNS.

The following example is based on python code running as a Lambda function, invoked using CloudWatch Events with constant text to provide customizable parameters to support the mirroring of multiple zones for both forward and reverse domains.

Prerequisites for the example

Before you get started, make sure you have all the prerequisites in place including installing the AWS CLI and creating a VPC.

  • Region

    Check that the region where your VPC is deployed has the Lambda and CloudWatch Events services available.

  • AWS Command Line Interface (AWS CLI)

    This example makes use of the AWS CLI; however, all actions can be performed via the AWS console as well. Make sure you have the latest version installed, which provides support for creating Lambda functions in a VPC and the required IAM permissions to create resources required. For more information, see Getting Set Up with the AWS Command Line Interface.

  • VPC

    For this example, create or use a VPC configured with at least two private subnets in different Availability Zones and connectivity to the source DNS server. If you are building a new VPC, see Scenario 4: VPC with a Private Subnet Only and Hardware VPN Access.

    Ensure that the VPC has the DNS resolution and DNS hostnames options set to yes, and that you have both connectivity to your source DNS server and the ability to access the AWS APIs. You can create an AWS managed NAT gateway to provide Internet access to AWS APIs or as an alternative leverage a proxy server.

    You may wish to consider creating subnets specifically for your Lambda function, allowing you to restrict the IP address ranges that need access to the source DNS server and configure network access controls accordingly.

    After the subnets are created, take note of them as you’ll need them later to set up the Lambda function: they are in the format subnet-ab12cd34. You also need a security group to assign to the Lambda function; this can be the default security group for the VPC or one you create with limited outbound access to your source DNS: the format is sg-ab12cd34.

  • DNS server

    You need to make sure that you modify DNS zone transfer settings so that your DNS server accepts AXFR queries from the Lambda function. Also, ensure that security groups or firewall policies allow connection via TCP port 53 from the VPC subnet IP ranges created above.

Setting up the example Lambda function

Before you get started, it’s important to understand how the Lambda function works and interacts with the other AWS services and your network resources:

  1. The Lambda function is invoked by CloudWatch Events and configured based on a JSON string passed to the function. This sets a number of parameters, including the DNS domain, source DNS server, and Route 53 zone ID. This allows a single Lambda function to be reused for multiple zones.
  2. A new ENI is created in your VPC subnets and attached to the Lambda function; this allows your function to access your internal network resources based on the security group that you defined.
  3. The Lambda function then transfers the source DNS zone from the IP specified in the JSON parameters. You need to ensure that your DNS server is configured to allow full zone transfers and allow AXFR queries to your DNS server, which happens over TCP port 53.
  4. The Route 53 DNS zone is retrieved via API.
  5. The two zone files are compared; the resulting differences are returned as a set of actions to be performed against Route 53.
  6. Updates to the Route 53 zone are made via API and, finally, the SOA is updated to match the source version.

You’re now ready to set up the example using the following instructions.

Step 1 – Create a Route 53 hosted zone

Before you create the Lambda function, there needs to be a target Route 53 hosted zone to mirror the DNS zone records into. This can either be a public or private zone; however, for the purposes of this example, you will create a private hosted zone that only responds to queries from the VPC you specify.

To create a Route 53 private hosted zone associated with your VPC, provide the region and VPC ID as part of the following command:

aws route53 create-hosted-zone \
--name <domainname> \
--vpc VPCRegion=<region>,VPCId=<vpc-aa11bb22> \
--caller-reference mirror-dns-lambda \
--hosted-zone-config Comment="My DNS Domain"

Save the HostedZone Id returned, since you will need it for future steps.

Step 2 – Create an IAM role for the Lambda function

In this step, you use the AWS CLI to create the Identity and Access Management (IAM) role that the Lambda function assumes when the function is invoked. You need to create an IAM policy with the required permissions and then attach this policy to the role.

Download the mirror-dns-policy.json and mirror-dns-trust.json files from the aws-lambda-ddns-function AWS Labs GitHub repo.

mirror-dns-policy.json

The policy includes EC2 permissions to create and manage ENIs required for the Lambda function to access your VPC, and Route 53 permissions to list and create resource records. The policy also allows the function to create log groups and log events as per standard Lambda functions.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": "arn:aws:logs:*:*:*"
    }, {

        "Effect": "Allow",
        "Action": [
            "ec2:CreateNetworkInterface",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DetachNetworkInterface",
            "ec2:DeleteNetworkInterface"
        ],
        "Resource": "*"
    }, {
        "Sid": "Manage Route 53 records",
        "Effect": "Allow",
        "Action": [
            "route53:ChangeResourceRecordSets",
            "route53:ListResourceRecordSets"
        ],

        "Resource": ["*"]
    }]
}

To restrict the Lambda function access, you can control the scope of changes to Route 53 by specifying the hosted zones that are being managed in the format "arn:aws:route53:::hostedzone/Z148QEXAMPLE8V". This policy can be updated later if additional hosted zone IDs are added.

mirror-dns-trust.json

The mirror-dns-trust.json file contains the trust policy that grants the Lambda service permission to assume the role; this is standard for creating Lambda functions.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}

Create IAM entities

The next step is to create the following IAM entities for the Lambda function:

  • IAM policy

    Create the IAM policy using the policy document in the mirror-dns-policy.json file, replacing with the local path to the file. The output of the create-policy command includes the Amazon Resource Locator (ARN). Save the ARN, as you need it for future steps.

    aws iam create-policy \
    --policy-name mirror-dns-lambda-policy \
    --policy-document file://<LOCAL PATH>/mirror-dns-policy.json

  • IAM role

    Create the IAM role using the trust policy in the mirror-dns-trust.json file, replacing with the local path to the file. The output of the create-role command includes the ARN associated with the role that you created. Save this ARN, as you need it when you create the Lambda function in the next section.

    aws iam create-role \
    --role-name mirror-dns-lambda-role \
    --assume-role-policy-document file://<LOCAL PATH>/mirror-dns-trust.json

    Attach the policy to the role. Use the ARN returned when you created the IAM policy for the
    –policy-arn input parameter.

    aws iam attach-role-policy \
    --role-name mirror-dns-lambda-role \
    --policy-arn <enter-your-policy-arn-here>

Step 3 – Create the Lambda function

The Lambda function uses modules included in the Python 2.7 Standard Library and the AWS SDK for Python module (boto3), which is preinstalled as part of the Lambda service. Additionally, the function uses the dnspython module, which provides DNS handling functions, and there is also an externalized lookup function.

The additional libraries and functions require that we create a deployment package for this example as follows:

  1. Create a new directory for the Lambda function and download the Python scripts lambdafunction.py and lookuprdtype.py from the aws-lambda-ddns-function AWS Labs GitHub repo. Alternatively, clone the repo locally.
  2. Install the additional dnspython module locally using the pip command. This creates a copy of the require module local to the function.

    pip install dnspython -t .

  3. Update the lambda_function.py to specify proxy server configuration, if required.

  4. Create a Lambda deployment package using the following command:

    zip -rq mirror-dns-lambda.zip lambda_function.py \
    lookup_rdtype.py dns*

Then, you’ll use the AWS CLI to create the Lambda function and upload the deployment package by executing the following command to create the function. Note that you need to update the commands to use the ARN of the IAM role that you created earlier, as well as the local path to the Lambda deployment file containing the Python code for the Lambda function.

aws lambda create-function --function-name mirror-dns-lambda \
--runtime python2.7 \
--role <enter-your-role-arn-here> \
--handler lambda_function.lambda_handler \
--timeout 60 \
--vpc-config SubnetIds=comma-separated-vpc-subnet-ids,SecurityGroupIds=comma-separated-security-group-ids \
--memory-size 128 \
--description "DNS Mirror Function"
--zip-file fileb://<LOCAL PATH>/mirror-dns-lambda.zip

The output of the command returns the FunctionArn of the newly-created function. Save this ARN, as you need it in the next section.

Configure a test event in order to validate that your Lambda function works; it should be in JSON format similar to the following. All keys are required as well as values for Domain, MasterDns, and ZoneId.

{
    "Domain": "mydomain.com",
    "MasterDns": "10.0.0.1",
    "ZoneId": "AA11BB22CC33DD",
    "IgnoreTTL": "False",
    "ZoneSerial": ""
}

Invoke the Lambda function to test that everything is working; after the function has been invoked, check the file named output to see if the function has worked (you should see a 200 return code). Alternatively, you can test in the AWS console, using the test event to see the log output.

aws lambda invoke \
--function-name mirror-dns-lambda \
--payload fileb://event.json output

Congratulations, you’ve now created a secondary mirrored DNS accessible to your VPC without the need for any servers!

Step 4 – Create the CloudWatch Events rule

After you’ve confirmed that the Lambda function is executing correctly, you can create the CloudWatch Events rule that triggers the Lambda function on a scheduled basis. First, create a new rule with a unique name and schedule expression. You can create rules that self-trigger on schedule in CloudWatch Events, using cron or rate expressions. The following example uses a rate expression to run every 15 minutes.

aws events put-rule \
--name mirror-dns-lambda-rule \
--schedule-expression 'rate(15 minutes)'

The output of the command returns the ARN to the newly-created CloudWatch Events rule. Save the ARN, as you need it to associate the rule with the Lambda function and to set the appropriate Lambda permissions.

Next, add the permissions required for the CloudWatch Events rule to execute the Lambda function. Note that you need to provide a unique value for the –statement-id input parameter. You also need to provide the ARN of the CloudWatch Events rule that you created.

aws lambda add-permission \
--function-name mirror-dns-lambda \
--statement-id Scheduled01 \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn <enter-your-cloudwatch-events-rule-arn-here>

Finally, set the target of the rule to be the Lambda function. Because you are going to pass parameters via a JSON string, the value for –targets also needs to be in JSON format. You need to construct a file containing a unique identifier for the target, the ARN of the Lambda function previously created, and the constant text that contains the function parameters. An example targets.json file would look similar to the following; note that every quote(") in the Input value must be escaped.

[{
        "Id": "RuleStatementId01",
        "Arn": "<arn-of-lambda-function>",
        "Input": "{\"Domain\": \"mydomain.com\",\"MasterDns\": \"10.0.0.1\",\"ZoneId\": \"AA11BB22CC33DD\",\"IgnoreTTL\": \"False\",\"ZoneSerial\": \"\"}"
}]

Activate the scheduled event by adding the following target:

aws events put-targets \
--rule mirror-dns-lambda-rule \
--targets file://targets.json

Because a single rule can have multiple targets, every domain that you want to mirror can be defined as another target with a different set of parameters; change the ID and Input values in the target JSON file.

Conclusion

Now that you’ve seen how you can combine various AWS services to automate the mirroring of DNS to Amazon Route 53 hosted zones, we hope that you are inspired to create your own solutions using Lambda in a VPC to enable hybrid integration. Lambda allows you to create highly scalable serverless infrastructures that allow you to reduce cost and operational complexity, while providing high availability. Coupled with CloudWatch Events, you can respond to events in real-time, such as when an instance changes its state or when a customer event is pushed to the CloudWatch Events service.

To learn more about Lambda and serverless infrastructures, see the AWS Lambda Developer Guide and the " Microservices without the Servers" blog post. To learn more about CloudWatch Events, see Using CloudWatch Events in the Amazon CloudWatch Developer Guide.

We’ve open-sourced the code used in this example in the aws-lambda-ddns-function AWS Labs GitHub repo and can’t wait to see your feedback and your ideas about how to improve the solution.

AWS Serverless Chatbot Competition

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/aws-serverless-chatbot-competition/

Today, we are pleased to announce the official AWS Serverless Chatbot competition!

Bots on Slack can help your team be more productive and accomplish more tasks. They can help you increase visibility into your operations or help your customers easily get information through a natural, conversational interface. However, building and running bots can be a time-consuming and difficult task. Developers must provision, manage, and scale the compute resources that run the bot code.

With AWS Lambda, it’s easy to build and run bots. Upload your code and Lambda takes care of everything required to run and scale your code with high availability.

Enter your bot for a chance to win a ticket to re:Invent 2016 in Las Vegas!

Enter the competition by building a Slack bot that runs on AWS Lambda and Amazon API Gateway. You’re also encouraged to integrate Slack APIs or other APIs, SDKs, and datasets so long as you are authorized to use them.

We’ll be selecting 8 winners for a prize. Each prize includes one ticket to AWS re:Invent and access to discounted hotel room rates, along with recognition at the Serverless State of the Union address, some cool swag, $100 in AWS credits, and publicity opportunities for the winning bots.

To help you get started, we’ve created a few very basic bots that take advantage of Slack’s triggers and webhooks. These bots are built using AWS Lambda and Amazon API Gateway. To get started, check out the diagram below and view the code samples and instructions on GitHub at: aws-serverless-chatbot-sample.

Here’s what you need to do to enter:

  1. Read the Rules and Eligibility Guidelines.
  2. Register for the AWS Serverless Chatbot Competition.
  3. Create AWS and Slack developer accounts.
  4. Visit the Resources Page to learn more about the APIs and services.
  5. Build your chatbot. Our sample code (aws-serverless-chatbot-sample) is a good place to start.
  6. Create your demo video and other materials for the submission.
  7. Submit your materials before 5 PM ET on September 29, 2016.

Serverless Cross Account Stream Replication Using AWS Lambda, Amazon DynamoDB, and Amazon Kinesis Firehose

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/serverless-cross-account-stream-replication-using-aws-lambda-amazon-dynamodb-and-amazon-kinesis-firehose/

This is a guest post by Richard Freeman, Ph.D., a solutions architect and data scientist at JustGiving. JustGiving in their own words: We are one of the world’s largest social platforms for giving that’s helped 27 million registered users in 196 countries raise $4.1 billion for over 27,000 good causes.”

At JustGiving, we want our analysts and data scientist to have access to production Amazon Kinesis data in near real-time and have the flexibility to make transformations, but without compromising on security. Rather than build a custom service using the Kinesis Client Library (KCL) or maintain an Amazon EC2 instance running with a Java Kinesis Agent, we used an AWS Lambda function that does not require us to maintain a running server. The Lambda function is used to process Amazon Kinesis events, enrich them, and write them to Amazon DynamoDB in another AWS account.

After the data is stored in DynamoDB, further systems can process the data as a stream; we persist the data to S3 via Amazon Kinesis Firehose using another Lambda function. This gives us a truly serverless environment, where all the infrastructure including the integration, connectors, security, and scalability is managed by AWS, and allows us to focus only on the stream transformation logic rather than on code deployment, systems integration, or platform.

This post shows you how to process a stream of Amazon Kinesis events in one AWS account, and persist it into a DynamoDB table and Amazon S3 bucket in another account, using only fully managed AWS services and without running any servers or clusters. For this walkthrough, I assume that you are familiar with DynamoDB, Amazon Kinesis, Lambda, IAM, and Python.

Overview

Amazon Kinesis Streams is a stream processing engine that can continuously capture and store terabytes of data per hour from hundreds of thousands of sources, such as website clickstreams, financial transactions, social media feeds, web logs, sensors, and location-tracking events.

Working with a production environment and a proof of concept (POC) environment that are in different AWS accounts is useful if you have web analytics traffic written to Amazon Kinesis in production and data scientists need near real-time and historical access to the data in another non-operational, non-critical, and more open AWS environment for experiments. In addition, you may want to persist the Amazon Kinesis records in DynamoDB without duplicates.

The following diagram shows how the two environments are structured.

The production environment contains multiple web servers running data producers that write web events to Amazon Kinesis, which causes a Lambda function to be invoked with a batch of records. The Lambda function (1) assumes a role in the POC account, and continuously writes those events to a DynamoDB table without duplicates.

After the data is persisted in DynamoDB, further processing can be triggered via DynamoDB Streams and invoke a Lambda function (2) that could be used to perform streaming analytics on the events or to persist the data to S3 (as described in this post). Note that the Lambda function (1) can also be combined with what the function (2) does, but we wanted to show how to process both Amazon Kinesis stream and DynamoDB Streams data sources. Also, having the second function (2) allows data scientists to project, transform, and enrich the stream for their requirements while preserving the original raw data stream.

Setting up the POC environment

The POC environment is where we run data science experiments, and is the target environment for the production Amazon Kinesis data. For this example, give the POC the account number: 999999999999.

Create a target table in the POC account

This DynamoDB table will be where the Kinesis events will be written to. To create our table we will show an example using the AWS console:

  1. Open the DynamoDB console.
  2. On the navigation pane, choose Create Table and configure the following:
    • For Table name, enter prod-raven-lambda-webanalytics-crossaccount.
    • For Primary key (hash/partition key), enter seqNumbershardIdHash and choose String.
    • Choose Add sort key.
    • For Sort key Name (range), enter sequenceNumber and choose String.
  3. Under Table settings , clear Use default settings.
  4. Choose Create.
  5. Under Tables , select the table prod-raven-lambda-webanalytics-crossaccount and configure the following:
    • Choose Capacity.
    • Under Provisioned Capacity :
    • For Read capacity units , enter 20.
    • For Write capacity units , enter 20.
    • Choose Save.

Notes:

  • Optionally, and depending on your records, you can also create a global secondary index if you want to search a subset of the records by user ID or email address – this is very useful for QA in development and staging environments.
  • Provisioned throughput capacity should be set based n the amount of expected events that will be read and written per second. If the reads or writes are above this capacity for a period of time, then throttling occurs, and writes must be retried. There are many strategies to deal with this, including having the capacity set dynamically depending on the current consumed read or write of events, or having exponential backoff retry logic.
  • DynamoDB is a schemaless database, so no other fields or columns need to be specified up front. Each item in the table can have a different number of elements.

Create an IAM policy in the POC account

First, create an IAM role that can be used by the Lambda functions to write to the DynamoDB table.

  1. In the POC account, pen the IAM console.
  2. On the navigation pane, choose Policies , Create Policy.
  3. Choose Create Your Own Policy and configure the following:
    • For Policy Name , enter prod-raven-lambda-webanalytics-crossaccount.
    • For Description , enter “Read/write access to the POC prod-raven-lambda-webanalytics-crossaccount table” or similar text.
    • For Policy Document , insert the following JSON:
{    
    "Version": "2012-10-17",    
    "Statement": [        
        {            
            "Sid": "Stmt345926348000",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:DescribeStream",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:GetRecords",
                "dynamodb:ListTables",
                "dynamodb:PutItem",
                "dynamodb:Query",
                "dynamodb:Scan",
                "dynamodb:UpdateItem",
                "dynamodb:UpdateTable"],
            "Resource": [ "arn:aws:dynamodb:<region>:<999999999999>:table/prod-raven-lambda-webanalytics-crossaccount" ]
        }    
    ]
}

The action array list specifies the allowable actions on the specified resource. Here, it is the DynamoDB table created earlier. Replace with your region, e.g., eu-west-1 and with your AWS account ID.

Create an IAM role

Next, create an IAM role which uses this policy, so that Lambda can assume an identity with the indicated privileges.

  1. Open the IAM console, choose Roles , Create New Role.
    • For Role Name , enter DynamoDB-ProdAnalyticsWrite-role.
    • Choose Role for Cross Account Access to provide access between AWS accounts you own.
    • For Establish Trust , enter the production AWS account ID, e.g., 111111111111.
    • Select the policy created earlier, prod-raven-lambda-webanalytics-crossaccount.
  2. Choose Review , Create Role.

This role will be created with an Amazon Resource Notation ID (ARN), e.g. arn:aws:iam::999999999999:role/DynamoDB-ProdAnalyticsWrite-role. You will now see the policy is attached and the trusted account ID is 111111111111.

Setting up the production environment

In the production environment, you use a Lambda function to read, parse, and transform the Amazon Kinesis records and write them to DynamoDB. Access to Amazon Kinesis and DynamoDB is fully managed through IAM policies and roles. At this point, you need to sign out of the POC account and sign in as a user with IAM administration rights in the production account 111111111111.

Create a Lambda policy

Create a policy that allows a production user to assume the role of a user in the POC account .

  1. In the production environment, open the IAM console and choose Policies.
  2. Choose Create Policy , Create Your Own Policy.
    1. For Policy Name , enter prod-raven-lambda-assumerole.
    2. For Description , enter “This policy allows the Lambda function to execute, write to CloudWatch and assume a role in POC” or similar text.
    3. For Policy Document , insert the following JSON:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:GetRecords",
                "kinesis:GetShardIterator",
                "kinesis:DescribeStream",
                "kinesis:ListStreams",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        },
        {
            "Sid": "",
            "Resource": "*",
            "Action": [
                "logs:*"
            ],
            "Effect": "Allow"
        },
        {
            "Sid": "Stmt1435680952001",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "iam:GenerateCredentialReport",
                "iam:Get*",
                "iam:List*"
            ],
            "Resource": [
            "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<999999999999>:role/DynamoDB-ProdAnalyticsWrite-role"
        }
    ]
}

Create a Lambda execution IAM role

  1. In the IAM console, choose Roles , Create New Role.
  2. For Set Role Name , enter LambdaKinesisDynamo and choose Next step.
  3. For Role Type , choose AWS Service Roles , AWS Lambda.
  4. Select the policy created earlier, prod-raven-lambda-assumerole** , and choose Next step **.
  5. Choose Review , Create Role.

Create the processor Lambda function

  1. 1. Open the Lambda console and choose Create a Lambda Function.
  2. 2. Select the blueprint kinesis-process-record-python.
  3. 3. Configure the event source:
    • For Event Source type , choose Kinesis.
    • For Kinesis Stream , select web-analytics (or your stream) .
    • For Batch Size , enter 300 (this depends on the frequency that events are added to Amazon Kinesis).
    • For Starting position , choose Trim horizon.
  4. 4. Configure the function:
    • For Name , enter ProdKinesisToPocDynamo.
    • For Runtime , choose Python 2.7.
    • For Edit code inline , add the Lambda function source code (supplied in the next section).
    • For Handler , choose lambdafunction.lambdahandler.
    • For Role , choose the role that you just created, LambdaKinesisDynamo.
    • For Memory ,choose 128 (this depends on the frequency that events are added to Amazon Kinesis)
    • For Timeout , choose 2min 0 sec (this depends on the frequency that events are added to Amazon Kinesis).
  5. 5. Review the configuration:
    • For Enable event source , choose Enable later (this can be enabled later via the Event sources tab).
  6. 6. Choose Create function.

Create the Lambda function code

At the time of this post, Lambda functions support the Python, Node.js, and Java languages. For this task, you implement the function in Python with the following basic logic:

  • Assume a role that allows the Lambda function to write to the DynamoDB table. Manually configure the DynamoDB client to use the newly-assumed cross account role.
  • Convert Amazon Kinesis events to the DynamoDB format.
  • Write the record to the DynamoDB table.

Assume a cross account role for access to the POC account DynamoDB table

This code snippet uses the AWS Security Token Service (AWS STS) which enables the creation of temporary IAM credentials for an IAM role. The assume_role() function returns a set of temporary credentials (consisting of an access key ID, a secret access key, and a security token) that can be used to access the DynamoDB table via the IAM role arn:aws:iam::999999999999:role/DynamoDB-ProdAnalyticsWrite-role.

Assume the role

import base64, json, boto3

def lambda_handler(event, context):
    client = boto3.client('sts')
    sts_response = client.assume_role(RoleArn='arn:aws:iam::<999999999999>:role/DynamoDB-PrdAnalyticsWrite-role',                              
                                      RoleSessionName='AssumePocRole', DurationSeconds=900)

Configure the DynamoDB client using a cross account role

    dynamodb = boto3.resource(service_name='dynamodb', region_name=<region>,
                              aws_access_key_id = sts_response['Credentials']['AccessKeyId'],
                              aws_secret_access_key = sts_response['Credentials']['SecretAccessKey',
                              aws_session_token = sts_response['Credentials']['SessionToken'])

You can now use any DynamoDB methods in the AWS SDK for Python (Boto 3) to write to DynamoDB tables in the POC environment, from the production environment.

In order to test the code, I recommend that you create a test event. The simplest way to do this is to print the event from Amazon Kinesis, copy it from the Lambda function to CloudWatch Logs, and convert it into valid JSON. You can now use it when you select the Lambda function and choose Actions , Configure Test Event. After you are happy with the records in DynamoDB, you can choose Event Source,** Disable** to disable the mapping between the Lambda function and Amazon Kinesis.

Here’s the Python code to write an Amazon Kinesis record to the DynamoDB table:

    tableName = 'prod-raven-lambda-webanalytics'
    for record in event['Records']:
        try:            
            payload_json = json.loads(base64.b64decode(record['kinesis']['data']))
            payload_json['sequenceNumber'] = record['kinesis']['sequenceNumber']
            payload_json['shardId'] = record['eventID'].split(':')[0]
            payload_json['seqNumbershardIdHash'] = payload_json['sequenceNumber'][-3:-1]+'_'+ payload_json['shardId']
            items={}
            for k, v in payload_json.items():
                if v is not None and v != '':
                    items[k] = v
            table = dynamodb.Table(tableName)
            response  = table.put_item(Item=items)
        except Exception as e:
            print(e.__doc__)
            print(e.message)
    return 'Successfully processed {} records.'.format(len(event['Records']))

The first part of the source snippet iterates over the batch of events that were sent to the Lambda function as part of the Amazon Kinesis event source, and decodes from base64 encoding. Then the fields used for the DynamoDB hash and range are extracted directly from the Amazon Kinesis record. This helps avoid storing duplicate records or idempotent processing, and allows for rapid retrieval without the need to do a full table scan (more on this later). Each record is then converted into the DynamoDB format, and finally written to the target DynamoDB table with the put_item()function.

As the data is now persisted in the POC environment, data scientists and analysts are free to experiment on the latest production data without having access to production or worrying about impacting the live website.

Records in the POC DynamoDB table

Now I’ll give technical details on how you can ensure that the DynamoDB records are idempotent, and different options for querying the data. The pattern I describe can be more widely used as it will work with any Amazon Kinesis records. The idea is to use the Amazon Kinesis sequence number, which is unique per stream, as the range, and a subset of it as a composite hash. In our experiments, we found that the taking the last two minus one digits (here, the 80 in red) from the sequence number was better distributed than the last two (02).

This schema ensures that you do not have duplicate Amazon Kinesis records without the need to maintain any state or position in the client, or change the records in any way, which is well suited for a Lambda function. For example if you deleted the Amazon Kinesis event source of the Lambda function and added a new one an hour later with a trim horizon, then this would mean that the existing DynamoDB rows would be overwritten, as the records would have identical hash and range keys. Not having duplicate records is very useful for any incremental loads into other systems and reduces the deduplication work needed downstream, and also allows us to readily replay Amazon Kinesis records.

An example of a populated DynamoDB table could look like the following graphic.

Notice that the composite key of hash and range is unique and that other fields captured can be useful in understanding the user click stream. For example you can see that if you run a query on attribute useremail_ and order by nanotimestamp_, you obtain the clickstream of an individual user. However, this means that DynamoDB has to analyze the whole table in what is known as a table scan, as you can only query on hash and range keys.

One solution is to use a global secondary index, as discussed earlier; however, you might suffer from the data writes being throttled in the main table should the index writes be throttled (AWS is working on an asynchronous update process), This can happen if the hash/range are not well distributed. If you need to use the index, one solution is to use a scalable pattern with global secondary indexes.

Another way is to use DynamoDB Streams as an event source to trigger another Lambda function that writes to a table specifically used for querying by user email, e.g., the hash could be useremail_ and the range could be nanotimestamp_. This also means that you are no longer limited to five indexes per main table and the throttling issues mentioned earlier do not affect the main table.

Using DynamoDB Streams is also more analytics-centric as it gives you dynamic flexibility on the primary hash and range, allowing data scientists to experiment with different configurations. In addition, you might use a Lambda function to write the records to one table for the current week and another one for the following week; this allows you to keep the most recent hot data in a smaller table, and older or colder data in separate tables that could be removed or archived on a regular basis.

Persisting records to S3 via Amazon Kinesis Firehose

Now that you have the data in DynamoDB, you can enable DynamoDB Streams to obtain a stream of events which can be fully sourced and processed in the POC. For example, you could perform time series analysis on this data or persist the data to S3 or Amazon Redshift for batch processing.

To write to S3, use a Lambda function that reads the records from DynamoDB Streams and writes them to Amazon Kinesis Firehose. Other similar approaches exist but they rely on having a running server with custom deployed code. Firehose is a fully managed service for delivering real-time streaming data to destinations such as S3.

First, set up an IAM policy and associate it with an IAM role so that the Lambda function can write to Firehose.

Create an IAM policy

  1. Open the IAM console.
  2. On the navigation pane, choose Policies , Create Policy.
  3. Choose Create Your Own Policy and configure the following:
    • For Policy Name , enter poc-raven-lambda-firehose.
    • For Description , enter Read access to DynamoDB, Put records to Kinesis Firehose, and Lambda execution rights.
    • For Policy Document , insert the following JSON:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:DescribeStream",
                "dynamodb:ListStreams",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "firehose:DescribeDeliveryStream",
                "firehose:ListDeliveryStreams",
                "firehose:PutRecord",
                "firehose:PutRecordBatch",
                "firehose:UpdateDestination"
            ],
            "Resource": [
                "*"
            ]
        }

    ]
}

Next, create an IAM role which uses this policy, so that Lambda can assume an identity with the required privileges of reading from DynamoDB Streams, writing to Amazon Kinesis streams.

Create an IAM role

  1. Open the IAM console.
  2. On the navigation pane, choose Roles , Create New Role.
  3. For Set Role Name , enter lambdadynamostreams_firehose and choose Next step.
  4. For Role Type , choose AWS Service Roles , AWS Lambda.
  5. Select the policy created earlier, poc-raven-lambda-firehose** , and choose Next step **.
  6. Choose Review , Create Role.

Create the Lambda function

  1. Open the Lambda console.
  2. Choose Create a Lambda function and select the dynamo-process-stream-python blueprint.
  3. Configure the event sources:
    • For Event Source type , choose DynamoDB.
    • For DynamoDB Table , enter prod-raven-lambda-webanalytics.
    • For Batch Size , enter 300 (this depends on the frequency that events are added to DynamoDB).
    • For Starting position , choose Trim horizon.
  4. Configure the function:
    • For Name , enter PocLambdaFirehose.
      -For Runtime , choose Python 2.7.
      -For Edit code inline , add the Lambda function source code (see code below).
    • For Handler , choose lambdafunction.lambdahandler.
    • For Role , select the role you created earlier, e.g., lambdadynamostreams_firehose. This grants the Lambda function access to DynamoDB Streams, Firehose, and CloudWatch metrics and logs.
    • For Memory (MB), choose 128.
    • For Timeout , enter 5 min 0 sec.
  5. Review
    • For Enable event source , choose Enable later (this can be enabled later via the Event sources tab).
    • Choose Create function.
from __future__ import print_function
import json
import boto3

def lambda_handler(event, context):
    firehose_client = boto3.client(service_name='firehose', region_name='eu-west-1')
    stream_name='stg-kinesis'
    for record in event['Records']:
        output_records = {}
        for key, value in record['dynamodb']['NewImage'].iteritems():
            if value.keys()[0] in ['S', 'N', 'B']:
                output_records[key]=value.values()[0]
        kinesis_data="".join([json.dumps(output_records),"\n"])        
        put_record(stream_name, kinesis_data, firehose_client)
    return 'done'

def put_record(stream_name, data, client):    
    client.put_record(DeliveryStreamName=stream_name,
                      Record={'Data':data}) 

First, the Lambda function iterates over all records and values, and flattens the JSON data structure from the DynamoDB format to JSON records, e.g., ‘event’:{‘S’:’page view’}becomes{‘event’:’page view’}. Then, the record is then encoded as a JSON record and sent to Firehose.

For testing, you can log events from your Amazon Kinesis stream by printing them as JSON to standard out; Lambda automatically delivers this information into CloudWatch Logs. This information can then be used as the test event in the Lambda console.

Configuring the Amazon Kinesis Firehose delivery stream

  1. Open the Amazon Kinesis Firehose console.
  2. Choose Create Delivery Stream.
    • For Destination , choose Amazon S3.
    • For Delivery stream name , enter: prod-s3-firehose.
    • For S3 bucket , choose Create new S3 bucket or Use an existing S3 bucket.
    • For S3 prefix , enter prod-kinesis-data.
  3. Choose Next to change the following Configuration settings.
    • For Buffer size , enter 5.
    • For Buffer interval , enter 300.
    • For Data compression , choose UNCOMPRESSED.
    • For Data Encryption , choose No Encryption.
    • For Error Logging , choose Enable.
    • For IAM role , choose Firehose Delivery IAM Role. This opens a new window: Amazon Kinesis Firehose is requesting permission to use resources in your account.
    • For IAM Role , choose Create a new IAM role.
    • For Role Name, choose firehosedeliveryrole.
  4. After the policy document is automatically generated, choose Allow.
  5. Choose Next.
  6. Review your settings and choose Create Delivery Stream.

After the Firehose delivery stream is created, select the stream and choose Monitoring to see activity. The following graphic shows some example metrics.

After the DynamoDB Streams event source for the Lambda function is enabled, new objects with production Amazon Kinesis records passing through the POC DynamoDB table are automatically created in the POC S3 bucket using Firehose. Firehose can also be used to import data to Amazon Redshift or Amazon Elasticsearch Service.

Benefits of serverless processing

Using a Lambda function for stream processing enables the following benefits:

  • Flexibility – Analytics code can be changed on the fly.
  • High availability – Runs in multiple Availability Zones in a region
  • Zero-maintenance and upgrade of the running instances, all services. are supported by AWS
  • Security – No use of keys or passwords. IAM roles can be used to integrate with Amazon Kinesis Streams, Amazon Kinesis Firehose, and DynamoDB.
  • Serverless compute – There are no EC2 instances or clusters to set up and maintain.
  • Automatic scaling – The number of Lambda functions invoked changes depending on the number of writes to Amazon Kinesis. Note that the upper concurrent limit is the number of Amazon Kinesis streams and DynamoDB Streams shards.
  • Low cost – You pay only for the execution time of the Lambda function.
  • Ease of use – Easy selection of language for functions, and very simple functions. In these examples, the functions just iterate over and parse JSON records.

Summary

In this post, I described how to persist events from an Amazon Kinesis stream into DynamoDB in another AWS account, without needing to build and host your own server infrastructure or using any keys. I also showed how to get the most out of DynamoDB, and proposed a schema that eliminates duplicate Amazon Kinesis records and reduces the need for further de-duplication downstream.

In a few lines of Python code, this pattern has allowed the data engineers at JustGiving the flexibility to project, transform, and enrich the Amazon Kinesis stream events as they arrive. Our data scientists and analysts now have full access to the production data in near-real time but in a different AWS account, allowing them to quickly undertake streaming analysis, project a subset of the data, and run experiments.

I also showed how to persist that data stream to S3 using Amazon Kinesis Firehose. This allows us to quickly access the historical clickstream dataset, without needing a permanent server or cluster running.

If you have questions or suggestions, please comment below. You may also be interested in seeing other material that digs deeper into the capabilities of Lambda and DynamoDB.

AWS Lambda Now Available in Singapore!

Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/aws-lambda-now-available-in-singapore/

We are happy to announce the next step in our regional launch plan has been completed: AWS Lambda is now available in Singapore. With yesterday’s launch, we now have both Amazon API Gateway and AWS Lambda available in Singapore, so you can now use both services together in seven regions:

Stay tuned for additional regions, and check out some of our recent blog posts for useful tips and tricks: