Join us this month to learn about AWS services and solutions. New this month, we have a fireside chat with the GM of Amazon WorkSpaces and our 2nd episode of the “How to re:Invent” series. We’ll also cover best practices, deep dives, use cases and more! Join us and register today!
AWS re:Invent June 13, 2018 | 05:00 PM – 05:30 PM PT – Episode 2: AWS re:Invent Breakout Content Secret Sauce – Hear from one of our own AWS content experts as we dive deep into the re:Invent content strategy and how we maintain a high bar. Compute
Containers June 25, 2018 | 09:00 AM – 09:45 AM PT – Running Kubernetes on AWS – Learn about the basics of running Kubernetes on AWS including how setup masters, networking, security, and add auto-scaling to your cluster.
June 19, 2018 | 11:00 AM – 11:45 AM PT – Launch AWS Faster using Automated Landing Zones – Learn how the AWS Landing Zone can automate the set up of best practice baselines when setting up new
June 21, 2018 | 01:00 PM – 01:45 PM PT – Enabling New Retail Customer Experiences with Big Data – Learn how AWS can help retailers realize actual value from their big data and deliver on differentiated retail customer experiences.
June 28, 2018 | 01:00 PM – 01:45 PM PT – Fireside Chat: End User Collaboration on AWS – Learn how End User Compute services can help you deliver access to desktops and applications anywhere, anytime, using any device. IoT
June 27, 2018 | 11:00 AM – 11:45 AM PT – AWS IoT in the Connected Home – Learn how to use AWS IoT to build innovative Connected Home products.
Mobile June 25, 2018 | 11:00 AM – 11:45 AM PT – Drive User Engagement with Amazon Pinpoint – Learn how Amazon Pinpoint simplifies and streamlines effective user engagement.
June 26, 2018 | 11:00 AM – 11:45 AM PT – Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway – Learn how you can reduce your on-premises infrastructure by using the AWS Storage Gateway to connecting your applications to the scalable and reliable AWS storage services. June 27, 2018 | 01:00 PM – 01:45 PM PT – Changing the Game: Extending Compute Capabilities to the Edge – Discover how to change the game for IIoT and edge analytics applications with AWS Snowball Edge plus enhanced Compute instances. June 28, 2018 | 11:00 AM – 11:45 AM PT – Big Data and Analytics Workloads on Amazon EFS – Get best practices and deployment advice for running big data and analytics workloads on Amazon EFS.
Join us this month to learn about some of the exciting new services and solution best practices at AWS. We also have our first re:Invent 2018 webinar series, “How to re:Invent”. Sign up now to learn more, we look forward to seeing you.
Note – All sessions are free and in Pacific Time.
Tech talks featured this month:
Analytics & Big Data
May 21, 2018 | 11:00 AM – 11:45 AM PT – Integrating Amazon Elasticsearch with your DevOps Tooling – Learn how you can easily integrate Amazon Elasticsearch Service into your DevOps tooling and gain valuable insight from your log data.
May 24, 2018 | 11:00 AM – 11:45 AM PT – Data Transformation Patterns in AWS – Discover how to perform common data transformations on the AWS Data Lake.
May 30, 2018 | 01:00 PM – 01:45 PM PT – Accelerating Life Sciences with HPC on AWS – Learn how you can accelerate your Life Sciences research workloads by harnessing the power of high performance computing on AWS.
Containers
May 24, 2018 | 01:00 PM – 01:45 PM PT –Building Microservices with the 12 Factor App Pattern on AWS – Learn best practices for building containerized microservices on AWS, and how traditional software design patterns evolve in the context of containers.
Databases
May 21, 2018 | 01:00 PM – 01:45 PM PT – How to Migrate from Cassandra to Amazon DynamoDB – Get the benefits, best practices and guides on how to migrate your Cassandra databases to Amazon DynamoDB.
May 23, 2018 | 01:00 PM – 01:45 PM PT – 5 Hacks for Optimizing MySQL in the Cloud – Learn how to optimize your MySQL databases for high availability, performance, and disaster resilience using RDS.
DevOps
May 23, 2018 | 09:00 AM – 09:45 AM PT – .NET Serverless Development on AWS – Learn how to build a modern serverless application in .NET Core 2.0.
Enterprise & Hybrid
May 22, 2018 | 11:00 AM – 11:45 AM PT – Hybrid Cloud Customer Use Cases on AWS – Learn how customers are leveraging AWS hybrid cloud capabilities to easily extend their datacenter capacity, deliver new services and applications, and ensure business continuity and disaster recovery.
IoT
May 31, 2018 | 11:00 AM – 11:45 AM PT – Using AWS IoT for Industrial Applications – Discover how you can quickly onboard your fleet of connected devices, keep them secure, and build predictive analytics with AWS IoT.
Machine Learning
May 22, 2018 | 09:00 AM – 09:45 AM PT – Using Apache Spark with Amazon SageMaker – Discover how to use Apache Spark with Amazon SageMaker for training jobs and application integration.
May 24, 2018 | 09:00 AM – 09:45 AM PT – Introducing AWS DeepLens – Learn how AWS DeepLens provides a new way for developers to learn machine learning by pairing the physical device with a broad set of tutorials, examples, source code, and integration with familiar AWS services.
May 30, 2018 | 09:00 AM – 09:45 AM PT– Introducing AWS Certificate Manager Private Certificate Authority (CA) – Learn how AWS Certificate Manager (ACM) Private Certificate Authority (CA), a managed private CA service, helps you easily and securely manage the lifecycle of your private certificates.
June 1, 2018 | 09:00 AM – 09:45 AM PT – Introducing AWS Firewall Manager – Centrally configure and manage AWS WAF rules across your accounts and applications.
May 30, 2018 | 11:00 AM – 11:45 AM PT – Accelerate Productivity by Computing at the Edge – Learn how AWS Snowball Edge support for compute instances helps accelerate data transfers, execute custom applications, and reduce overall storage costs.
This post courtesy of Paul Johnston, AWS Senior Developer Advocate – Serverless
Welcome to the first edition of the AWS Serverless ICYMI (In case you missed it) quarterly recap! Every quarter we’ll share all of the most recent product launches, feature enhancements, blog posts, webinars, Twitch live streams, and other interesting things that you might have missed!
Alexa Random Restaurant – Python-based backend for an Alexa skill that returns an open restaurant in a specified city using the Yelp API. Published by: Harsha Warrdhan Sharma
Podless – A serverless application that downloads podcasts to an S3 bucket. Published by: Stilvoid
Crypto-monitor – Collect and store crypto currency prices and send yourself an alert if one changes significantly. Published by: Drew Dresser
DailyDoggo – Send a daily link to a random dog picture to a phone number, via AWS Lambda and SNS. Published by: Kevin McCandless
These runtimes give Lambda developers and development teams even greater options for coding serverless, on-demand, compute solutions.
The AWS SAM 1.4.0 release was one of its biggest. The release added features for configuring many aspects of Amazon API Gateway, including CORS support, regional endpoints, binary media types, and stage settings. It also included per function concurrency support, tags and TableName for SimpleTable, and many documentation updates. Check out the release notes for the full list!
AppSync came out of the whitelisted preview and added a whole bunch of new features:
Here are the three webinars we delivered in Q1. We hold several Serverless webinars throughout the year, so look out for them in the Serverless section of the AWS Online Tech Talks page:
Keep an eye on AWS on Twitch for more Serverless videos and on the Join us on the Twitch AWS page for information about upcoming broadcasts and recent live streams.
Case studies
We’ve published several new case studies this quarter to help you with understanding how other organizations are using serverless technologies:
If you haven’t read the AWS Well Architected Framework Serverless Application Lens document, then it’s worth taking the time to do so. The document covers common serverless applications scenarios and identifies key elements to ensure that your workloads are architected according to best practices.
From now on, if you find issues with documentation we have open-sourced, you can tell us via a Pull Request rather than tweeting or emailing us. The current available serverless repositories are here:
We’re always looking to help people start learning how to build serverless applications. Our serverless web application workshops are online and you can do the hands-on labs yourself: Build a Serverless web application
Still looking for more?
The Serverless landing page has lots of information including a resources page containing case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials. Check it out!
If you build (or want to build) data-driven web and mobile apps and need real-time updates and the ability to work offline, you should take a look at AWS AppSync. Announced in preview form at AWS re:Invent 2017 and described in depth here, AWS AppSync is designed for use in iOS, Android, JavaScript, and React Native apps. AWS AppSync is built around GraphQL, an open, standardized query language that makes it easy for your applications to request the precise data that they need from the cloud.
I’m happy to announce that the preview period is over and that AWS AppSync is now generally available and production-ready, with six new features that will simplify and streamline your application development process:
Console Log Access – You can now see the CloudWatch Logs entries that are created when you test your GraphQL queries, mutations, and subscriptions from within the AWS AppSync Console.
Console Testing with Mock Data – You can now create and use mock context objects in the console for testing purposes.
Subscription Resolvers – You can now create resolvers for AWS AppSync subscription requests, just as you can already do for query and mutate requests.
Batch GraphQL Operations for DynamoDB – You can now make use of DynamoDB’s batch operations (BatchGetItem and BatchWriteItem) across one or more tables. in your resolver functions.
CloudWatch Support – You can now use Amazon CloudWatch Metrics and CloudWatch Logs to monitor calls to the AWS AppSync APIs.
CloudFormation Support – You can now define your schemas, data sources, and resolvers using AWS CloudFormation templates.
A Brief AppSync Review Before diving in to the new features, let’s review the process of creating an AWS AppSync API, starting from the console. I click Create API to begin:
I enter a name for my API and (for demo purposes) choose to use the Sample schema:
The schema defines a collection of GraphQL object types. Each object type has a set of fields, with optional arguments:
If I was creating an API of my own I would enter my schema at this point. Since I am using the sample, I don’t need to do this. Either way, I click on Create to proceed:
The GraphQL schema type defines the entry points for the operations on the data. All of the data stored on behalf of a particular schema must be accessible using a path that begins at one of these entry points. The console provides me with an endpoint and key for my API:
It also provides me with guidance and a set of fully functional sample apps that I can clone:
When I clicked Create, AWS AppSync created a pair of Amazon DynamoDB tables for me. I can click Data Sources to see them:
I can also see and modify my schema, issue queries, and modify an assortment of settings for my API.
Let’s take a quick look at each new feature…
Console Log Access The AWS AppSync Console already allows me to issue queries and to see the results, and now provides access to relevant log entries.In order to see the entries, I must enable logs (as detailed below), open up the LOGS, and check the checkbox. Here’s a simple mutation query that adds a new event. I enter the query and click the arrow to test it:
I can click VIEW IN CLOUDWATCH for a more detailed view:
Console Testing with Mock Data You can now create a context object in the console where it will be passed to one of your resolvers for testing purposes. I’ll add a testResolver item to my schema:
Then I locate it on the right-hand side of the Schema page and click Attach:
I choose a data source (this is for testing and the actual source will not be accessed), and use the Put item mapping template:
Then I click Select test context, choose Create New Context, assign a name to my test content, and click Save (as you can see, the test context contains the arguments from the query along with values to be returned for each field of the result):
After I save the new Resolver, I click Test to see the request and the response:
Subscription Resolvers Your AWS AppSync application can monitor changes to any data source using the @aws_subscribe GraphQL schema directive and defining a Subscription type. The AWS AppSync client SDK connects to AWS AppSync using MQTT over Websockets and the application is notified after each mutation. You can now attach resolvers (which convert GraphQL payloads into the protocol needed by the underlying storage system) to your subscription fields and perform authorization checks when clients attempt to connect. This allows you to perform the same fine grained authorization routines across queries, mutations, and subscriptions.
Batch GraphQL Operations Your resolvers can now make use of DynamoDB batch operations that span one or more tables in a region. This allows you to use a list of keys in a single query, read records multiple tables, write records in bulk to multiple tables, and conditionally write or delete related records across multiple tables.
In order to use this feature the IAM role that you use to access your tables must grant access to DynamoDB’s BatchGetItem and BatchPutItem functions.
CloudWatch Logs Support You can now tell AWS AppSync to log API requests to CloudWatch Logs. Click on Settings and Enable logs, then choose the IAM role and the log level:
CloudFormation Support You can use the following CloudFormation resource types in your templates to define AWS AppSync resources:
AWS::AppSync::GraphQLApi – Defines an AppSync API in terms of a data source (an Amazon Elasticsearch Service domain or a DynamoDB table).
AWS::AppSync::ApiKey – Defines the access key needed to access the data source.
AWS::AppSync::GraphQLSchema – Defines a GraphQL schema.
AWS::AppSync::DataSource – Defines a data source.
AWS::AppSync::Resolver – Defines a resolver by referencing a schema and a data source, and includes a mapping template for requests.
Here’s a simple schema definition in YAML form:
AppSyncSchema:
Type: "AWS::AppSync::GraphQLSchema"
DependsOn:
- AppSyncGraphQLApi
Properties:
ApiId: !GetAtt AppSyncGraphQLApi.ApiId
Definition: |
schema {
query: Query
mutation: Mutation
}
type Query {
singlePost(id: ID!): Post
allPosts: [Post]
}
type Mutation {
putPost(id: ID!, title: String!): Post
}
type Post {
id: ID!
title: String!
}
Available Now These new features are available now and you can start using them today! Here are a couple of blog posts and other resources that you might find to be of interest:
We have several upcoming tech talks in the month of April and early May. Come join us to learn about AWS services and solution offerings. We’ll have AWS experts online to help answer questions in real-time. Sign up now to learn more, we look forward to seeing you.
Note – All sessions are free and in Pacific Time.
April & early May — 2018 Schedule
Compute
April 30, 2018 | 01:00 PM – 01:45 PM PT – Best Practices for Running Amazon EC2 Spot Instances with Amazon EMR (300) – Learn about the best practices for scaling big data workloads as well as process, store, and analyze big data securely and cost effectively with Amazon EMR and Amazon EC2 Spot Instances.
May 1, 2018 | 01:00 PM – 01:45 PM PT – How to Bring Microsoft Apps to AWS (300) – Learn more about how to save significant money by bringing your Microsoft workloads to AWS.
May 2, 2018 | 01:00 PM – 01:45 PM PT – Deep Dive on Amazon EC2 Accelerated Computing (300) – Get a technical deep dive on how AWS’ GPU and FGPA-based compute services can help you to optimize and accelerate your ML/DL and HPC workloads in the cloud.
April 25, 2018 | 01:00 PM – 01:45 PM PT – Intro to Open Source Databases on AWS (200) – Learn how to tap the benefits of open source databases on AWS without the administrative hassle.
April 24, 2018 | 11:00 AM – 11:45 AM PT – Deploy your Desktops and Apps on AWS (300) – Learn how to deploy your desktops and apps on AWS with Amazon WorkSpaces and Amazon AppStream 2.0
IoT
May 2, 2018 | 11:00 AM – 11:45 AM PT – How to Easily and Securely Connect Devices to AWS IoT (200) – Learn how to easily and securely connect devices to the cloud and reliably scale to billions of devices and trillions of messages with AWS IoT.
April 30, 2018 | 11:00 AM – 11:45 AM PT – Offline GraphQL Apps with AWS AppSync (300) – Come learn how to enable real-time and offline data in your applications with GraphQL using AWS AppSync.
Networking
May 2, 2018 | 09:00 AM – 09:45 AM PT – Taking Serverless to the Edge (300) – Learn how to run your code closer to your end users in a serverless fashion. Also, David Von Lehman from Aerobatic will discuss how they used [email protected] to reduce latency and cloud costs for their customer’s websites.
May 3, 2018 | 09:00 AM – 09:45 AM PT – Protect Your Game Servers from DDoS Attacks (200) – Learn how to use the new AWS Shield Advanced for EC2 to protect your internet-facing game servers against network layer DDoS attacks and application layer attacks of all kinds.
May 1, 2018 | 11:00 AM – 11:45 AM PT – Building Data Lakes That Cost Less and Deliver Results Faster (300) – Learn how Amazon S3 Select And Amazon Glacier Select increase application performance by up to 400% and reduce total cost of ownership by extending your data lake into cost-effective archive storage.
May 3, 2018 | 11:00 AM – 11:45 AM PT – Integrating On-Premises Vendors with AWS for Backup (300) – Learn how to work with AWS and technology partners to build backup & restore solutions for your on-premises, hybrid, and cloud native environments.
Starting on March 8th you might have seen AWS Quest popping up in different places. Now that we are a bit over halfway through the game, we thought it would be a great time give everyone a peek behind the curtain.
The whole idea started about a year ago during an casual conversation with Jeff when I first joined AWS. While we’re usually pretty good at staying focused in our meetings, he brought up that he had just finished a book he really enjoyed and asked me if I had read it. (A book that has since been made into a movie.) I don’t think there was a way for him to even imagine that as a huge fan of games, both table top and video games, how stoked I would be about the idea of bringing a game to our readers.
We got to talking about how great it would be to attempt a game that would involve the entiresuite of AWS products and our various platforms. This idea might appear to be easy, but it has kept us busy with Lone Shark for about a year and we haven’t even scratched the surface of what we would like to do. Being able to finally share this first game with our customers has been an absolute delight.
From March 8-27th, each day we have been and will be releasing a new puzzle. The clues for the puzzles are hidden somewhere all over AWS, and once customers have found the clues they can figure out the puzzle which results in a word. That word is the name of a component to rebuild Ozz, Jeff’s robot buddy.
We wanted to try make sure that anyone could play and we tried to surround each puzzle with interesting Easter eggs. So far, it seems to be working and we are seeing some really cool collaborative effort between customers to solve the puzzles. From tech talks to women who code, posts both recent and well in the past, and to Twitter and podcasts, we wanted to hide the puzzles in places our customers might not have had a chance to really explore before. Given how much Jeff enjoyed doing a live Twitch stream so much I won’t be surprised when he tells me he wants to do a TV show next.
The learnings we have already gathered as we are just a little past halfway in the quest are mind boggling. We have learned that there will be a guy who figures out how to build a chicken coop in 3D to solve a puzzle, or build a script to crawl a site looking for any reply to a blog post that might be a clue. There were puzzles we completely expected people to get stuck on that they have solved in a snap. They have really kept us on our toes, which isn’t a bad thing. It really doesn’t hurt that the players are incredibly adept at thinking outside the box, and we can’t wait to tell you how the puzzles were solved at the end.
We still have a little under a week of puzzles to go, before you can all join Jeff and special guests on a live Twitch stream to reassemble Ozz 2.0! And you don’t have to hold off for the next time we play, as there are still many puzzles to be solved and every player matters! Just keep an eye out for new puzzles to appear everyday until March 27th, join the Reddit, come to the AMA, or take a peek into the chat and get solving!
Time to wipe off your brow, and get back into solving the last of the puzzles! I am going to try to go explain to my mother and father what exactly I am doing with those two masters degrees and how much fun it really is…
Can you believe it’s already the month of March? With some great new Tech Talks available this month, there’s no better time to grow your knowledge about AWS services and solutions.
AWS Online Tech Talks
March 2018– Schedule
Below is the full schedule for the live, online technical sessions being held during the month of March. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts.
March 28, 2018 | 11:00 AM – 12:00 PM PT – Deep Dive on Amazon Athena (300) – Dive deep into the most common Amazon Athena use cases, including working with other AWS services.
Compute
March 26, 2018 | 01:00 PM – 01:45 PM PT – High Performance Computing in the Cloud (200) – Learn how AWS is enabling faster time to results and higher ROI when it comes to solving the big problems in science, engineering and business with high performance computing in the cloud.
March 27, 2018 | 01:00 PM – 01:45 PM PT – Introduction to Hybrid Cloud on AWS (200) – Learn how AWS is building the industry’s broadest capabilities for Hybrid Cloud deployments.
March 28, 2018 | 01:00 PM – 02:00 PM PT – Media Processing Workflows at High Velocity and Scale using AI and ML (200) – Hear how AWS customers have improved media supply chains using AI in areas such as metadata tagging (Rekognition and Comprehend), translations, transcriptions, and cloud services (Elemental).
March 22, 2018 | 09:00 AM – 09:45 AM PT – New Mobile CLI and Console Experience (200) – Learn how AWS Mobile Services has introduced a new CLI and streamlined console experience in order to simplify and speed up the development of mobile applications with innovative AWS features and back-end functionality.
Networking
March 28, 2018 | 09:00 AM – 09:45 AM PT –Deep Dive on New AWS Networking Features (300) – Learn how AWS PrivateLink, Direct Connect gateway, and new features with Elastic Load Balancers (ELB) come together to meet the needs of a modern enterprise.
March 29, 2018 | 09:00 AM – 09:45 AM PT – Navigating GDPR Compliance on AWS (300) – Get a walkthrough of potential General Data Protection Regulation (GDPR) obligations and see how the AWS cloud offers services and features that are consistent with GDPR considerations in the ramp-up to the May 25th, 2018 enforcement date.
March 27, 2018 | 11:00 AM – 11:45 AM PT–Enterprise Applications with Amazon EFS (300) – Join us for a technical deep dive on Amazon EFS, where you’ll learn tips and tricks for integrating your enterprise applications with Amazon EFS.
March 29, 2018 | 11:00 AM – 11:45 AM PT – Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select (300) – Join us for a webinar where we’ll demonstrate how Amazon S3 Select can increase analytics query performance up to 400%, and Amazon Glacier Select makes it practical to extend queries to archive storage, significantly reducing data lake storage costs.
We have been busy adding new features and capabilities to Amazon Redshift, and we wanted to give you a glimpse of what we’ve been doing over the past year. In this article, we recap a few of our enhancements and provide a set of resources that you can use to learn more and get the most out of your Amazon Redshift implementation.
In 2017, we made more than 30 announcements about Amazon Redshift. We listened to you, our customers, and delivered Redshift Spectrum, a feature of Amazon Redshift, that gives you the ability to extend analytics to your data lake—without moving data. We launched new DC2 nodes, doubling performance at the same price. We also announced many new features that provide greater scalability, better performance, more automation, and easier ways to manage your analytics workloads.
To see a full list of our launches, visit our what’s new page—and be sure to subscribe to our RSS feed.
Major launches in 2017
Amazon Redshift Spectrum—extend analytics to your data lake, without moving data
We launched Amazon Redshift Spectrum to give you the freedom to store data in Amazon S3, in open file formats, and have it available for analytics without the need to load it into your Amazon Redshift cluster. It enables you to easily join datasets across Redshift clusters and S3 to provide unique insights that you would not be able to obtain by querying independent data silos.
With Redshift Spectrum, you can run SQL queries against data in an Amazon S3 data lake as easily as you analyze data stored in Amazon Redshift. And you can do it without loading data or resizing the Amazon Redshift cluster based on growing data volumes. Redshift Spectrum separates compute and storage to meet workload demands for data size, concurrency, and performance. Redshift Spectrum scales processing across thousands of nodes, so results are fast, even with massive datasets and complex queries. You can query open file formats that you already use—such as Apache Avro, CSV, Grok, ORC, Apache Parquet, RCFile, RegexSerDe, SequenceFile, TextFile, and TSV—directly in Amazon S3, without any data movement.
“For complex queries, Redshift Spectrum provided a 67 percent performance gain,” said Rafi Ton, CEO, NUVIAD. “Using the Parquet data format, Redshift Spectrum delivered an 80 percent performance improvement. For us, this was substantial.”
DC2 nodes—twice the performance of DC1 at the same price
We launched second-generation Dense Compute (DC2) nodes to provide low latency and high throughput for demanding data warehousing workloads. DC2 nodes feature powerful Intel E5-2686 v4 (Broadwell) CPUs, fast DDR4 memory, and NVMe-based solid state disks (SSDs). We’ve tuned Amazon Redshift to take advantage of the better CPU, network, and disk on DC2 nodes, providing up to twice the performance of DC1 at the same price. Our DC2.8xlarge instances now provide twice the memory per slice of data and an optimized storage layout with 30 percent better storage utilization.
“Redshift allows us to quickly spin up clusters and provide our data scientists with a fast and easy method to access data and generate insights,” said Bradley Todd, technology architect at Liberty Mutual. “We saw a 9x reduction in month-end reporting time with Redshift DC2 nodes as compared to DC1.”
On average, our customers are seeing 3x to 5x performance gains for most of their critical workloads.
We introduced short query acceleration to speed up execution of queries such as reports, dashboards, and interactive analysis. Short query acceleration uses machine learning to predict the execution time of a query, and to move short running queries to an express short query queue for faster processing.
We launched results caching to deliver sub-second response times for queries that are repeated, such as dashboards, visualizations, and those from BI tools. Results caching has an added benefit of freeing up resources to improve the performance of all other queries.
We also introduced late materialization to reduce the amount of data scanned for queries with predicate filters by batching and factoring in the filtering of predicates before fetching data blocks in the next column. For example, if only 10 percent of the table rows satisfy the predicate filters, Amazon Redshift can potentially save 90 percent of the I/O for the remaining columns to improve query performance.
We launched query monitoring rules and pre-defined rule templates. These features make it easier for you to set metrics-based performance boundaries for workload management (WLM) queries, and specify what action to take when a query goes beyond those boundaries. For example, for a queue that’s dedicated to short-running queries, you might create a rule that aborts queries that run for more than 60 seconds. To track poorly designed queries, you might have another rule that logs queries that contain nested loops.
Customer insights
Amazon Redshift and Redshift Spectrum serve customers across a variety of industries and sizes, from startups to large enterprises. Visit our customer page to see the success that customers are having with our recent enhancements. Learn how companies like Liberty Mutual Insurance saw a 9x reduction in month-end reporting time using DC2 nodes. On this page, you can find case studies, videos, and other content that show how our customers are using Amazon Redshift to drive innovation and business results.
In addition, check out these resources to learn about the success our customers are having building out a data warehouse and data lake integration solution with Amazon Redshift:
You can enhance your Amazon Redshift data warehouse by working with industry-leading experts. Our AWS Partner Network (APN) Partners have certified their solutions to work with Amazon Redshift. They offer software, tools, integration, and consulting services to help you at every step. Visit our Amazon Redshift Partner page and choose an APN Partner. Or, use AWS Marketplace to find and immediately start using third-party software.
To see what our Partners are saying about Amazon Redshift Spectrum and our DC2 nodes mentioned earlier, read these blog posts:
If you are evaluating or considering a proof of concept with Amazon Redshift, or you need assistance migrating your on-premises or other cloud-based data warehouse to Amazon Redshift, our team of product experts and solutions architects can help you with architecting, sizing, and optimizing your data warehouse. Contact us using this support request form, and let us know how we can assist you.
If you are an Amazon Redshift customer, we offer a no-cost health check program. Our team of database engineers and solutions architects give you recommendations for optimizing Amazon Redshift and Amazon Redshift Spectrum for your specific workloads. To learn more, email us at [email protected].
Larry Heathcote is a Principle Product Marketing Manager at Amazon Web Services for data warehousing and analytics. Larry is passionate about seeing the results of data-driven insights on business outcomes. He enjoys family time, home projects, grilling out and the taste of classic barbeque.
The Twelve-Factor App methodology is twelve best practices for building modern, cloud-native applications. With guidance on things like configuration, deployment, runtime, and multiple service communication, the Twelve-Factor model prescribes best practices that apply to a diverse number of use cases, from web applications and APIs to data processing applications. Although serverless computing and AWS Lambda have changed how application development is done, the Twelve-Factor best practices remain relevant and applicable in a serverless world.
In this post, I directly apply and compare the Twelve-Factor methodology to serverless application development with Lambda and Amazon API Gateway.
The Twelve Factors
As you’ll see, many of these factors are not only directly applicable to serverless applications, but in fact a default mechanism or capability of the AWS serverless platform. Other factors don’t fit, and I talk about how these factors may not apply at all in a serverless approach.
A general software development best practice is to have all of your code in revision control. This is no different with serverless applications.
For a single serverless application, your code should be stored in a single repository in which a single deployable artifact is generated and from which it is deployed. This single code base should also represent the code used in all of your application environments (development, staging, production, etc.). What might be different for serverless applications is the bounds for what constitutes a “single application.”
Here are two guidelines to help you understand the scope of an application:
If events are shared (such as a common Amazon API Gateway API), then the Lambda function code for those events should be put in the same repository.
Otherwise, break functions along event sources into their own repositories.
Following these two guidelines helps you keep your serverless applications scoped to a single purpose and help prevent complexity in your code base.
Code that needs to be used by multiple functions should be packaged into its own library and included inside your deployment package. Going back to the previous factor on codebase, if you find that you need to often include special processing or business logic, the best solution may be to try to create a purposeful library yourself. Every language that Lambda supports has a model for dependencies/libraries, which you can use:
Both Lambda and API Gateway allow you to set configuration information, using the environment in which each service runs.
In Lambda, these are called environment variables and are key-value pairs that can be set at deployment or when updating the function configuration. Lambda then makes these key-value pairs available to your Lambda function code using standard APIs supported by the language, like process.env for Node.js functions. For more information, see Programming Model, which contains examples for each supported language.
Lambda also allows you to encrypt these key-value pairs using KMS, such that they can be used to store secrets such as API keys or passwords for databases. You can also use them to help define application environment specifics, such as differences between testing or production environments where you might have unique databases or endpoints with which your Lambda function needs to interface. You could also use these for setting A/B testing flags or to enable or disable certain function logic.
For API Gateway, these configuration variables are called stage variables. Like environment variables in Lambda, these are key-value pairs that are available for API Gateway to consume or pass to your API’s backend service. Stage variables can be useful to send requests to different backend environments based on the URL from which your API is accessed. For example, a single configuration could support both beta.yourapi.com vs. prod.yourapi.com. You could also use stage variables to pass information to a Lambda function that causes it to perform different logic.
Because Lambda doesn’t allow you to run another service as part of your function execution, this factor is basically the default model for Lambda. Typically, you reference any database or data store as an external resource via HTTP endpoint or DNS name. These connection strings are ideally passed in via the configuration information, as previously covered.
The separation of build, release, and run stages follows the development best practices of continuous integration and delivery. AWS recommends that you have a CI &CD process no matter what type of application you are building. For serverless applications, this is no different. For more information, see the Building CI/CD Pipelines for Serverless Applications (SRV302) re:Invent 2017 session.
An example minimal pipeline (from presentation linked above)
This is inherent in how Lambda is designed so there is nothing more to consider. Lambda functions should always be treated as being stateless, despite the ability to potentially store some information locally between execution environment re-use. This is because there is no guaranteed affinity to any execution environment, and the potential for an execution environment to go away between invocations exists. You should always store any stateful information in a database, cache, or separate data store via a backing service.
This factor also does not apply to Lambda, as execution environments do not expose any direct networking to your functions. Instead of a port, Lambda functions are invoked via one or more triggering services or AWS APIs for Lambda. There are currently three different invocation models:
Lambda was built with massive concurrency and scale in mind. A recent post on this blog, titled Managing AWS Lambda Function Concurrency explained that for a serverless application, “the unit of scale is a concurrent execution” and that these are consumed by your functions.
Lambda automatically scales to meet the demands of invocations sent at your function. This is in contrast to a traditional compute model using physical hosts, virtual machines, or containers that you self-manage. With Lambda, you do not need to manage overall capacity or apply scaling policies.
Each AWS account has an overall AccountLimit value that is fixed at any point in time, but can be easily increased as needed. As of May 2017, the default limit is 1000 concurrent executions per AWS Region. You can also set and manage a reserved concurrency limit, which provides a limit to how much concurrency a function can have. It also reserves concurrency capacity for a given function out of the total available for an account.
Shutdown doesn’t apply to Lambda because Lambda is intrinsically event-driven. Invocations are tied directly to incoming events or triggers.
However, speed at startup does matter. Initial function execution latency, or what is called “cold starts”, can occur when there isn’t a “warmed” compute resource ready to execute against your application invocations. In the AWS Lambda Execution Model topic, it explains that:
“It takes time to set up an execution context and do the necessary “bootstrapping”, which adds some latency each time the Lambda function is invoked. You typically see this latency when a Lambda function is invoked for the first time or after it has been updated because AWS Lambda tries to reuse the execution context for subsequent invocations of the Lambda function.”
The Best Practices topic covers a number of issues around how to think about performance of your functions. This includes where to place certain logic, how to re-use execution environments, and how by configuring your function for more memory you also get a proportional increase in CPU available to your function. With AWS X-Ray, you can gather some insight as to what your function is doing during an execution and make adjustments accordingly.
Along with continuous integration and delivery, the practice of having independent application environments is a solid best practice no matter the development approach. Being able to safely test applications in a non-production environment is key to development success. Products within the AWS Serverless Platform do not charge for idle time, which greatly reduces the cost of running multiple environments. You can also use the AWS Serverless Application Model (AWS SAM), to manage the configuration of your separate environments.
SAM allows you to model your serverless applications in greatly simplified AWS CloudFormation syntax. With SAM, you can use CloudFormation’s capabilities—such as Parameters and Mappings—to build dynamic templates. Along with Lambda’s environment variables and API Gateway’s stage variables, those templates give you the ability to deploy multiple environments from a single template, such as testing, staging, and production. Whenever the non-production environments are not in use, your costs for Lambda and API Gateway would be zero. For more information, see the AWS Lambda Applications with AWS Serverless Application Model 2017 AWS online tech talk.
In a typical non-serverless application environment, you might be concerned with log files, logging daemons, and centralization of the data represented in them. Thankfully, this is not a concern for serverless applications, as most of the services in the platform handle this for you.
With Lambda, you can just output logs to the console via the native capabilities of the language in which your function is written. For more information about Go, see Logging (Go). Similar documentation pages exist for other languages. Those messages output by your code are captured and centralized in Amazon CloudWatch Logs. For more information, see Accessing Amazon CloudWatch Logs for AWS Lambda.
API Gateway provides two different methods for getting log information:
Execution logs Includes errors or execution traces (such as request or response parameter values or payloads), data used by custom authorizers, whether API keys are required, whether usage plans are enabled, and so on.
Access logs Provide the ability to log who has accessed your API and how the caller accessed the API. You can even customize the format of these logs as desired.
Capturing logs and being able to search and view them is one thing, but CloudWatch Logs also gives you the ability to treat a log message as an event and take action on them via subscription filters in the service. With subscription filters, you could send a log message matching a certain pattern to a Lambda function, and have it take action based on that. Say, for example, that you want to respond to certain error messages or usage patterns that violate certain rules. You could do that with CloudWatch Logs, subscription filters, and Lambda. Another important capability of CloudWatch Logs is the ability to “pivot” log information into a metric in CloudWatch. With this, you could take a data point from a log entry, create a metric, and then an alarm on a metric to show a breached threshold.
This is another factor that doesn’t directly apply to Lambda due to its design. Typically, you would have your functions scoped down to single or limited use cases and have individual functions for different components of your application. Even if they share a common invoking resource, such as an API Gateway endpoint and stage, you would still separate the individual API resources and actions to their own Lambda functions.
The Seven-and-a-Half–Factor model and you
As we’ve seen, Twelve-Factor application design can still be applied to serverless applications, taking into account some small differences! The following diagram highlights the factors and how applicable or not they are to serverless applications:
NOTE: Disposability only half applies, as discussed in this post.
Conclusion
If you’ve been building applications for cloud infrastructure over the past few years, the Twelve-Factor methodology should seem familiar and straight-forward. If you are new to this space, however, you should know that the general best practices and default working patterns for serverless applications overlap heavily with what I’ve discussed here.
It shouldn’t require much work to adhere rather closely to the Twelve-Factor model. When you’re building serverless applications, following the applicable points listed here helps you simplify development and any operational work involved (though already minimized by services such as Lambda and API Gateway). The Twelve-Factor methodology also isn’t all-or-nothing. You can apply only the practices that work best for you and your applications, and still benefit.
We’re just over three weeks away from the Raspberry Jam Big Birthday Weekend 2018, our community celebration of Raspberry Pi’s sixth birthday. Instead of an event in Cambridge, as we’ve held in the past, we’re coordinating Raspberry Jam events to take place around the world on 3–4 March, so that as many people as possible can join in. Well over 100 Jams have been confirmed so far.
Find a Jam near you
There are Jams planned in Argentina, Australia, Bolivia, Brazil, Bulgaria, Cameroon, Canada, Colombia, Dominican Republic, France, Germany, Greece, Hungary, India, Iran, Ireland, Italy, Japan, Kenya, Malaysia, Malta, Mexico, Netherlands, Norway, Papua New Guinea, Peru, Philippines, Poland, South Africa, Spain, Taiwan, Turkey, United Kingdom, United States, and Zimbabwe.
Take a look at the events map and the full list (including those who haven’t added their event to the map quite yet).
We will have Raspberry Jams in 35 countries across six continents
Birthday kits
We had some special swag made especially for the birthday, including these T-shirts, which we’ve sent to Jam organisers:
There is also a poster with a list of participating Jams, which you can download:
Raspberry Jam photo booth
I created a Raspberry Jam photo booth that overlays photos with the Big Birthday Weekend logo and then tweets the picture from your Jam’s account — you’ll be seeing plenty of those if you follow the #PiParty hashtag on 3–4 March.
Check out the project on GitHub, and feel free to set up your own booth, or modify it to your own requirements. We’ve included text annotations in several languages, and more contributions are very welcome.
There’s still time…
If you can’t find a Jam near you, there’s still time to organise one for the Big Birthday Weekend. All you need to do is find a venue — a room in a school or library will do — and think about what you’d like to do at the event. Some Jams have Raspberry Pis set up for workshops and practical activities, some arrange tech talks, some put on show-and-tell — it’s up to you. To help you along, there’s the Raspberry Jam Guidebook full of advice and tips from Jam organisers.
The packed. And they packed. And they packed some more. Who’s expecting one of these #rjam kits for the Raspberry Jam Big Birthday Weekend?
Download the Raspberry Jam branding pack, and the special birthday branding pack, where you’ll find logos, graphical assets, flyer templates, worksheets, and more. When you’re ready to announce your event, create a webpage for it — you can use a site like Eventbrite or Meetup — and submit your Jam to us so it will appear on the Jam map!
We are six
We’re really looking forward to celebrating our birthday with thousands of people around the world. Over 48 hours, people of all ages will come together at more than 100 events to learn, share ideas, meet people, and make things during our Big Birthday Weekend.
Since we released the first Raspberry Pi in 2012, we’ve sold 17 million of them. We’re also reaching almost 200000 children in 130 countries around the world through Code Club and CoderDojo, we’ve trained over 1500 Raspberry Pi Certified Educators, and we’ve sent code written by more than 6800 children into space. Our magazines are read by a quarter of a million people, and millions more use our free online learning resources. There’s plenty to celebrate and even more still to do: we really hope you’ll join us from a Jam near you on 3–4 March.
Noted below are the upcoming scheduled live, online technical sessions being held during the month of January. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts.
With the introduction of AWS Organizations and AWS CloudFormation StackSets, you can create and manage standard AWS Identity and Access Management (IAM) roles, customer managed policies, and federated identity providers across a set of accounts in your organization. This tech talk goes through the details of setting up a CloudFormation StackSet in your master account, and creating stack instances in each account that set up roles and policies in each account in an organizational unit (OU). We also will discuss how to update the stacks and how to integrate StackSets into your account creation process. This tech talk is a followup to a set of blog posts on the AWS Security Blog published during the summer of 2017.
As part of the AWS Online Tech Talks series, AWS will present Cloud-Native DDoS Mitigation with AWS Shield on Wednesday, December 13. This tech talk will start at 9:00 A.M. Pacific Time and end at 9:40 A.M. Pacific Time.
Distributed Denial of Service (DDoS) mitigation can help you maintain application availability, but traditional solutions are hard to scale and require expensive hardware. AWS Shield is a managed DDoS protection service that helps you safeguard web applications running in the AWS Cloud. In this tech talk, you will learn simple techniques for using AWS Shield to help you build scalable DDoS defenses into your applications without investing in costly infrastructure. You also will learn how AWS Shield helps you monitor your applications to detect DDoS attempts and how to respond to in-progress events.
As part of the AWS Online Tech Talks series, AWS will present How to Integrate AWS Directory Service with Office 365 on Wednesday, November 15. This tech talk will start at 9:00 A.M. Pacific Time and end at 9:40 A.M. Pacific Time.
If you want to support Active Directory–aware workloads in AWS and Office 365 simultaneously using a managed Active Directory in the cloud, you need a nonintuitive integration to synchronize identities between deployments. AWS has recently introduced the ability for you to authenticate your Office 365 permissions using AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD) by using a custom configuration of Active Directory Federation Services (AD FS). In this webinar, AWS Directory Service Product Manager Ron Cully shows how to configure your AWS Managed Microsoft AD environment to synchronize with Office 365. He will provide detailed configuration settings, architectural considerations, and deployment steps for a highly available, secure, and easy-to-manage solution in the AWS Cloud.
You also will learn how to:
Deploy AWS Managed Microsoft AD.
Deploy Microsoft Azure AD Connect and AD FS with AWS Managed Microsoft AD.
Authenticate user access to Office 365 by using AWS Managed Microsoft AD.
Leaves are crunching under my boots, Halloween is tomorrow, and pumpkin is having its annual moment in the sun – it’s fall everybody! And just in time to celebrate, we have whipped up a fresh batch of pumpkin spice Tech Talks. Grab your planner (Outlook calendar) and pencil these puppies in. This month we are covering re:Invent, serverless, and everything in between.
November 2017 – Schedule
Noted below are the upcoming scheduled live, online technical sessions being held during the month of November. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts.
As part of the AWS Online Tech Talks series, AWS will present CloudHSM – Secure, Scalable Key Storage in AWS on Wednesday, October 25. This tech talk will start at 9:00 A.M. Pacific Time and end at 9:40 A.M. Pacific Time.
Applications handling confidential or sensitive data are subject to corporate or regulatory requirements and therefore need validated control of encryption keys and cryptographic operations. AWS CloudHSM brings to your AWS resources the security and control of traditional HSMs. This Tech Talk will show how you can leverage CloudHSM to build scalable, reliable applications without sacrificing either security or performance. Attend this Tech Talk to learn how you can use CloudHSM to quickly and easily build secure, compliant, fast, and flexible applications.
You also will:
Learn about the challenges CloudHSM can help you address.
Understand how CloudHSM can secure your workloads and data.
As part of the AWS Online Tech Talks series, AWS will present How to Use AWS WAF to Mitigate OWASP Top 10 Attacks on Thursday, September 28. This tech talk will start at 9:00 A.M. Pacific Time and end at 9:40 A.M. Pacific Time.
The Open Web Application Security Project (OWASP) Top 10 identifies the most critical vulnerabilities that web developers must address in their applications. AWS WAF, a web application firewall, helps you address the vulnerabilities identified in the OWASP Top 10. In this webinar, you will learn how to use AWS WAF to write rules to match common patterns of exploitation and block malicious requests from reaching your web servers.
As a school supply aficionado, the month of September has always held a special place in my heart. Nothing sets the tone for success like getting a killer deal on pens and a crisp college ruled notebook. Even if back to school shopping trips have secured a seat in your distant memory, this is still a perfect time of year to stock up on office supplies and set aside some time for flexing those learning muscles. A great way to get started: scan through our September Tech Talks and check out the ones that pique your interest. This month we are covering re:Invent, AI, and much more.
September 2017 – Schedule
Noted below are the upcoming scheduled live, online technical sessions being held during the month of September. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts.
The AWS Online Tech Talks series covers a broad range of topics at varying technical levels. These sessions feature live demonstrations & customer examples led by AWS engineers and Solution Architects. Check out the AWS YouTube channel for more on-demand webinars on AWS technologies.
Welcome to mid-August, everyone–the season of beach days, family road trips, and an inbox full of “out of office” emails from your coworkers. Just in case spending time indoors has you feeling a bit blue, we’ve got a piping hot batch of AWS Online Tech Talks for you to check out. Kick up your feet, grab a glass of ice cold lemonade, and dive into our latest Tech Talks on Compute and DevOps.
August 2017 – Schedule
Noted below are the upcoming scheduled live, online technical sessions being held during the month of August. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts.
The AWS Online Tech Talks series covers a broad range of topics at varying technical levels. These sessions feature live demonstrations & customer examples led by AWS engineers and Solution Architects. Check out the AWS YouTube channel for more on-demand webinars on AWS technologies.
– Sara (Hello everyone, I’m a co-op from Northeastern University joining the team until December.)
It’s unbelievable that 2017 has flown by so quickly, yet here we are already in the month of July. A little-known fact about the 7th month of the year is that its name, July, is in honor of the Roman general, Julius Cæsar. The Roman State named the month on his behalf since it the month of his birth. Prior to this designation, the month of July was called Quintilis.
I, also, thought it was interesting to learn that in the month of July, several countries celebrate their Independence Day. These countries are the United States, Bahamas, Kiribati, São Tomé, Príncipe, Liberia, Maldives, Algeria, Cape Verde, Venezuela, Burundi, Rwanda, and Somalia. Seems that the month of July was ripe for freedom and independence for all parts of the world.
Below is the upcoming schedule for the live, online technical sessions scheduled for the month of July. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts. All schedule times for the online tech talks are shown in the Pacific Time (PDT) time zone.
The AWS Online Tech Talks series covers a broad range of topics at varying technical levels. These sessions feature live demonstrations & customer examples led by AWS engineers and Solution Architects. Check out the AWS YouTube channel for more on-demand webinars on AWS technologies.
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.