Tag Archives: AWS

How Disney Streaming Services Uses Amazon Pinpoint to Send Personalized Messages to Millions of Users in Real Time

Post Syndicated from Hannah Nilsson original https://aws.amazon.com/blogs/messaging-and-targeting/how-disney-streaming-services-uses-amazon-pinpoint-to-send-personalized-messages-to-millions-of-users-in-real-time/

At AWS re:Invent 2018, Billy Liu and Jimmy Tam from Disney Streaming Services took the stage to talk about how they use Amazon Pinpoint to meet some of their unique digital user engagement needs. Disney Streaming Services supports several mobile apps, including MLB At Bat, Ballpark and Beat The Streak from Major League Baseball, along with the NHL mobile app. Billy and Jimmy shared their story on the re:Invent Launchpad stage, as well as during the Digital User Engagement Leadership Session with Simon Poile from AWS. In these sessions, they discussed how they use Amazon Pinpoint—along with other AWS services including Amazon Kinesis, AWS Lambda, Amazon S3, and AWS Glue—to target customers, monitor the performance of their campaigns in real time, and gain a deeper understanding of their users’ needs and desires.

Targeting the right customer at the right time 
When you consider the use cases for MLB’s suite of apps, you can quickly see why sending the right message to the right customer is a more complicated task than it might seem at first glance. For each of the 30 Major League Baseball teams, users can opt to receive eight different types of messages. Each of these eight message types is available in both English and Spanish. And on top of all that, each push notification sent has to target combinations of these segments when two teams play each other. There are thousands of possible segments and combinations of segments to consider with each message sent.

To address this issue, Disney Streaming Services uses Amazon Pinpoint to dynamically create unique segments and campaigns for every event in milliseconds. In the most demanding usage scenarios, Amazon Pinpoint scales to create over 300 segments and campaigns per hour, and over 20 segments and campaigns per minute. To learn more about how they solved this challenge, take a look at the recording of their session.

How Disney Streaming Services Targets The Right Customer
Monitoring campaign performance in real time
With the fast-paced nature of Disney Streaming’s notifications and the sheer number of campaigns and segments they are targeting, monitoring their performance directly in the Amazon Pinpoint console is not scalable for their use case. However, they must have real-time notifications to let them know if their campaigns are lagging or not reaching the expected number of recipients.

To meet this unique need, Disney Streaming developed a solution that uses AWS Step Functions, Amazon Cloudwatch, AWS Lambda, and Amazon Pinpoint. This solution monitors each campaign that is created. When a campaign is executed, their solution streams data about the execution and delivery of that campaign, and sends alerts when the team needs to take a closer look at how their campaigns are performing. You can learn more about the specifics of their monitoring solution in the recording of their session.

How Disney Streaming Services Monitors Campaign Performance

Understanding fans
After a campaign has been sent, Disney Streaming analyzes the performance of campaigns. By performing this analysis, they can better understand how customers engaged with notifications, and ensure fans are receiving a compelling experience.

To achieve this, Disney Streaming uses the event streaming and exporting features of Amazon Pinpoint. They stream engagement events by using Amazon Kinesis. These events let them know how fans interacted with the application, and allow them to drill down into various performance metrics on a per-team basis. They then store these metrics in S3, which are picked up by their data lake team for further processing. By using this solution, they can create near-real-time reports for the unique audiences.  How Disney Streaming Services Understands their Fans
They also use the Amazon Pinpoint API to export all of the details about the users to an S3 bucket using Lambda Triggers. An AWS Glue job processes the exported data and outputs the results to another S3 bucket. The data lake team then uses this data to glean additional insights about their audience.

How Disney Streaming Services Understands their Fans

Removing unengaged customers 
Disney Streaming also uses a custom solution to engage with customers who are still able to receive messages, and to ensure that reports only include engaged users. For example, if a customer uninstalls the MLB or NHL apps, re-installs an app but doesn’t set their messaging preferences, or starts using a device on a different platform, that customer might not be able to be contacted. Disney Streaming needs to remove these unreachable customers from campaigns so that they can maintain accurate reports on audience sizes, helping keep costs low, and reduce campaign latency.

To delete unreachable customers in real time, the Disney Streaming team uses Amazon Pinpoint to detect when they attempt to send a push notification to an unreachable customer. Their Kinesis Firehose stream then outputs campaign data to an S3 bucket, and an AWS Glue job filters out the customers who are unreachable. Finally, a Lambda function removes the endpoint by making a call to the Amazon Pinpoint API. You can find more details about how Disney Streaming Services implemented this solution in the recording of their session.

How Disney Streaming Services removes unengaged customers

You can learn more about the needs that Disney Streaming Service considered when they chose a Digital User Engagement solution by watching the recording of their discussion on the re:Invent 2018 Launchpad stage. You can also watch the Digital User Engagement Leadership Session to learn more about AWS’ Digital User Engagement solutions, information on recent feature launches, and to learn more about how Disney Streaming created solutions to their engagement challenges.

Optimizing a Lift-and-Shift for Security

Post Syndicated from Jonathan Shapiro-Ward original https://aws.amazon.com/blogs/architecture/optimizing-a-lift-and-shift-for-security/

This is the third and final blog within a three-part series that examines how to optimize lift-and-shift workloads. A lift-and-shift is a common approach for migrating to AWS, whereby you move a workload from on-prem with little or no modification. This third blog examines how lift-and-shift workloads can benefit from an improved security posture with no modification to the application codebase. (Read about optimizing a lift-and-shift for performance and for cost effectiveness.)

Moving to AWS can help to strengthen your security posture by eliminating many of the risks present in on-premise deployments. It is still essential to consider how to best use AWS security controls and mechanisms to ensure the security of your workload. Security can often be a significant concern in lift-and-shift workloads, especially for legacy workloads where modern encryption and security features may not present. By making use of AWS security features you can significantly improve the security posture of a lift-and-shift workload, even if it lacks native support for modern security best practices.

Adding TLS with Application Load Balancers

Legacy applications are often the subject of a lift-and-shift. Such migrations can help reduce risks by moving away from out of date hardware but security risks are often harder to manage. Many legacy applications leverage HTTP or other plaintext protocols that are vulnerable to all manner of attacks. Often, modifying a legacy application’s codebase to implement TLS is untenable, necessitating other options.

One comparatively simple approach is to leverage an Application Load Balancer or a Classic Load Balancer to provide SSL offloading. In this scenario, the load balancer would be exposed to users, while the application servers that only support plaintext protocols will reside within a subnet which is can only be accessed by the load balancer. The load balancer would perform the decryption of all traffic destined for the application instance, forwarding the plaintext traffic to the instances. This allows  you to use encryption on traffic between the client and the load balancer, leaving only internal communication between the load balancer and the application in plaintext. Often this approach is sufficient to meet security requirements, however, in more stringent scenarios it is never acceptable for traffic to be transmitted in plaintext, even if within a secured subnet. In this scenario, a sidecar can be used to eliminate plaintext traffic ever traversing the network.

Improving Security and Configuration Management with Sidecars

One approach to providing encryption to legacy applications is to leverage what’s often termed the “sidecar pattern.” The sidecar pattern entails a second process acting as a proxy to the legacy application. The legacy application only exposes its services via the local loopback adapter and is thus accessible only to the sidecar. In turn the sidecar acts as an encrypted proxy, exposing the legacy application’s API to external consumers via TLS. As unencrypted traffic between the sidecar and the legacy application traverses the loopback adapter, it never traverses the network. This approach can help add encryption (or stronger encryption) to legacy applications when it’s not feasible to modify the original codebase. A common approach to implanting sidecars is through container groups such as pod in EKS or a task in ECS.

Implementing the Sidecar Pattern With Containers

Figure 1: Implementing the Sidecar Pattern With Containers

Another use of the sidecar pattern is to help legacy applications leverage modern cloud services. A common example of this is using a sidecar to manage files pertaining to the legacy application. This could entail a number of options including:

  • Having the sidecar dynamically modify the configuration for a legacy application based upon some external factor, such as the output of Lambda function, SNS event or DynamoDB write.
  • Having the sidecar write application state to a cache or database. Often applications will write state to the local disk. This can be problematic for autoscaling or disaster recovery, whereby having the state easily accessible to other instances is advantages. To facilitate this, the sidecar can write state to Amazon S3, Amazon DynamoDB, Amazon Elasticache or Amazon RDS.

A sidecar requires customer development, but it doesn’t require any modification of the lift-and-shifted application. A sidecar treats the application as a blackbox and interacts with it via its API, configuration file, or other standard mechanism.

Automating Security

A lift-and-shift can achieve a significantly stronger security posture by incorporating elements of DevSecOps. DevSecOps is a philosophy that argues that everyone is responsible for security and advocates for automation all parts of the security process. AWS has a number of services which can help implement a DevSecOps strategy. These services include:

  • Amazon GuardDuty: a continuous monitoring system which analyzes AWS CloudTrail Events, Amazon VPC Flow Log and DNS Logs. GuardDuty can detect threats and trigger an automated response.
  • AWS Shield: a managed DDOS protection services
  • AWS WAF: a managed Web Application Firewall
  • AWS Config: a service for assessing, tracking, and auditing changes to AWS configuration

These services can help detect security problems and implement a response in real time, achieving a significantly strong posture than traditional security strategies. You can build a DevSecOps strategy around a lift-and-shift workload using these services, without having to modify the lift-and-shift application.

Conclusion

There are many opportunities for taking advantage of AWS services and features to improve a lift-and-shift workload. Without any alteration to the application you can strengthen your security posture by utilizing AWS security services and by making small environmental and architectural changes that can help alleviate the challenges of legacy workloads.

About the author

Dr. Jonathan Shapiro-Ward is an AWS Solutions Architect based in Toronto. He helps customers across Canada to transform their businesses and build industry leading cloud solutions. He has a background in distributed systems and big data and holds a PhD from the University of St Andrews.

Optimizing a Lift-and-Shift for Cost Effectiveness and Ease of Management

Post Syndicated from Jonathan Shapiro-Ward original https://aws.amazon.com/blogs/architecture/optimizing-a-lift-and-shift-for-cost/

Lift-and-shift is the process of migrating a workload from on premise to AWS with little or no modification. A lift-and-shift is a common route for enterprises to move to the cloud, and can be a transitionary state to a more cloud native approach. This is the second blog post in a three-part series which investigates how to optimize a lift-and-shift workload. The first post is about performance.

A key concern that many customers have with a lift-and-shift is cost. If you move an application as is  from on-prem to AWS, is there any possibility for meaningful cost savings? By employing AWS services, in lieu of self-managed EC2 instances, and by leveraging cloud capability such as auto scaling, there is potential for significant cost savings. In this blog post, we will discuss a number of AWS services and solutions that you can leverage with minimal or no change to your application codebase in order to significantly reduce management costs and overall Total Cost of Ownership (TCO).

Automate

Even if you can’t modify your application, you can change the way you deploy your application. The adopting-an-infrastructure-as-code approach can vastly improve the ease of management of your application, thereby reducing cost. By templating your application through Amazon CloudFormation, Amazon OpsWorks, or Open Source tools you can make deploying and managing your workloads a simple and repeatable process.

As part of the lift-and-shift process, rationalizing the workload into a set of templates enables less time to spent in the future deploying and modifying the workload. It enables the easy creation of dev/test environments, facilitates blue-green testing, opens up options for DR, and gives the option to roll back in the event of error. Automation is the single step which is most conductive to improving ease of management.

Reserved Instances and Spot Instances

A first initial consideration around cost should be the purchasing model for any EC2 instances. Reserved Instances (RIs) represent a 1-year or 3-year commitment to EC2 instances and can enable up to 75% cost reduction (over on demand) for steady state EC2 workloads. They are ideal for 24/7 workloads that must be continually in operation. An application requires no modification to make use of RIs.

An alternative purchasing model is EC2 spot. Spot instances offer unused capacity available at a significant discount – up to 90%. Spot instances receive a two-minute warning when the capacity is required back by EC2 and can be suspended and resumed. Workloads which are architected for batch runs – such as analytics and big data workloads – often require little or no modification to make use of spot instances. Other burstable workloads such as web apps may require some modification around how they are deployed.

A final alternative is on-demand. For workloads that are not running in perpetuity, on-demand is ideal. Workloads can be deployed, used for as long as required, and then terminated. By leveraging some simple automation (such as AWS Lambda and CloudWatch alarms), you can schedule workloads to start and stop at the open and close of business (or at other meaningful intervals). This typically requires no modification to the application itself. For workloads that are not 24/7 steady state, this can provide greater cost effectiveness compared to RIs and more certainty and ease of use when compared to spot.

Amazon FSx for Windows File Server

Amazon FSx for Windows File Server provides a fully managed Windows filesystem that has full compatibility with SMB and DFS and full AD integration. Amazon FSx is an ideal choice for lift-and-shift architectures as it requires no modification to the application codebase in order to enable compatibility. Windows based applications can continue to leverage standard, Windows-native protocols to access storage with Amazon FSx. It enables users to avoid having to deploy and manage their own fileservers – eliminating the need for patching, automating, and managing EC2 instances. Moreover, it’s easy to scale and minimize costs, since Amazon FSx offers a pay-as-you-go pricing model.

Amazon EFS

Amazon Elastic File System (EFS) provides high performance, highly available multi-attach storage via NFS. EFS offers a drop-in replacement for existing NFS deployments. This is ideal for a range of Linux and Unix usecases as well as cross-platform solutions such as Enterprise Java applications. EFS eliminates the need to manage NFS infrastructure and simplifies storage concerns. Moreover, EFS provides high availability out of the box, which helps to reduce single points of failure and avoids the need to manually configure storage replication. Much like Amazon FSx, EFS enables customers to realize cost improvements by moving to a pay-as-you-go pricing model and requires a modification of the application.

Amazon MQ

Amazon MQ is a managed message broker service that provides compatibility with JMS, AMQP, MQTT, OpenWire, and STOMP. These are amongst the most extensively used middleware and messaging protocols and are a key foundation of enterprise applications. Rather than having to manually maintain a message broker, Amazon MQ provides a performant, highly available managed message broker service that is compatible with existing applications.

To use Amazon MQ without any modification, you can adapt applications that leverage a standard messaging protocol. In most cases, all you need to do is update the application’s MQ endpoint in its configuration. Subsequently, the Amazon MQ service handles the heavy lifting of operating a message broker, configuring HA, fault detection, failure recovery, software updates, and so forth. This offers a simple option for reducing management overhead and improving the reliability of a lift-and-shift architecture. What’s more is that applications can migrate to Amazon MQ without the need for any downtime, making this an easy and effective way to improve a lift-and-shift.

You can also use Amazon MQ to integrate legacy applications with modern serverless applications. Lambda functions can subscribe to MQ topics and trigger serverless workflows, enabling compatibility between legacy and new workloads.

Integrating Lift-and-Shift Workloads with Lambda via Amazon MQ

Figure 1: Integrating Lift-and-Shift Workloads with Lambda via Amazon MQ

Amazon Managed Streaming Kafka

Lift-and-shift workloads which include a streaming data component are often built around Apache Kafka. There is a certain amount of complexity involved in operating a Kafka cluster which incurs management and operational expense. Amazon Kinesis is a managed alternative to Apache Kafka, but it is not a drop-in replacement. At re:Invent 2018, we announced the launch of Amazon Managed Streaming Kafka (MSK) in public preview. MSK provides a managed Kafka deployment with pay-as-you-go pricing and an acts as a drop-in replacement in existing Kafka workloads. MSK can help reduce management costs and improve cost efficiency and is ideal for lift-and-shift workloads.

Leveraging S3 for Static Web Hosting

A significant portion of any web application is static content. This includes videos, image, text, and other content that changes seldom, if ever. In many lift-and-shifted applications, web servers are migrated to EC2 instances and host all content – static and dynamic. Hosting static content from an EC2 instance incurs a number of costs including the instance, EBS volumes, and likely, a load balancer. By moving static content to S3, you can significantly reduce the amount of compute required to host your web applications. In many cases, this change is non-disruptive and can be done at the DNS or CDN layer, requiring no change to your application.

Reducing Web Hosting Costs with S3 Static Web Hosting

Figure 2: Reducing Web Hosting Costs with S3 Static Web Hosting

Conclusion

There are numerous opportunities for reducing the cost of a lift-and-shift. Without any modification to the application, lift-and-shift workloads can benefit from cloud-native features. By using AWS services and features, you can significantly reduce the undifferentiated heavy lifting inherent in on-prem workloads and reduce resources and management overheads.

About the author

Dr. Jonathan Shapiro-Ward is an AWS Solutions Architect based in Toronto. He helps customers across Canada to transform their businesses and build industry leading cloud solutions. He has a background in distributed systems and big data and holds a PhD from the University of St Andrews.

Optimizing a Lift-and-Shift for Performance

Post Syndicated from Jonathan Shapiro-Ward original https://aws.amazon.com/blogs/architecture/optimizing-a-lift-and-shift-for-performance/

Many organizations begin their cloud journey with a lift-and-shift of applications from on-premise to AWS. This approach involves migrating software deployments with little, or no, modification. A lift-and-shift avoids a potentially expensive application rewrite but can result in a less optimal workload that a cloud native solution. For many organizations, a lift-and-shift is a transitional stage to an eventual cloud native solution, but there are some applications that can’t feasibly be made cloud-native such as legacy systems or proprietary third-party solutions. There are still clear benefits of moving these workloads to AWS, but how can they be best optimized?

In this blog series post, we’ll look at different approaches for optimizing a black box lift-and-shift. We’ll consider how we can significantly improve a lift-and-shift application across three perspectives: performance, cost, and security. We’ll show that without modifying the application we can integrate services and features that will make a lift-and-shift workload cheaper, faster, more secure, and more reliable. In this first blog, we’ll investigate how a lift-and-shift workload can have improved performance through leveraging AWS features and services.

Performance gains are often a motivating factor behind a cloud migration. On-premise systems may suffer from performance bottlenecks owing to legacy infrastructure or through capacity issues. When performing a lift-and-shift, how can you improve performance? Cloud computing is famous for enabling horizontally scalable architectures but many legacy applications don’t support this mode of operation. Traditional business applications are often architected around a fixed number of servers and are unable to take advantage of horizontal scalability. Even if a lift-and-shift can’t make use of auto scaling groups and horizontal scalability, you can achieve significant performance gains by moving to AWS.

Scaling Up

The easiest alternative to scale up to compute is vertical scalability. AWS provides the widest selection of virtual machine types and the largest machine types. Instances range from small, burstable t3 instances series all the way to memory optimized x1 series. By leveraging the appropriate instance, lift-and-shifts can benefit from significant performance. Depending on your workload, you can also swap out the instances used to power your workload to better meet demand. For example, on days in which you anticipate high load you could move to more powerful instances. This could be easily automated via a Lambda function.

The x1 family of instances offers considerable CPU, memory, storage, and network performance and can be used to accelerate applications that are designed to maximize single machine performance. The x1e.32xlarge instance, for example, offers 128 vCPUs, 4TB RAM, and 14,000 Mbps EBS bandwidth. This instance is ideal for high performance in-memory workloads such as real time financial risk processing or SAP Hana.

Through selecting the appropriate instance types and scaling that instance up and down to meet demand, you can achieve superior performance and cost effectiveness compared to running a single static instance. This affords lift-and-shift workloads far greater efficiency that their on-prem counterparts.

Placement Groups and C5n Instances

EC2 Placement groups determine how you deploy instances to underlying hardware. One can either choose to cluster instances into a low latency group within a single AZ or spread instances across distinct underlying hardware. Both types of placement groups are useful for optimizing lift-and-shifts.

The spread placement group is valuable in applications that rely on a small number of critical instances. If you can’t modify your application  to leverage auto scaling, liveness probes, or failover, then spread placement groups can help reduce the risk of simultaneous failure while improving the overall reliability of the application.

Cluster placement groups help improve network QoS between instances. When used in conjunction with enhanced networking, cluster placement groups help to ensure low latency, high throughput, and high network packets per second. This is beneficial for chatty applications and any application that leveraged physical co-location for performance on-prem.

There is no additional charge for using placement groups.

You can extend this approach further with C5n instances. These instances offer 100Gbps networking and can be used in placement group for the most demanding networking intensive workloads. Using both placement groups and the C5n instances require no modification to your application, only to how it is deployed – making it a strong solution for providing network performance to lift-and-shift workloads.

Leverage Tiered Storage to Optimize for Price and Performance

AWS offers a range of storage options, each with its own performance characteristics and price point. Through leveraging a combination of storage types, lift-and-shifts can achieve the performance and availability requirements in a price effective manner. The range of storage options include:

Amazon EBS is the most common storage service involved with lift-and-shifts. EBS provides block storage that can be attached to EC2 instances and formatted with a typical file system such as NTFS or ext4. There are several different EBS types, ranging from inexpensive magnetic storage to highly performant provisioned IOPS SSDs. There are also storage-optimized instances that offer high performance EBS access and NVMe storage. By utilizing the appropriate type of EBS volume and instance, a compromise of performance and price can be achieved. RAID offers a further option to optimize EBS. EBS utilizes RAID 1 by default, providing replication at no additional cost, however an EC2 instance can apply other RAID levels. For instance, you can apply RAID 0 over a number of EBS volumes in order to improve storage performance.

In addition to EBS, EC2 instances can utilize the EC2 instance store. The instance store provides ephemeral direct attached storage to EC2 instances. The instance store is included with the EC2 instance and provides a facility to store non-persistent data. This makes it ideal for temporary files that an application produces, which require performant storage. Both EBS and the instance store are expose to the EC2 instance as block level devices, and the OS can use its native management tools to format and mount these volumes as per a traditional disk – requiring no significant departure from the on prem configuration. In several instance types including the C5d and P3d are equipped with local NVMe storage which can support extremely IO intensive workloads.

Not all workloads require high performance storage. In many cases finding a compromise between price and performance is top priority. Amazon S3 provides highly durable, object storage at a significantly lower price point than block storage. S3 is ideal for a large number of use cases including content distribution, data ingestion, analytics, and backup. S3, however, is accessible via a RESTful API and does not provide conventional file system semantics as per EBS. This may make S3 less viable for applications that you can’t easily modify, but there are still options for using S3 in such a scenario.

An option for leveraging S3 is AWS Storage Gateway. Storage Gateway is a virtual appliance than can be run on-prem or on EC2. The Storage Gateway appliance can operate in three configurations: file gateway, volume gateway and tape gateway. File gateway provides an NFS interface, Volume Gateway provides an iSCSI interface, and Tape Gateway provides an iSCSI virtual tape library interface. This allows files, volumes, and tapes to be exposed to an application host through conventional protocols with the Storage Gateway appliance persisting data to S3. This allows an application to be agnostic to S3 while leveraging typical enterprise storage protocols.

Using S3 Storage via Storage Gateway

Figure 1: Using S3 Storage via Storage Gateway

Conclusion

A lift-and-shift can achieve significant performance gains on AWS by making use of a range of instance types, storage services, and other features. Even without any modification to the application, lift-and-shift workloads can benefit from cutting edge compute, network, and IO which can help realize significant, meaningful performance gains.

About the author

Dr. Jonathan Shapiro-Ward is an AWS Solutions Architect based in Toronto. He helps customers across Canada to transform their businesses and build industry leading cloud solutions. He has a background in distributed systems and big data and holds a PhD from the University of St Andrews.

Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis

Post Syndicated from David Bailey original https://aws.amazon.com/blogs/architecture/stream-amazon-cloudwatch-logs-to-a-centralized-account-for-audit-and-analysis/

A key component of enterprise multi-account environments is logging. Centralized logging provides a single point of access to all salient logs generated across accounts and regions, and is critical for auditing, security and compliance. While some customers use the built-in ability to push Amazon CloudWatch Logs directly into Amazon Elasticsearch Service for analysis, others would prefer to move all logs into a centralized Amazon Simple Storage Service (Amazon S3) bucket location for access by several custom and third-party tools. In this blog post, I will show you how to forward existing and any new CloudWatch Logs log groups created in the future to a cross-account centralized logging Amazon S3 bucket.

The streaming architecture I use in the destination logging account is a streamlined version of the architecture and AWS CloudFormation templates from the Central logging in Multi-Account Environments blog post by Mahmoud Matouk. This blog post assumes some knowledge of CloudFormation, Python3 and the boto3 AWS SDK. You will need to have or configure an AWS working account and logging account, an IAM access and secret key for those accounts, and a working environment containing Python and the boto3 SDK. (For assistance, see the Getting Started Resource Center and Start Building with SDKs and Tools.) All CloudFormation templates and Python code used in this article can be found in this GitHub Repository.

Setting Up the Solution

You need to create or use an existing S3 bucket for storing CloudFormation templates and Python code for an AWS Lambda function. This S3 bucket is referred to throughout the blog post as the <S3 infrastructure-bucket>. Ensure that the bucket does not block new bucket policies or cross-account access by checking the bucket’s Permissions tab and the Public access settings button.

You also need a bucket policy that allows each account that needs to stream logs to access it when we create the AWS Lambda function below. To do so, update your bucket policy to include each new account you create and the <S3 infrastructure-bucket> ARN from the top of the Bucket policy editor page to modify this template:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                  "03XXXXXXXX85",
                  "29XXXXXXXX02",
                  "13XXXXXXXX96",
                  "37XXXXXXXX30",
                  "86XXXXXXXX95"
                ]
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::<S3 infrastructure-bucket>",
                "arn:aws:s3:::<S3 infrastructure-bucket>/*"
            ]
        }
    ]
}

Clone a local copy of the CloudFormation templates and Python code from the GitHub repository. Compress the CentralLogging.py and lambda.py into a .zip file for the lambda function we create below and name it AddSubscriptionFilter.zip. Load these local files into the <S3 infrastructure-bucket>. I recommend using folders called /python for the .py files, /lambdas for the AddSubscriptionFilter.zip file and /cfn for the CloudFormation templates.

Multi-Account Configuration and the Central Logging Account

One form of multi-account configuration is the Landing Zone offering, which provides a core logging account for storing all logs for auditing. I use this account configuration as an example in this blog post. Initially, the Landing Zone setup creates several stack sets and resources, including roles, security groups, alarms, lambda functions, a cloud trail stream and an S3 bucket.

If you are not using a Landing Zone, create an appropriately named S3 bucket in the account you have chosen as a logging account. This S3 bucket will be referred to later as the <LoggingS3Bucket>. To mimic what the Landing Zone calls its logging bucket, you can use the format aws-landing-zone-logs-<Account Number><Region>, or simply pick an appropriate name for the centralized logging location. In a production environment, remember that it is critical to lock down the access to logging resources and the permissions allowed within the account to prevent deletion or tampering with the logs.

Figure 1 - Initial Landing Zone logging account resources

Figure 1 – Initial Landing Zone logging account resources

The S3 bucket – aws-landing-zone-logs-<Account Number><Region> is the most important resource created by the stack-sets for logging purposes. It contains all of the logs streamed to it from all of the accounts. Initially, the Landing Zone only sends the AWS CloudTrail and AWS Config logs to this S3 bucket.

In order to send all of the other CloudWatch Logs that are necessary for auditing, we need to add a destination and streaming mechanism to the logging account.

Logging Account Insfrastructure

The additional infrastructure required in the central logging account provides a destination for the log group subscription filters and a stream for log events that are sent from all accounts and appropriate regions to load them into the <LoggingS3Bucket> repository. The selection of these particular AWS resources is important, because Kinesis Data Streams is the only resource currently supported as a destination for cross-account CloudWatch Logs subscription filters.

The centralLogging.yml CloudFormation template automates the creation of the entire required infrastructure in the core logging account. Make sure to run it in each of the regions in which you need to centralize logs. The log group subscription filter and destination regions must match in order to successfully stream the logs.

Installation Instructions:

  1. Modify the centralLogging.yml template to add your account numbers for all of the accounts you want to stream logs from into the DestinationPolicy where you see the <AccountNumberHere> placeholders. Remove any unused placeholders.
  2. In the same DestinationPolicy, modify the final arn statement, replacing <region> with the region it will be run in (e.g., us-east-1), and the <logging account number> with the account number of the logging account where this template is to be run.
  3. Log in to the core logging account and access the AWS management console using administrator credentials.
  4. Navigate to CloudFormation and click the Create Stack button.
  5. Select Specify an Amazon S3 template URL and enter the Link for the centralLogging.yml template found in the <S3 infrastructure-bucket>.
  6. Enter a stack name, such as CentralizedLogging, and the one parameter called LoggingS3Bucket. Enter in the ARN of the logging bucket: arn:aws:s3::: <LoggingS3Bucket>. This can be obtained by opening the S3 console, clicking on the bucket icon next to this bucket, and then clicking the Copy Bucket ARN button.
  7. Skip the next page, acknowledge the creation of IAM resources, and Create the stack.
  8. When the stack completes, select the stack name to go to stack details and open the Outputs. Copy the value of the DestinationArnExport, which will be needed as a parameter for the script in the next section.

Upon successful creation of this CloudFormation stack, the following new resources will be created:

  • Amazon CloudWatch Logs Destination
  • Amazon Kinesis Stream
  • Amazon Kinesis Firehose Stream
  • Two AWS Identity and Access Management (IAM) Roles
Figure 2 - New infrastructure required in the centralized logging account

Figure 2 – New infrastructure required in the centralized logging account

Because the Landing Zone is a multi-account offering, the Log Destination is required to be the destination for all subscription filters. The key feature of the destination is its DestinationPolicy. Whenever a new account is added to the environment, its account number needs to be added to this DestinationPolicy in order for logs to be sent to it from the new account. Add the new account number in the centralLogging.yml CloudFormation template, and run an update in CloudFormation to complete the addition. A sample Destination Policy looks like this:

{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Effect" : "Allow",
      "Principal" : {
        "AWS" : [
          "03XXXXXXXX85",
          "29XXXXXXXX02",
          "13XXXXXXXX96",
          "37XXXXXXXX30",
          "86XXXXXXXX95"
        ]
      },
      "Action" : "logs:PutSubscriptionFilter",
      "Resource" : "arn:aws:logs:<Region>:<LoggingAccountNumber>:destination:CentralLogDestination"
    }
  ]
}

The Kinesis Stream get records from the Logs Destination and holds them for 48 hours. Kinesis Streams scale by adding shards. The CloudFormation template starts the stream with two shards. You need to monitor this as instances and applications are deployed into the accounts, however, because all CloudWatch log objects will flow through this stream, and it will need to be scaled up at some point. To scale, change the number of shards (ShardCount) in the Kinesis Stream resource (KinesisLoggingStream) to the required number. See the Amazon Kinesis Data Streams FAQ documentation to confirm the capacity and throughput of each shard.

Kinesis Firehose provides a simple and efficient mechanism to retrieve the records from the Kinesis Stream and load them into the <LoggingS3Bucket> repository. It uses the CloudFormation template parameter to know where to load the logs. All of the CloudWatch logs loaded by Firehose will be under the prefix /CentralizedAccountsLog. The buffering hints for Firehose suggest that the logs be loaded every 5 minutes or 50 MB. Leave the CompressionFormat UNCOMPRESSED, since the logs are already compressed.

There are two AWS Identity and Access Management (IAM) roles created for this infrastructure. The first, CWLtoKinesisRole is used by the destination to allow CloudWatch Logs from all regions to use the destination to put the log object records into the Kinesis Stream, as well as to pass the role. The second, FirehoseDeliveryRole, allows Firehose to get the log object records from the Kinesis Stream, and then to load them into S3 logging bucket.

Once you have successfully created this infrastructure, the next step is to add the subscription filters to existing log groups.

Adding Subscription Filters to Existing Log Groups

The next step in the process is to add subscription filters for the Log Destination in the core logging account to all existing log groups. Several log groups are created by the Landing Zone, or you may have created them by using various AWS services or by logging application events. For every new AWS account, you will need to run the init_account_central_logging.py Python script to add the subscription filters to all the existing log groups.

The init_account_central_logging.py script takes one parameter, which is the Log Destination ARN. Use the Destination ARN you copied from the stack details output in the previous section as the parameter to the script.

The init_account_central_logging.py script first adds this Destination ARN to the AWS Systems Manager Parameter Store so that the core logic that creates the subscription filter can use it. The script then gets a list of all existing log groups, iterates over them, deletes any existing subscription filters (because there can only be one subscription filter per log group and attempting to create another would cause an error), and then adds the new subscription filter to the centralized logging account to the Log Destination.

Figure 3 - Run script to add subscription filters to existing log groups

Figure 3 – Run script to add subscription filters to existing log groups

Installation Instructions:

  1. Make sure that Python and boto3 are installed and accessible in the client computer – consider loading into a virtual environment to keep dependencies separate.
  2. Set the AWS_PROFILE environment variable to the appropriate AWS account profile.
  3. Log in to the proper account, and obtain administrator or other credentials with appropriate permissions, and add the account access key and secret key to the AWS credentials file.
  4. Set the region and output in the AWS config file.
  5. Download and place two python files into a working directory: init_account_central_logging.py and CentralLogging.py.
  6. Run the script using the command python3 ./init_account_central_logging.py -d <LogDestinationArn>.

Use the AWS Management Console to validate the results. Navigate to CloudWatch Logs and view all of the log groups. Each one should now have a subscription filter named “Logs (CentralLogDestination).”

Automatically Adding Subscription Filters to New Log Groups

The final step to set up the centralized log streaming capability is to run a CloudFormation script to create resources that automatically add subscription filters to new log groups. New log groups are created in accounts by resources (e.g., Lambda functions) and by applications. A subscription filter must be added to every new log group in order to deliver its log events to the logging account,

The AddSubscriptionFilter.yml CloudFormation template contains resources to automatically add subscription filters.

First, it creates a role that allows it to access the lambda code that is stored in a centralized location – the <S3 infrastructure-bucket>. (Remember that its S3 bucket policy must contain this account number in order to access the lambda code.)

Second, the template creates the AddSubscriptionLambda, which reuses the core logic shared by the script in the last section. It retrieves the proper destination from the Parameter Store, deletes any existing subscription filter from the log group, and adds the new subscription filter to the newly created log group. This lambda function is triggered by a CloudWatch event rule.

Third, the CloudFormation creates a Lambda Permission, which allows the event trigger to invoke this particular lambda.

Finally, the CloudFormation template creates an Amazon CloudWatch Events Rule that acts as a trigger for the lambda. This rule looks for an event coming from CloudTrail that signals the creation of a new log group. For each create log group event found, it invokes the AddSubscriptionLambda.

Figure 4 - Infrastructure to automatically add a subscription filter to a new log group and the log flow to the centralized account

Figure 4 – Infrastructure to automatically add a subscription filter to a new log group and the log flow to the centralized account

Installation Instructions:

(Important note: This functionality requires that the LogDestination parameter be properly set to the LogDestinationArn in the Parameter Store before the Lambda will run successfully. The script in the previous step sets this parameter, or it can be done manually. Make certain that the destination specified is in this same region.)

  1. Ensure that the <S3 infrastructure-bucket> has the AddSubscriptionFilter.zip file containing the Python code files lambda.py and CentralLogging.py.
  2. Log in to the appropriate account, and access using administrator credentials. Make sure that the region is set properly.
  3. Navigate to Cloudformation and click the Create Stack button.
  4. Select Specify an Amazon S3 template URL and enter the Link for the AddSubscriptionFilter.yml template found in <S3 infrastructure-bucket>
  5. Enter a stack name, such as AddSubscription.
  6. Enter the two parameters, the <S3 infrastructure-bucket> name (not ARN) and the folder and file name (e.g., lambdas/AddSubscriptionFilter.zip)
  7. Skip the next page, acknowledge the creation of IAM resources, and Create the stack.

In order to test that the automated addition of subscription filters is working properly, use the AWS Management Console to navigate to CloudWatch Logs and click the Actions button. Select Create New Log Group and enter a random log group name, such as “testLogGroup.” When first created, the log group will not have a subscription filter. After a few minutes, refresh the display and you should see the new subscription filter on the log group. At this point, you can delete the test log group.

New Account Setup

As a reminder, when you add new accounts that you want to have stream log events to the central logging account, you will need to configure the new accounts in two places in order for this functionality to work properly.

First, add the account number to the LoggingDestination property DestinationPolicy in the centralLogging.yml template. Then, update the CloudFormation stack.

Second, modify the bucket policy for the <S3 infrastructure-bucket>. Select the Permissions tab, then the Bucket Policy button. Add the new account to allow cross-account access to the lambda code by adding the line “arn:aws:iam::<new account number>:root” to the Principal.AWS list.

Conclusion

Centralized logging is a key component in enterprise multi-account architectures. In this blog post, I have built on the central logging in multi-account environments streaming architecture to automatically subscribe all CloudWatch Logs log groups to send all log events to an S3 bucket in a designated logging account. The solution uses a script to add subscription filters to existing log groups, and a lambda function to automatically place a subscription filter on all new log groups created within the account. This can be used to forward application logs, security logs, VPC flow logs, or any other important logs that are required for audit, security, or compliance purposes.

About the author

David BaileyDavid Bailey is a Cloud Infrastructure Architect with AWS Professional Services specializing in serverless application architecture, IoT, and artificial intelligence. He has spent decades architecting and developing complex custom software applications, as well as teaching internationally on object-oriented design, expert systems, and neural networks.

 

 

New: Application integration with AWS Cloud Map for service discovery

Post Syndicated from AWS Admin original https://aws.amazon.com/blogs/architecture/new-application-integration-with-aws-cloud-map-for-service-discovery/

By: Alexandr Moroz, Sr. Product Manager, Amazon Route 53; Madhuri Peri, Sr. IoT Architect, AWS Professional Services; Aaron Molitor, Sr. Infrastructure Architect, AWS Professional Services; and Sarma Palli, Sr. DevOps Architect, AWS Professional Services

AWS Cloud Map enables you to map your cloud. You can define friendly names for any resource, such as Amazon S3 buckets, Amazon DynamoDB tables, Amazon SQS queues, or custom cloud services built on Amazon EC2, Amazon ECS, Amazon EKS, or AWS Lambda. Your applications can then discover resource location and metadata by friendly name using the AWS SDK and authenticated API queries. Resources can be further filtered and discovered by custom attributes such as deployment stage or version.

What’s new with API service discovery

If you want an enterprise application component such as a database hosted on Amazon EC2 instances to provide an endpoint to your database service, you have to register your applications’ EC2 IP address with AWS Cloud Map. You could register additional metadata attributes, like INSTANCE_STATUS, and then use this attribute with AWS Cloud Map to identify when the service is READY so that querying applications can only attempt a connection when they see a READY status in AWS Cloud Map. In cases where different microservices or enterprise applications have endpoints that have to be discovered, you can use AWS Cloud Map to register those as well. Examples of such endpoints include types of ELB load balancers, including ELB Classic, Application Load Balancers (ALB), and Network Load Balancers (NLB) with Auto Scaling groups.

Compute stack choices

Modern application architectures require a way to expose and advertise the service endpoint, register and de-register the endpoints, and query them. The dependencies of applications are expected to be handled by the applications themselves where a service registry becomes critical.

These microservices could follow different patterns of architecture lending themselves to use:

  1. Traditional workloads running on Amazon EC2 fronted by Auto Scaling groups or an ELB load balancer such as ELB Classic, Application Load Balancer, or Network Load Balancer.
  2. Amazon API Gateway and AWS Lambda for event-driven workflows.
  3. Container-based workloads on Amazon Elastic Container Service (ECS) using EC2 or Fargate launch types and Amazon Elastic Container Service for Kubernetes (EKS) for workloads that run as services (long-running) or daemons or run to completion (Batch / cron type).

This image shows a typical enterprise application composed of components that run different architectures. There is a web server running on Amazon EKS, a backend on Amazon ECS, a serverless event registration service, and payments running on EC2 Auto Scaling groups (ASG) while leveraging databases on Amazon Relational Database Service (RDS).

 

From a service discovery perspective, this is how the applications would want to be discovered and queried:

Let’s see how we can register each of these microservices (which are running on different cloud compute products) with AWS Cloud Map using both DNS-based and API-based service discovery and leveraging attributes for discovery when components are ready for traffic.

Microservice endpoints and discovery

AWS Cloud Map is a managed solution that lets you map logical names to the components/resources for an application. It allows applications to discover the resources using one of the AWS SDKs, RESTful API calls, or DNS queries. AWS Cloud Map serves registered resources, which can be Amazon DynamoDB tables, Amazon Simple Queue Service (SQS) queues, any higher-level application services that are built using EC2 instances or ECS tasks, or using a serverless stack.

When you register a resource, you can specify attributes and clients that can use the attributes to filter which resources are to be returned. For example, an application can request resources in a particular deployment stage, like Gamma or Prod. Additionally, you can choose to enable health checking for your IP-based resources, ensuring that AWS Cloud Map returns only healthy endpoints. Each API call is authenticated, and developers can control access to service locations and configuration using AWS Identity and Access Management (IAM).  This ensures that clients always discover the services that they’re authorized to use.

Let’s cover fundamentals

There are two aspects to service discovery:

  • The microservices themselves that register/de-register
  • Other microservices that are discover / query microservices

To register a microservice, follow these steps:

  1. Create a namespace.
  2. Create a service.
  3. Register instances with the service.

Steps 1 and 2  are performed once for each service. A utility function for registration and de-registration of a microservice has to be created. This utility function can be invoked for microservices regardless of the compute stack choice and deployed through your CI/CD/DevOps processes.

Step 3 is an ongoing operation that has to be updated each time the underlying EC2 compute that powers it changes. Examples include: EC2 Amazon Machine Image (AMI) changes, code changes for the service, and version changes.

Creating a namespace

A namespace is a logical group of services that share the same domain name, such as example1.example.com or example2.example.com. If you want these namespaces to be queried from only within your VPC, opt for a private namespace. If you want them to be accessible over the Internet, create a public namespace. In our example, the public namespace could be example1, but the total count of use of items in example1 in a tracker/reporting service could be an internal service.

Microservice using DNS-based service discovery:

Microservice using API-based service discovery:

Creating a service

When you register a service, AWS Cloud Map will create a record in the hosted zone – which is a combination of the name of the service and the name of the namespace. You could optionally define a health check for the service, too.

If the service you are creating is meant for DNS-based discovery using one of the A, AAAA, or SRV records, then you can create your service using the following syntax. Examples of this could be your application code running on an EC2 instance or as a container (ECS/EKS).

For services that are meant to be used only in an API-only namespace, the API call would look like this:

Register the compute backend with the service

Container IP address register/de-register

Amazon ECS is tightly integrated with AWS Cloud Map to enable service discovery for compute workloads running in ECS. When you enable service discovery for ECS services, it automatically keeps track of all task instances in AWS Cloud Map. Now your applications can discover them using DNS queries and AWS Cloud Map DiscoverInstances API calls. The ECS control plane that issues the calls would register the IP address of the task (and containers) with the AWS Cloud Map.

When the task goes away – either because a newer version has been deployed or there is a crash or a restart – the ECS control plane handles the de-registration process as well.

If you are using ECS for running containers, this is done seamlessly with ECS and AWS Cloud Map API integration.

API Gateway URL and AWS Lambda

When you create a microservice with an API namespace, you could use any attributes you prefer, without providing the IP/port information.

EC2 instance IP address registration and de-registration

As the EC2 instances are coming online, the userdata section of the bootstrap configuration will issue commands to register the EC2 instance’s IP address with the service. An alternate approach would be to run a Lambda function that runs against a microservice’s Auto Scaling group, lists the IP addresses, and registers the instance against the service.

If you are using an EC2 instance, if the instance is integrated with an Auto Scaling group, lifecycle hooks could also be used to run the de-register scripts. Another approach would be to use a Lambda function that runs periodically against an Auto Scaling group or even fires on Auto Scaling group notification events.

Query/Discovery

Both DNS and API service discovery are now supported by the AWS Cloud Map service discovery. Supported DNS record types are – A, AAAA, SRV, and CNAME.

It is typical in a microservices architecture for a service to be able to discover other services. We recommend that you query only by name and/or endpoint, and do not use the IP address of the compute stack (AWS Lambda / container/ EC2) that is backing the service.

The API commands list_services and get_services provide the information on what services are available and their corresponding details.

A DNS protocol also has clients caching the responses, so make sure that you handle caching settings. AWS Cloud Map uses regional endpoints here. Any A records created will use either a WEIGHTED response or MULTIVALUE answer policy. If you are using a Java-based compute stack, you might not want to choose DNS-based service discovery as the JVM caches DNS name lookups. When the JVM resolves a hostname to an IP address, it caches the IP address for a specified period of time, TTL. In such cases, you could use API-based service discovery and leverage the same approach as your other microservices that can use AWS Cloud Map.

DiscoverInstances API

DiscoverInstances API discovers registered instances for a specified namespace and service using regional endpoints. Updates to your services, such as new instances registered or existing instances removed, will be available in the API faster than via DNS. The API provides the ability to decorate the resources with additional metadata (service attributes) that can be used during discovery. For example, get the services with attributes of blue or green or other application attributes. These attributes can be used to complement health checks while performing discovery (such as finding out whether the instance is ready or not).

Here is a screenshot that shows how the registered ECS task instances appear in the AWS Cloud Map console:

The idea is that as the container or EC2 instance comes online or goes offline, it needs to issue a call to the AWS Cloud Map API to register or de-register the compute IP address.

Get started by visiting the AWS Cloud Map page. To learn more, take a look at the demo code in the GitHub repo here. If your compute workloads use EKS, please refer to this blog post that shows how to make EKS automatically publish all services in AWS Cloud Map.

AWS Storage Update: Amazon S3 & Amazon S3 Glacier Launch Announcements for Archival Workloads

Post Syndicated from AWS Admin original https://aws.amazon.com/blogs/architecture/amazon-s3-amazon-s3-glacier-launch-announcements-for-archival-workloads/

By Matt Sidley, Senior Product Manager for S3

Customers have built archival workloads for several years using a combination of S3 storage classes, including S3 Standard, S3 Standard-Infrequent Access, and S3 Glacier. For example, many media companies are using the S3 Glacier storage class to store their core media archives. Most of this data is rarely accessed, but when they need data back (for example, because of breaking news), they need it within minutes. These customers have found S3 Glacier to be a great fit because they can retrieve data in 1-5 minutes and save up to 82% on their storage costs. Other customers in the financial services industry use S3 Standard to store recently generated data, and lifecycle older data to S3 Glacier.

We launched Glacier in 2012 as a secure, durable, and low-cost service to archive data. Customers can use Glacier either as an S3 storage class or through its direct API. Using the S3 Glacier storage class is popular because many applications are built to use the S3 API and with a simple lifecycle policy, older data can be easily shifted to S3 Glacier. S3 Glacier continues to be the lowest-cost storage from any major cloud provider that durably stores data across three Availability Zones or more and allows customers to retrieve their data in minutes.

We’re constantly listening to customer feedback and looking for ways to make it easier to build applications in the cloud. Today we’re announcing six new features across Amazon S3 and S3 Glacier.

Amazon S3 Object Lock

S3 Object Lock is a new feature that prevents data from being deleted during a customer-defined retention period. You can use Object Lock with any S3 storage class, including S3 Glacier. There are many use cases for S3 Object Lock, including customers who want additional safeguards for data that must be retained, and for customers migrating from existing write-once-read-many (WORM) systems to AWS. You can also use S3 Lifecycle policies to transition data and S3 Object Lock will maintain WORM protection as your data is tiered.

S3 Object Lock can be configured in one of two modes: Governance or Compliance. When deployed in Governance mode, only AWS accounts with specific IAM permissions are able to remove the lock. If you require stronger immutability to comply with regulations, you can use Compliance mode. In Compliance mode, the lock cannot be removed by any user, including the root account. Take a look here:

S3 Object Lock is helpful in industries where long-term records retention is mandated by regulations or compliance rules. S3 Object Lock has been assessed for SEC Rule 17a-4(f), FINRA Rule 4511, and CFTC Regulation 1.31 by Cohasset Associates. Cohasset Associates is a management consulting firm specializing in records management and information governance. Read more and find a copy of the Cohasset Associates Assessment report in our documentation here.

New S3 Glacier Features

One of the things we hear from customers about using S3 Glacier is that they prefer to use the most common S3 APIs to operate directly on S3 Glacier objects. Today we’re announcing the availability of S3 PUT to Glacier, which enables you to use the standard S3 “PUT” API and select any storage class, including S3 Glacier, to store the data. Data can be stored directly in S3 Glacier, eliminating the need to upload to S3 Standard and immediately transition to S3 Glacier with a zero-day lifecycle policy. You can “PUT” to S3 Glacier like any other S3 storage class:

Many customers also want to keep a low-cost durable copy of their data in a second region for disaster recovery. We’re also announcing the launch of S3 Cross-Region Replication to S3 Glacier. You can now directly replicate data into the S3 Glacier storage class in a different AWS region.

Restoring Data from S3 Glacier

S3 Glacier provides three restore speeds for you to access your data: expedited (to retrieve data in 1-5 minutes), standard (3-5 hours), or bulk (5-12 hours). With S3 Restore Speed Upgrade, you can now issue a second restore request at a faster restore speed and get your data back sooner. This is useful if you originally requested standard or bulk speed, but later determine that you need a faster restore speed.

After a restore from S3 Glacier has been requested, you likely want to know when the restore completes. Now, with S3 Restore Notifications, you’ll receive a notification when the restoration has completed and the data is available. Many applications today are being built using AWS Lambda and event-driven actions, and you can now use the restore notification to automatically trigger the next step in your application as soon as S3 Glacier data is restored. For example, you can use notifications and Lambda functions to package and fulfill digital orders using archives restored from S3 Glacier.

Here, I’ve set up notifications to fire when my restores complete so I can use Lambda to kick off a piece of analysis I need to run:

You might need to restore many objects from S3 Glacier; for example, to pull all of your log files within a given time range. Using the new feature in Preview, you can provide a manifest of those log files to restore and, with one request, initiate a restore on millions or even trillions of objects just as easily as you can on just a few. S3 Batch Operations automatically manages retries, tracks progress, sends notifications, generates completion reports, and delivers events to AWS CloudTrail for all changes made and tasks executed.

To get started with the new features on Amazon S3, visit https://aws.amazon.com/s3/. We’re excited about these improvements and think they’ll make it even easier to build archival applications using Amazon S3 and S3 Glacier. And we’re not yet done. Stay tuned, as we have even more coming!

Your guide to AWS Digital User Engagement at re:Invent 2018

Post Syndicated from Hannah Nilsson original https://aws.amazon.com/blogs/messaging-and-targeting/your-guide-to-aws-digital-user-engagement-at-reinvent-2018/

Plan your Digital User Engagement agenda at re:Invent 2018

With roughly 50,000 attendees expected to descend upon Las Vegas next week for the seventh AWS re:Invent, our re:Invent 2018 agenda has unsurprisingly grown along with the crowd size. This year, we have added a full day of additional content, expanded our campus, added expo hours, and included over 1,000+ breakout sessions, hackathons, bootcamps, workshops, and more ways to get hands-on experiences with AWS. With so much going on, we wanted to create a quick guide for how you can connect with the AWS Digital User Engagement team to learn what’s new with the technology and tactics for effectively engaging your customers.

Meet us in the Expo Hall

Stop by the Mobile booth in the Venetian on Level 2 to learn more about our latest features (psst—did you know we just released a voice channel and event-based campaigns?), pick up some Amazon Pinpoint swag, and discuss strategies for transformative digital user engagement with our team of experts.

Expo hours are:

  • Monday, 11/26 from 4pm-7pm PT
  • Tuesday, 11/27 from 8am-8pm PT
  • Wednesday, 11/28 from 10am-6pm PT
  • Thursday, 11/29 from 10am-4pm PT

Livestream our Launchpad Session with Disney Streaming Services:

Unable to attend re:Invent in person? Check out our session with Disney Streaming Services live on Twitch. William Liu and Jimmy Tam from Disney Streaming will join us to discuss how migrating to Amazon Pinpoint from their internal digital engagement platform has allowed them to send billions of messages to their end users in real time.

  • Tuesday, 11/27, 5:15 PM – 5:35 PM  PT
  • Watch it live on Twitch, or stop by the Launchpad stage in the Expo Hall

Attend the Digital User Engagement Sessions:

Must-attend sessions:

(DIG302-L) Leadership Session: Overview of Amazon Digital User Engagement Solutions
Wednesday, 11/27, 3:15 PM – 4:15 PM PT – Venetian, Level 4, Delfino 4002
In this session, Disney Streaming Services will share how they utilize AWS Digital User Engagement platform to send billions of users relevant content in real time. Simon Poile, GM of AWS Digital User Engagement, will also describe how AWS provides the Amazon customer-centric culture of innovation, key technology building blocks, and a user engagement platform to help companies better engage their users. Plus, session attendees can claim a special piece of Amazon Pinpoint swag.

Enable Your Marketing Teams to Engage Users with Relevant & Personalized Content (DIG204)
Monday, Nov 26, 4:45 PM – 5:45 PM– Aria East, Level 2, Mariposa 8
This chalk talk is delivered by the innovation team from Claro, a major Brazilian telecommunications company that serves over 50 million customers with broadband, mobile, cable TV, and video-on-demand services. Members of the Claro innovation team describe how they enable their marketing departments to engage customers through campaigns that send personalized messages, such as billing reminders, updates on internet credits, and upcoming program alerts. They share how they then measure customer engagement and user behaviors, all using Amazon Pinpoint.

Game On! Building Hulu’s Real-Time Notification Platform for Live TV with Amazon Pinpoint (MOB304)
Wednesday, Nov 28, 1:45 PM – 2:45 PM– Venetian, Level 4, Marcello 4505
Notifying their viewers when their favorite teams are playing helps Hulu drive growth and improve viewer engagement, but building this feature was a complex process. Managing their live TV metadata while generating audiences in real-time in high scalability scenarios posed unique challenges for the engineering team at Hulu. In this session, Hulu will talk about their challenges in building their real-time notification platform, how Pinpoint helped them with their goals, and how they architected their solution for global scale and deliverability.

Full session list for re:Invent attendees:

Monday, November 26, 2o18

Perform Social Media Sentiment Analysis with Amazon Pinpoint & Amazon Comprehend (MOB314)
Monday, 11/26, 10:00 AM – 12:15 PM PT – MGM, Level 1, Grand Ballroom 124
In this workshop, we show you how to easily deploy an AWS solution that ingests all Tweets from any Twitter handle, uses Amazon Comprehend to generate a sentiment score, and then automatically engages customers with a dynamic, personalized message. The intended audience is developers and marketers who want to leverage AWS to create powerful user engagement scenarios. We highlight how quickly you can deploy a machine learning marketing solution. We cover Amazon Pinpoint, the AWS user engagement service, and Amazon Comprehend, the AWS natural language processing service that uses artificial intelligence and machine learning to find insights and relationships in text.

Delight your customers through natural language conversational experiences – powered by Amazon Lex and Amazon Pinpoint (AIM360)
Monday, 11/26, 11:30 AM – 12:30 PM PT – Mirage, St. Croix A
In this chalk talk, we first describe various use cases to engage customers through natural language conversations. We then showcase how to prepare, implement, and continuously improve such solutions. We use Amazon Lex to drive the chatbot interactions and capture user events in Amazon Pinpoint analytics. The event data is then used to measure user engagement and sentiment within conversations.

Understand & Remediate Message Deliverability Issues to Improve Customer Reach (DIGF202)
Monday, 11/26, 12:15 PM – 1:15 PM PT – Mirage, St. Thomas B
Several factors determine whether your messages reach your recipients. In this chalk talk, learn some of the critical factors that affect the deliverability of your messages across messaging channels, such as email and SMS. We walk you through some of the best practices, and we introduce you to deliverability tools, such as Phone Number Validate. With these tools, you can improve your customer reach, increasing the effectiveness of your campaigns and ultimately drive more revenue. The intended audience is developers and marketers who send outbound messages. We cover the AWS services Amazon Pinpoint and Amazon Simple Email Service (Amazon SES).

Effectively Engage Millions of Users in Seconds (MOB321 -R)
Monday, 11/26, 1:45 PM -2:45 PM PT – Aria East, Level 1, Joshua 4
In this session, we describe the challenges that companies face with high volume messaging-based engagement and the solutions AWS provides. Learn from current customers about how AWS can be used to message millions of users in seconds, while being cost effective and maintaining high deliverability rates. The intended audience is developers who are supporting their company’s growth or marketing activities using AWS services.

The repeat of this session (MOB321 -R1) will take place: Tuesday, 11/27, 5:30pm-6:30pm PT – Bellagio, Level 1, Grand Ballroom 1.

How to Use Amazon Pinpoint and Amazon Kinesis for Events-Based Messaging (MOB408-R)  
Monday, Nov 26, 2:30 PM – 3:30 PM – Aria West, Level 3, Starvine 10, Table 7
In this session, we demonstrate when and how to engage users in real time based on events and user behaviors to drive contextual and relevant user interactions. The intended audience is developers who are supporting marketing activities using AWS services.

The repeats of this session will take place:

Enable Your Marketing Teams to Engage Users with Relevant & Personalized Content (DIG204)
Monday, Nov 26, 4:45 PM – 5:45 PM– Aria East, Level 2, Mariposa 8
This chalk talk is delivered by the innovation team from Claro, a major Brazilian telecommunications company that serves over 50 million customers with broadband, mobile, cable TV, and video-on-demand services. Members of the Claro innovation team describe how they enable their marketing departments to engage customers through campaigns that send personalized messages, such as billing reminders, updates on internet credits, and upcoming program alerts. They share how they then measure customer engagement and user behaviors, all using Amazon Pinpoint.

Tuesday, November 27, 2018

Personalize User Targeting through ML & Measure User Engagement with Your Brand (DIG203)
Tuesday, Nov 27, 10:00 AM – 11:00 AM– Mirage, St. Croix A
In this session, we describe the critical role that multi-channel analytics plays in targeting users and how to use machine learning (ML) to predict the best content, channel, and time to engage. The intended audience is developers and marketers who need to target users and measure their engagement levels. We cover the AWS service Amazon Pinpoint.

Engage Users in Real-Time through Event-Based Messaging (MOB322-R)
Tuesday, Nov 27, 4:00 PM – 5:00 PM– Venetian, Level 4, Lando 4305
In this session, we describe when and how to engage users in real time, based on external events and user behaviors, to drive contextual and relevant user interactions. The intended audience is developers who support marketing activities using AWS services. We cover Amazon Pinpoint and Amazon Kinesis in this session.

The repeat of this session (MOB322-R1) will take place: Thursday, Nov 29, 1:00 PM – 2:00 PM– Venetian, Level 3, Murano 3202

Wednesday, November 28, 2018

Game On! Building Hulu’s Real-Time Notification Platform for Live TV with Amazon Pinpoint (MOB304)
Wednesday, Nov 28, 1:45 PM – 2:45 PM– Venetian, Level 4, Marcello 4505
Notifying their viewers when their favorite teams are playing helps Hulu drive growth and improve viewer engagement, but building this feature was a complex process. Managing their live TV metadata while generating audiences in real-time in high scalability scenarios posed unique challenges for the engineering team at Hulu. In this session, Hulu will talk about their challenges in building their real-time notification platform, how Pinpoint helped them with their goals, and how they architected their solution for global scale and deliverability.

Thursday, November 29, 2018:

Listen to Your Customers’ Social Voice & Engage Them with Delightful Experiences (DIG301)
Thursday, Nov 29, 12:15 PM – 1:15 PM– MGM, Level 1, Grand Ballroom 116
In this session, we demonstrate how to easily deploy an AWS solution that ingests all Tweets from any Twitter handle, uses Amazon Comprehend to generate a sentiment score, and then automatically engages customers with a personalized message. The intended audience includes developers and marketers who want to leverage AWS to create powerful user engagement scenarios. We highlight how quickly a machine-learning marketing solution can be deployed. We cover the AWS services Amazon Pinpoint, a digital user engagement service, and Amazon Comprehend, a natural language processing service that uses artificial intelligence and machine learning to find insights and relationships in text

Please note that session times and locations are subject to change. View the session catalogue for the most up-to-date information. For more helpful hints on how to make the most of your re:Invent experience, please visit our “How to re:Invent” guides.

Which session are you most excited about? Let us know in the comments!

Event-based campaigns let you automatically send messages to your customer when they perform certain actions

Post Syndicated from Zach Barbitta original https://aws.amazon.com/blogs/messaging-and-targeting/event-based-campaigns-let-you-automatically-send-messages-to-your-customer-when-they-perform-certain-actions/

Today, we added a new campaign management feature to Amazon Pinpoint: event-based campaigns. You can now use Amazon Pinpoint to set up campaigns that send messages (such as text messages, push notifications, and emails) to your customers when they take specific actions. For example, you can set up a campaign to send a message when a customer creates a new account, or when they spend a certain dollar amount in your app, or when they add an item to their cart but don’t purchase it. Event-based campaigns help you send messages that are timely, personalized, and relevant to your customers, which ultimately increases their trust in your brand and gives them a reason to return.

You can create event-based campaigns by using the Amazon Pinpoint console, or by using the Amazon Pinpoint API. Event-based campaigns are an effective way to implement both transactional and targeted campaign use cases. For transactional workloads, imagine that you want to send an email to customers immediately after they choose the Password reset button in your app. You can create an event-based campaign that addresses this use case in as few as four clicks.

For targeted campaign workloads, event-based campaigns are a great way to introduce cross-sale opportunities for complementary items. For example, if a customer adds a phone to their shopping cart, you can send an in-app push notification that offers them a special price on a case that fits the device that they’re purchasing.

When you use the campaign wizard in the Amazon Pinpoint console, you now have the option to create event-based campaigns. Rather than define a time to send your message to customers, you select specific events, attributes, and metric values. As you go through the event scheduling component of the wizard, you define the event that you want Amazon Pinpoint to look for, as shown in the following image.

In order to use this feature, you have to set up your mobile and web apps to send event data to Amazon Pinpoint. To learn more about sending event data to Amazon Pinpoint, see Reporting Events in Your Application in the Amazon Pinpoint Developer Guide.

A real-world scenario

Let’s look at a scenario that explores how you can use event-based campaigns. Say you have a music streaming service such as Amazon Music. You’ve secured a deal for a popular artist to record a set of live tracks that will only be available on your service, and you want to let people who’ve previously listened to that artist know about this exciting exclusive.

To create an event-based campaign, you complete the usual steps involved in creating a campaign in Amazon Pinpoint (you can learn more about these steps in the Campaigns section of the Amazon Pinpoint User Guide). When you arrive at step 4 of the campaign creation wizard, it asks you when the campaign should be sent. At this point, you have two options: you can send the campaign at a specific time, or you can send it when an event occurs. In this example, we choose When an event occurs.

Next, you choose the event that causes the campaign to be sent. In our example, we’ll choose the event named song.played. If we stopped here and launched the campaign, Amazon Pinpoint would send our message to every customer who generated that event—in this case, every customer who played any song in our app, ever. That’s not what we want, but fortunately we can refine our criteria by choosing attributes and metrics.

In the Attributes box, we’ll choose the artistName attribute, and then choose the attribute value that corresponds with the name of the band we want to promote, The Alexandrians.

We’ve already narrowed down our list of recipients quite a bit, but we can go even further. In the Metrics box, we can specify certain quantitative values. For example, we could refine our event so that only customers who spent more than 10 minutes listening to songs by The Alexandrians are sent notifications. To complete this step, we choose the timePlayed metric and set it to look for endpoints for which this metric is greater than 600 seconds. When we finish setting up the triggering event, we see something similar to the following example:

Now that we’ve finished setting up the event that will result in our campaign to be sent, we can finish creating the campaign as we normally would.

Transparent and affordable pricing

There are no additional charges associated with creating event-based campaigns. You pay only for the number of endpoints that you target, the number of messages that you send, and the number of analytics events that you send to Amazon Pinpoint. To learn more about the costs associated with using Amazon Pinpoint, see our Pricing page.

As an example, assume that your app has 50,000 monthly active users, and that you target each user once a day with a push notification. In this scenario, you’d pay around $55.50 per month to send messages to all 50,000 users. Note that this pricing is subject to change, and is not necessarily reflective of what your actual costs may be.

Limitations and best practices

There are a few limitations and best practices that you should consider when you create event-based campaigns:

  • You can only create an event-based campaign if the campaign uses a dynamic segment (as opposed to an imported segment).
  • Event-based campaigns only consider events that are reported by recent versions of the AWS Mobile SDK. Your apps should use the following versions of the SDK in order to work with event-based campaigns:
    • AWS Mobile SDK for Android: version 2.7.2 or later
    • AWS Mobile SDK for iOS: version 2.6.30 or later
  • Because of this restriction, we recommend that you set up your segments so that they only include customers who use a version of your app that runs a compatible version of the SDK.
  • Choose your events carefully. For example, if you send an event-based campaign every time a session.start event occurs, you might quickly overwhelm your users with messages. You can limit the number of messages that Amazon Pinpoint sends to a single endpoint in a 24-hour period. For more information, see General Settings in the Amazon Pinpoint User Guide.

Ready to get started?

You can use these features today in every AWS Region where Amazon Pinpoint is available. We hope that these additions help you to better understand your customers and find exciting new ways to use Amazon Pinpoint to connect and engage with them.

 

Introducing the Amazon Pinpoint Voice Channel

Post Syndicated from Kadir Rathnavelu original https://aws.amazon.com/blogs/messaging-and-targeting/introducing-the-amazon-pinpoint-voice-channel/

Today, Amazon Pinpoint added a new voice channel. This new channel enables you to deliver voice messages to your customers over the phone. When you use the voice channel, you provide a text script, and, optionally, instructions for converting that text to speech. Amazon Pinpoint converts your script into lifelike speech and then calls your customers over the phone to deliver your message. Pinpoint Voice is a great way to deliver transactional messages such as one-time passwords, appointment reminders, order confirmations, and more.

Phone calls offer several advantages over other channels; they’re highly reliable and very extensible, and just about everybody has a phone number. The addition of voice messages to Pinpoint’s existing email, SMS, and push notification channels provides you with even more flexibility in delivering the right message on the right channel for your customers.

For example, to send a One-Time Password (OTP) to your customer over the phone, you first provide the Amazon Pinpoint SMS and Voice API with a set of instructions, including the content of the message and the destination phone number. The content can be a plain-text or Speech Synthesis Markup Language (SSML) messages.

You can use SSML to personalize your messages by adding pauses, changing the pronunciation, emphasizing key words, and much more. Amazon Pinpoint takes your message and uses Amazon Polly to convert it into lifelike speech in one of the dozens of languages and voices available. Amazon Pinpoint then and calls the destination number and delivers the message.

{
  "DestinationPhoneNumber":"+12065550199",
  "OriginationPhoneNumber":"+14155550142",
  "CallerId":"+12065550199",
  "Content":{
    "SSMLMessage":{
      "Text":"<speak> Your one-time password is 1 <break time='0.5s'/> 4 <break time='0.5s'/> 5 <break time='0.5s'/> 7 </speak>",
      "Voice":"Russell",
      "Language":"en-AU"
    }
  }
}

As the call progresses, Amazon Pinpoint provides call metrics in real time to Amazon Kinesis Data Firehose and Amazon CloudWatch Logs. These metrics tell you when a call has been initiated, when it’s ringing, when it’s been answered, when it’s completed, and whether or not the recipient answered. You can use this information to measure the effectiveness of your messages and to optimize your future voice engagements.

In addition to delivering one-to-one transactional messages, you can also use voice messages as a backup channel when you aren’t able to deliver messages through email, SMS, or push notifications. For example, it’s not unusual for customers to provide you with phone numbers that aren’t able to receive text messages, such as land-line or VoIP numbers. You can use the Phone Number Validate feature of Amazon Pinpoint to determine the type of phone number provided. When you detect that a phone number can’t receive text messages, you can send voice messages instead. By using Amazon Pinpoint Voice, you can reach a segment of customers who you wouldn’t be able to reach by sending text messages.

To send voice messages using Amazon Pinpoint, you need to have at least one dedicated phone number. You can lease local phone numbers in a variety of countries directly from the Amazon Pinpoint console. In certain countries and regions, you can also use these dedicated phone numbers to send SMS messages to your customers. See Supported Countries and Regions for more information about leasing phone numbers.

To learn more, see the Amazon Pinpoint User Guide and the Amazon Pinpoint SMS and Voice API Reference.

Amazon Pinpoint’s Analytics Suite Now Includes Event Metric Visualizations

Post Syndicated from Zach Barbitta original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-pinpoints-analytics-suite-now-includes-event-metric-visualizations/

Today, we’re enhancing Amazon Pinpoint’s analytics capabilities by giving you the ability to filter event data based on certain attributes and metrics. We’re also adding new analytics features help you visualize event data over time. These new features help you quickly understand how your customers are using your app, and make it easy to keep track of behavioral trends over time.

For example, if you want to understand how much time a customer spends performing a certain action within your app, such as watching a video, you can now visualize this in as few as three clicks. Start by signing in to the console and choosing the project that contains the data that you want to view. Next, in the navigation pane, under Analytics, choose Events. On the Events chart, you’ll see two new menus: Event Attributes and Metrics, and Event Attribute Values. Use these menus to refine the data in the chart to include only the events and attributes that you want to see.

When you select a metric value, Amazon Pinpoint shows a new time-series chart. This chart includes summary filters, so you can quickly see the total time an action occurs, the average, the minimum and maximum values over the time period, and the number of events. You can use this data when considering how to improve the user experience of your app, or when creating new engagement campaigns for certain groups of customers.

For example, you could see if the average time spent on your app is increasing or decreasing, or if the total price of goods sold is trending in the right direction. You can then create engagement campaigns that specifically target segments of users who spend less time in your app, or those who are making fewer in-app purchases.

As always, the data in Pinpoint is yours. You can easily export it to CSV format for analysis in external applications. For deeper analysis, we recommend that you set up Amazon Pinpoint Event Streams, which let you stream your data to nearly any destination. To learn more about setting up Event Streams, see Streaming Events from Amazon Pinpoint to Redshift on the AWS Messaging and Targeting Blog.

You can use these features now in all AWS Regions where Amazon Pinpoint is available. We hope that these additions help you to better understand your customers and find exciting new ways to use Amazon Pinpoint to connect and engage with them.

Well-Architected: “To thrive, to evolve, to delight”

Post Syndicated from Philip Fitzsimons original https://aws.amazon.com/blogs/architecture/well-architected-to-thrive-to-evolve-to-delight/

Today we published a subtle but significant update to the AWS Well-Architected Framework. We looked to see where AWS Solutions Architects and customers diverged in how they judged something to be Well-Architected, and we restructured our questions and answers to help close that gap.

There are many definitions of IT architecture, but, at its core, a good architecture is like a well-designed building: It creates a space that we love to be in. In technology, good architecture creates a space that allows our code to thrive, to evolve, to delight. Bad architecture inhibits the ability of our code to meet our expectations, and it exposes us to risk, wasted effort, extra costs, and bad outcomes. How do you know if your architectures are good?

In 2012, we created the AWS Well-Architected Framework as a way to answer that question. AWS Solutions Architects review thousands of workloads every year using the framework. We learn from these reviews: new ideas, bad ideas, and constant innovation. We curate those learnings, from our customers and our teams, to create a framework that remains current but also tested and pragmatic.

We use Kaizen 改善to help us continually improve the framework. We bring data—your anecdotes and challenges—and use it to understand where we can improve the framework. We experiment, ask five whys, draw pictures of fish, and whiteboard, whiteboard, whiteboard. We draft guidelines for writing best practices, we shape our best practices to fit, we iterate. We have it reviewed by our principal community, taken for a spin by AWS Solutions Architects, and then we publish an update to the framework. We joyfully spin round a Deming cycle.

With this update every question has been refined, and some have been split into two to ensure they focus on a single topic. The framework has also been updated to reflect new services and features, and is available as a whitepaper PDF and free as Kindle books. We believe that if you follow the well-architected way, your architectures will create a space where your code and functionality will delight your customers. You can find free training and all of the whitepapers on the AWS Well-Architected homepage.

Philip Fitzsimons is the leader of the AWS Well-Architected Team

A simpler way to assess the network exposure of EC2 instances: AWS releases new network reachability assessments in Amazon Inspector

Post Syndicated from Catherine Dodge original https://aws.amazon.com/blogs/security/amazon-inspector-assess-network-exposure-ec2-instances-aws-network-reachability-assessments/

Performing network security assessments allows you to understand your cloud infrastructure and identify risks, but this process traditionally takes a lot of time and effort. You might need to run network port-scanning tools to test routing and firewall configurations, then validate what processes are listening on your instance network ports, before finally mapping the IPs identified in the port scan back to the host’s owner. To make this process simpler for our customers, AWS recently released the Network Reachability rules package in Amazon Inspector, our automated security assessment service that enables you to understand and improve the security and compliance of applications deployed on AWS. The existing Amazon Inspector host assessment rules packages check the software and configurations on your Amazon Elastic Compute Cloud (Amazon EC2) instances for vulnerabilities and deviations from best practices.

The new Network Reachability rules package analyzes your Amazon Virtual Private Cloud (Amazon VPC) network configuration to determine whether your EC2 instances can be reached from external networks such as the Internet, a virtual private gateway, AWS Direct Connect, or from a peered VPC. In other words, it informs you of potential external access to your hosts. It does this by analyzing all of your network configurations—like security groups, network access control lists (ACLs), route tables, and internet gateways (IGWs)—together to infer reachability. No packets need to be sent across the VPC network, nor must attempts be made to connect to EC2 instance network ports—it’s like packet-less network mapping and reconnaissance!

This new rules package is the first Amazon Inspector rules package that doesn’t require an Amazon Inspector agent to be installed on your Amazon EC2 instances. If you do optionally install the Amazon Inspector agent on your EC2 instances, the network reachability assessment will also report on the processes listening on those ports. In addition, installing the agent allows you to use Amazon Inspector host rules packages to check for vulnerabilities and security exposures in your EC2 instances.

To determine what is reachable, the Network Reachability rules package uses the latest technology from the AWS Provable Security initiative, referring to a suite of AWS technology powered by automated reasoning. It analyzes your AWS network configurations such as Amazon Virtual Private Clouds (VPCs), security groups, network access control lists (ACLs), and route tables to prove reachability of ports. What is automated reasoning, you ask? It’s fancy math that proves things are working as expected. In more technical terms, it’s a method of formal verification that automatically generates and checks mathematical proofs, which help to prove systems are functioning correctly. Note that Network Reachability only analyzes network configurations, so any other network controls, like on-instance IP filters or external firewalls, are not accounted for in the assessment. See documentation for more details.

Tim Kropp, Technology & Security Lead at Bridgewater Associates talked about how Bridgewater benefitted from Network Reachability Rules. “AWS provides tools for organizations to know if all their compliance, security, and availability requirements are being met. Technology created by the AWS Automated Reasoning Group, such as the Network Reachability Rules, allow us to continuously evaluate our live networks against these requirements. This grants us peace of mind that our most sensitive workloads exist on a network that we deeply understand.”

Network reachability assessments are priced per instance per assessment (instance-assessment). The free trial offers the first 250 instance-assessments for free within your first 90 days of usage. After the free trial, pricing is tiered based on your monthly volume. You can see pricing details here.

Using the Network Reachability rules package

Amazon Inspector makes it easy for you to run agentless network reachability assessments on all of your EC2 instances. You can do this with just a couple of clicks on the Welcome page of the Amazon Inspector console. First, use the check box to Enable network assessments, then select Run Once to run a single assessment or Run Weekly to run a weekly recurring assessment.
 

Figure 1: Assessment setup

Figure 1: Assessment setup

Customizing the Network Reachability rules package

If you want to target a subset of your instances or modify the recurrence of assessments, you can select Advanced setup for guided steps to set up and run a custom assessment. For full customization options including getting notifications for findings, select Cancel and use the following steps.

  1. Navigate to the Assessment targets page of the Amazon Inspector console to create an assessment target. You can select the option to include all instances within your account and AWS region, or you can assess a subset of your instances by adding tags to them in the EC2 console and inputting those tags when you create the assessment target. Give your target a name and select Save.
     
    Figure 2: Assessment target

    Figure 2: Assessment target

    Optional agent installation: To get information about the processes listening on reachable ports, you’ll need to install the Amazon Inspector agent on your EC2 instances. If your instances allow the Systems Manager Run command, you can select the Install Agents option while creating your assessment target. Otherwise, you can follow the instructions here to install the Amazon Inspector agent on your instances before setting up and running the Amazon Inspector assessments using the steps above. In addition, installing the agent allows you to use Amazon Inspector host rules packages to check for vulnerabilities and security exposures in your EC2 instances.

  2. Go to the Assessment templates page of the Amazon Inspector console. In the Target name field, select the assessment target that you created in step 1. From the Rules packages drop-down, select the Network Reachability-1.1 rules package. You can also set up a recurring schedule and notifications to your Amazon Simple Notification Service topic. (Learn more about Amazon SNS topics here). Now, select Create and Run. That’s it!

    Alternately, you can run the assessment by selecting the template you just created from the Assessment templates page and then selecting Run, or by using the Amazon Inspector API.

You can view your findings on the Findings page in the Amazon Inspector console. You can also download a CSV of the findings from Amazon Inspector by using the Download button on the page, or you can use the AWS application programming interface (API) to retrieve findings in another application.

Note: You can create any CloudWatch Events rule and add your Amazon Inspector assessment as the target using the assessment template’s Amazon Resource Name (ARN), which is available in the console. You can use CloudWatch Events rules to automatically trigger assessment runs on a schedule or based on any other event. For example, you can trigger a network reachability assessment whenever there is a change to a security group or another VPC configuration, allowing you to automatically be alerted about insecure network exposure.

Understanding your EC2 instance network exposure

You can use this new rules package to analyze the accessibility of critical ports, as well as all other network ports. For critical ports, Amazon Inspector will show the exposure of each and will offer findings per port. When critical, well-known ports (based on Amazon’s standard guidance) are reachable, findings will be created with higher severities. When the Amazon Inspector agent is installed on the instance, each reachable port with a listener will also be reported. The following examples show network exposure from the Internet. There are analogous findings for network exposure via VPN, Direct Connect, or VPC peering. Read more about the finding types here.

Example finding for a well-known port open to the Internet, without installation of the Amazon Inspector Agent:
 

Figure 3: Finding for a well-known port open to the Internet

Figure 3: Finding for a well-known port open to the Internet

Example finding for a well-known port open to the Internet, with the Amazon Inspector Agent installed and a listening process (SSH):
 

Figure 4: Finding for a well-known port open to the Internet, with the Amazon Inspector Agent installed and a listening process (SSH)

Figure 4: Finding for a well-known port open to the Internet, with the Amazon Inspector Agent installed and a listening process (SSH)

Note that the findings provide the details on exactly how network access is allowed, including which VPC and subnet the instance is in. This makes tracking down the root cause of the network access straightforward. The recommendation includes information about exactly which Security Group you can edit to remove the access. And like all Amazon Inspector findings, these can be published to an SNS topic for additional processing, whether that’s to a ticketing system or to a custom AWS Lambda function. (See our blog post on automatic remediation of findings for guidance on how to do this.) For example, you could use Lambda to automatically remove ingress rules in the Security Group to address a network reachability finding.

Summary

With this new functionality from Amazon Inspector, you now have an easy way of assessing the network exposure of your EC2 instances and identifying and resolving unwanted exposure. We’ll continue to tailor findings to align with customer feedback. We encourage you to try out the Network Reachability Rules Package yourself and post any questions in the Amazon Inspector forum.

Want more AWS Security news? Follow us on Twitter.

Author

Catherine Dodge

Catherine is a Senior Technical Program Manager in AWS Security. She helps teams use cutting edge AI technology to build security products to delight customers. She has over 15 years of experience in the cybersecurity field, mostly spent at the assembly level, either pulling apart malware or piecing together shellcode. In her spare time, she’s always tearing something down around the house, preferably ivy or drywall.

Author

Stephen Quigg

Stephen — known as “Squigg,” internally — is a Principal Security Solutions Architect at AWS. His job is helping customers understand AWS security and how they can meet their most demanding security requirements when using the AWS platform. It’s not all about solving hard problems though, he gets just as much delight when an AWS customer creates their first VPC! When he’s not with his customers, you can find him up in his loft making bleeping noises on a bunch of old synthesizers.

Exciting New Email Features Available Now in Amazon Pinpoint

Post Syndicated from Brent Meyer original https://aws.amazon.com/blogs/messaging-and-targeting/exciting-new-email-features-available-now-in-amazon-pinpoint/

Yesterday, we launched several exciting new email-related features in Amazon Pinpoint. The biggest of these features is the ability to send transactional emails. Transactional emails are a great way to send one-to-one emails to your customers without creating segments or campaigns first. For example, when a customer makes a purchase in your app, or requests a password reset, you can send them an email immediately and directly with the information that they need. You can send transactional emails by using the Amazon Pinpoint API or an AWS SDK, or by using the Amazon Pinpoint SMTP interface.

Along with the ability to send transactional emails, we also added a dashboard that gives you deep insights into the performance of the emails you’ve sent. You can use this dashboard to view several response metrics related to the transactional emails you’ve sent, including your delivery, bounce, and complaint rates, the number of emails that were opened and clicked, and more. You can filter this dashboard to include data from 1 to 30 days.

We’ve also added the ability to lease dedicated IP addresses through Amazon Pinpoint. Dedicated IP addresses are IP addresses that are reserved for your exclusive use. Dedicated IP addresses give you complete control over the reputations of the IP addresses that you use to send email through Amazon Pinpoint. When you use dedicated IP addresses, you can create pools of your dedicated IP addresses. Dedicated IP pools are groups of IP addresses you use for different purposes. For example, you can create a pool of addresses to use when sending transactional messages, and another pool for sending marketing communications.

We also added the following email-related capabilities:

  • Content personalization: Generic, one-size-fits-all emails tend to have lower engagement rates than personalized messages. When you send a transactional email, you can include personalization tags, such as the recipient’s name or location. When Amazon Pinpoint sends the mail, it replaces the personalization tags with the appropriate values for the recipient.
  • Event destinations: You can use event destinations to send information about email events to various destinations, including Amazon CloudWatch, Amazon SNS, and Amazon Kinesis Data Firehose. Email events include email opens, clicks, bounces, complaints, and rejections.
  • Raw email support: You can use the Amazon Pinpoint API or SMTP interface to send raw, MIME-formatted emails. MIME emails can include custom headers and file attachments.

These features are available now in all AWS Regions where Amazon Pinpoint is available. We hope that these features help you find exciting new ways to use Amazon Pinpoint to connect and engage with your customers.

How to Test and Debug AWS CodeDeploy Locally Before You Ship Your Code

Post Syndicated from Kirankumar Chandrashekar original https://aws.amazon.com/blogs/devops/how-to-test-and-debug-aws-codedeploy-locally-before-you-ship-your-code/

AWS CodeDeploy is a powerful service for automating deployments to Amazon EC2, AWS Lambda, and on-premises servers. However, it can take some effort to get complex deployments up and running or to identify the error in your application when something goes wrong.

When I set up new deployments or debug existing ones, I like to test and debug locally for these reasons:

  • To speed up the iteration process.
  • To isolate potential issues.
  • To validate code.

You can test application code packages on any machine that has the CodeDeploy agent installed before you deploy it through the service. Likewise, to debug locally, you just need to install the CodeDeploy agent on any machine, including your local server or EC2 instance.

In this blog post, I will walk you through the steps to validate and debug a sample application package using the codedeploy-local command. You can find the sample package in this GitHub repository.

 

 

Prerequisites

Install the CodeDeploy agent on any supported instance type. For information, see Use the AWS CodeDeploy Agent to Validate a Deployment Package on a Local Machine in the AWS CodeDeploy User Guide.

Step 1

Verify the CodeDeploy agent is installed and ready for local testing. By default, codedeploy-local is installed in the following locations:

On Amazon Linux, RHEL, or Ubuntu Server:

/opt/codedeploy-agent/bin/codedeploy-local

On Windows Server:

C:\ProgramData\Amazon\CodeDeploy\bin

For simplicity, I am creating an alias for /opt/codedeploy-agent/bin/codedeploy-local as codedeploy-local so I can use the absolute path. This is optional.

alias codedeploy-local='sudo /opt/codedeploy-agent/bin/codedeploy-local'

When I execute the codedeploy-local command on the Linux terminal, I get the following response from the agent, which indicates that the agent is installed:

[[email protected] ~]$ codedeploy-local 
ERROR: Expecting appspec file at location /home/ec2-user/appspec.yml but it is not found there. Please either run the CLI from within a directory containing the appspec.yml file or specify a bundle location containing an appspec.yml file in its root directory

If you receive an error that the codedeploy-local command is not available or the package was not found, go back to the prerequisites and install the agent.

Step 2
To test the sample application package using the codedeploy-local command, I have to make sure that the application package is available on the local machine. The sample package I am testing here is an Apache (httpd)-based application.

Use wget to download the package to the local machine.

wget https://s3.amazonaws.com/aws-codedeploy-us-east-1/samples/latest/SampleApp_Linux.zip

Now that the sample package is available locally, I can either unzip the package or use the zip file for testing with the codedeploy-local command.

To test the zip file (archive) package (SampleApp_Linux.zip) with the codedeploy-local command, use the -l or –bundle-location option along with the -t or –type option as shown:

On Linux server:

codedeploy-local --bundle-location /home/ec2-user/CodeDeployPackage/SampleApp_Linux.zip -t zip --deployment-group my-deployment-group

On Windows server:

codedeploy-local --bundle-location C:/path/to/local/bundle.zip --type zip --deployment-group my-deployment-group

To unarchive the zip file, either change the directory (cd) to the top-level directory or provide the absolute path to the application package.

The package can be executed by providing the absolute path to the content as shown here:

codedeploy-local --bundle-location /path/to/local/bundle/directory

Or by changing the directory (cd) to the location of the unarchived package and executing the following command:

codedeploy-local

Executing the codedeploy-local command in the directory where the sample package is unzipped shows whether the deployment was successful or failed.

Here is a successful deployment execution and result:

[email protected] CodeDeployPackage]$ ls -a
.  ..  appspec.yml  index.html  LICENSE.txt  SampleApp_Linux.zip  scripts

[email protected] CodeDeployPackage]$ codedeploy-local
Starting to execute deployment from within folder /opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local
See the deployment log at /opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local/logs/scripts.log for more details
AppSpec file valid. Local deployment successful

Step 3

Check the codedeploy-local logs and the deployment archive.

In the previous step, I was able to see that the local deployment was successful. The output included:

  • The log location.
  • The location where the deployment-archive was uploaded. It will be used as a staging directory for that deployment.

Because the –deployment-group, -g option was not provided, a local deployment group folder was created in the following location:

/opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local

The following shows the listing of the files in the codedeploy-local deployment directory for a deployment:

[email protected] ~]$ ls /opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local
deployment-archive  logs

[[email protected] deployment-archive]$ ls -a /opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local/deployment-archive/
.  ..  appspec.yml  index.html  LICENSE.txt  SampleApp_Linux.zip  scripts

[[email protected] deployment-archive]$ ls -a /opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local/logs
.  ..  scripts.log

In the directory path generated for each deployment, default-local-deployment-group  is the name of the deployment group and d-H3OZK261S-local is the deployment ID.

The scripts.log shows the execution logs for the codedeploy-local command for a deployment group and deployment ID. Here is an example of a scripts.log that shows the execution of each lifecycle event defined in the appspec.yml:

[[email protected] deployment-archive]$ cat /opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-H3OZK261S-local/logs/scripts.log
2018-03-13 23:02:37 LifecycleEvent - ApplicationStop
2018-03-13 23:02:37 Script - scripts/stop_server
2018-03-13 23:02:37 [stdout]Stopping httpd: [  OK  ]
2018-03-13 23:02:37 LifecycleEvent - BeforeInstall
2018-03-13 23:02:37 Script - scripts/install_dependencies
2018-03-13 23:02:37 [stdout]Loaded plugins: priorities, update-motd, upgrade-helper
2018-03-13 23:02:37 [stdout]Package httpd-2.2.34-1.16.amzn1.x86_64 already installed and latest version
2018-03-13 23:02:37 [stdout]Nothing to do
2018-03-13 23:02:37 Script - scripts/start_server
2018-03-13 23:02:37 [stdout]Starting httpd: [  OK  ]

There is another log file in this location that comes in handy when deploying the code on the local machine:

/var/log/aws/codedeploy-agent/codedeploy-local.log

You can enable verbose logging in the codedeploy-agent configuration file by setting the parameter :verbose: to true.

By default, the location of the configuration file is:

Amazon Linux, RHEL, or Ubuntu Server instances

/etc/codedeploy-agent/conf/codedeployagent.yml

Windows Server

C:/ProgramData/Amazon/CodeDeploy/conf.yml

Other features for debugging issues locally with codedeploy-local

The codedeploy-local command has other features that you can use to debug and troubleshoot issues.

Override the lifecycle hooks mentioned in the appspec.yml file

You can use codedeploy-local to override the lifecycle hooks provided in the appspec.yml. In this example, only the ApplicationStop lifecycle hook defined in the appspec.yml file will be executed. All other hooks will be ignored.

codedeploy-local -e ApplicationStop

In the same way, you can override the order in which the CodeDeploy agent executes multiple lifecycle hooks. This feature can help you determine and change the sequence before the deployment is performed on the server. For information, see AppSpec ‘hooks’ Section in the AWS CodeDeploy User Guide.

For example, this command executes the BeforeInstall lifecycle hook first and then executes the ApplicationStop lifecycle hook.

codedeploy-local -e BeforeInstall,ApplicationStop

Execute scripts specifically for codedeploy-local

If there are scripts that are used for local testing only and not required for the CodeDeploy deployment, then you can use the $DEPLOYMENT_GROUP_NAME variable, which has a value equal to LocalFleet.

Here are other environment variables and their values:

$APPLICATION_NAME: The location of the deployment package (for example, /home/ec2-user/CodeDeployPackage)

$DEPLOYMENT_ID: Unique per deployment (for example, d-LTVP5L6YY-local)

$DEPLOYMENT_GROUP_ID: The name of the deployment group. When the -g option is used for the command, this value will be passed. For example, in codedeploy-local -g testing, this value is testing. If this option is not set, the value of this environment variable is default-local-deployment-group

$LIFECYCLE_EVENT: The lifecycle hook that echoed this environment variable (for example, ApplicationStop)

Override the CodeDeploy agent configuration

You can override the CodeDeploy agent configuration and use your own configuration file from a custom location. This functionality makes it possible to test multiple configurations with the local deployments using the option -c, –agent-configuration-file while executing the codedeploy-local command. For the options to use, see AWS CodeDeploy Agent Configuration Reference in the AWS CodeDeploy User Guide.

By default, configuration files are stored in the following locations:

On Amazon Linux, RHEL, or Ubuntu Server:

/etc/codedeploy-agent/conf/codedeployagent.yml

On Windows Server:

C:/ProgramData/Amazon/CodeDeploy/conf.yml

Using custom configuration helps when verbose logging is required for package testing. You can do this just by using the -c or –agent-configuration-file option and without changing the default configuration file. Here is an example that shows the use of this option:

codedeploy-local -e BeforeInstall,ApplicationStop -c /<;-local-path->;/

For example, on Amazon Linux, RHEL, or Ubuntu Server instances, when the config file is in /etc/codedeployagent.yml, the command is:

codedeploy-local -e BeforeInstall,ApplicationStop -c /etc/codedeployagent.yml

For example, on Windows Server instances, when the config file is in C:/ProgramData/conf.yml, the command is:

codedeploy-local -e BeforeInstall,ApplicationStop -c C:/ProgramData/conf.yml

Point to an application package in an S3 bucket or GitHub repository

If the application package is stored in an S3 bucket or GitHub repository, codedeploy-local can be executed without downloading the file onto the local machine. You can do this using the -l, –bundle-location and -t, –type with the codedeploy-local command.

Here is an example for deploying a sample application package located in an S3 bucket:

codedeploy-local -l s3://aws-codedeploy-us-east-1/samples/latest/SampleApp_Linux.zip -t zip

Here is an example for deploying a sample application package from a public GitHub repository:

codedeploy-local --bundle-location https://api.github.com/repos/awslabs/aws-codedeploy-sample-tomcat/zipball/master --type zip

If you use GitHub, make sure that the application package with the appspec.yaml is in the root of the directory. If these contents are in a subfolder path, download the package to the local instance or server and then:

  • Execute codedeploy-local from the directory where the file exists.

-OR-

  • Use the -t, –type  option with the value of directory and -l, –bundle-location as the local path.

Troubleshooting common errors using codedeploy-local

The codedeploy-local command can be used to detect if the appspec.yml is in valid YAML format. If the format is invalid, you get the following error:

/usr/share/ruby/vendor_ruby/2.0/psych.rb:205:in `parse': (<unknown>): mapping values are not allowed in this context at line 10 column 13 (Psych::SyntaxError)

If there is an invalid lifecycle hook in the appspec.yml file, the deployment fails with this error:

ERROR: appspec.yml file contains unknown lifecycle events: ["BeforeInstall1"]

The name of a lifecycle hook is case-sensitive. The following error is returned because the BeforeInstall lifecycle hook was entered as Beforeinstall:

ERROR: appspec.yml file contains unknown lifecycle events: ["Beforeinstall"]

If there is any error in the scripts provided for execution in any lifecycle hooks (for example, a problem in the BeforeInstall script), the execution logs show something like this:

codedeploy-local -g testing
Starting to execute deployment from within folder /opt/codedeploy-agent/deployment-root/testing/d-6UBAIVVSK-local
Your local deployment failed while trying to execute your script at /opt/codedeploy-agent/deployment-root/testing/d-6UBAIVVSK-local/deployment-archive/scripts/install_dependencies
See the deployment log at /opt/codedeploy-agent/deployment-root/testing/d-6UBAIVVSK-local/logs/scripts.log for more details

For the preceding error, when you look at the logs in the deployment directory for the deployment group, you will see something like this:

cat /opt/codedeploy-agent/deployment-root/testing/d-6UBAIVVSK-local/logs/scripts.log
2018-03-21 03:34:04 LifecycleEvent - ApplicationStop
2018-03-21 03:34:04 Script - scripts/stop_server
2018-03-21 03:34:04 [stdout]LocalFleet
2018-03-21 03:34:04 [stdout]/home/ec2-user/CodeDeployPackage
2018-03-21 03:34:04 [stdout]d-6UBAIVVSK-local
2018-03-21 03:34:04 [stdout]testing
2018-03-21 03:34:04 [stdout]ApplicationStop
2018-03-21 03:34:04 [stdout]Stopping httpd: [  OK  ]
2018-03-21 03:34:04 LifecycleEvent - BeforeInstall
2018-03-21 03:34:04 Script - scripts/install_dependencies
2018-03-21 03:34:04 [stdout]Loaded plugins: priorities, update-motd, upgrade-helper
2018-03-21 03:34:04 [stdout]No package httpd1 available.
2018-03-21 03:34:04 [stderr]Error: Nothing to do

This log snippet shows that the install_dependencies script had a package called httpd1 that is not available for installation.

If the appspec.yml is not found in the root of the application package, you will see an error like this:

/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:213:in `parse_app_spec': The CodeDeploy agent did not find an AppSpec file within the unpacked revision directory at revision-relative path "appspec.yml". The revision was unpacked to directory "/opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-BE59ORH9I-local/deployment-archive", and the AppSpec file was expected but not found at path "/opt/codedeploy-agent/deployment-root/default-local-deployment-group/d-BE59ORH9I-local/deployment-archive/appspec.yml". Consult the AWS CodeDeploy Appspec documentation for more information at http://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file.html (RuntimeError)
    from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:100:in `initialize'
    from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:147:in `new'
    from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:147:in `block (3 levels) in map'
    from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:146:in `each'
    from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:146:in `block (2 levels) in map'
    from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:68:in `execute_command'
    from /opt/codedeploy-agent/lib/aws/codedeploy/local/deployer.rb:85:in `block in execute_events'
    from /opt/codedeploy-agent/lib/aws/codedeploy/local/deployer.rb:84:in `each'
    from /opt/codedeploy-agent/lib/aws/codedeploy/local/deployer.rb:84:in `execute_events'
    from /opt/codedeploy-agent/bin/codedeploy-local:117:in `<main>'

Conclusion

The codedeploy-local command can be used to validate and debug an application package for deployments to Amazon EC2 instances or on-premises servers. With codedeploy-local, you can test and fix errors on a local machine during the code development phase. CodeDeploy local deployments also make it possible for you to change the order of the lifecycle hooks so you can restructure the appspec.yaml to add commands on the fly.

Store, Protect, Optimize Your Healthcare Data with AWS: Part 2

Post Syndicated from Stephen Jepsen original https://aws.amazon.com/blogs/architecture/store-protect-optimize-your-healthcare-data-with-aws-part-2/

Leveraging Analytics and Machine Learning Tools for Readmissions Prediction

This blog post was co-authored by Ujjwal Ratan, a senior AI/ML solutions architect on the global life sciences team.

In Part 1, we looked at various options to ingest and store sensitive healthcare data using AWS. The post described our shared responsibility model and provided a reference architecture that healthcare organizations could use as a foundation to build a robust platform on AWS to store and protect their sensitive data, including protected health information (PHI). In Part 2, we will dive deeper into how customers can optimize their healthcare datasets for analytics and machine learning (ML) to address clinical and operational challenges.

There are a number of factors creating pressures for healthcare organizations, both providers and payers, to adopt analytic tools to better understand their data: regulatory requirements, changing reimbursement models from volume- to value-based care, population health management for risk-bearing organizations, and movement toward personalized medicine. As organizations deploy new solutions to address these areas, the availability of large and complex datasets from electronic health records, genomics, images (for example, CAT, PET, MRI, ultrasound, X-ray), and IoT has been increasing. With these data assets growing in size, healthcare organizations want to leverage analytic and ML tools to derive new actionable insights across their departments.

One example of the use of ML in healthcare is diagnostic image analysis, including digital pathology. Pathology is extremely important in diagnosing and treating patients, but it is also extremely time-consuming and largely a manual process. While the complexity and quantity of workloads are increasing, the number of pathologists is decreasing. According to one study, the number of active pathologists could drop by 30 percent by 2030 compared to 2010 levels. (1) A cloud architecture and solution can automate part of the workflow, including sample management, analysis, storing, sharing, and comparison with previous samples to complement existing provider workflows effectively. A recent study using deep learning to analyze metastatic breast cancer tissue samples resulted in an approximately 85% reduction in human error rate. (2)

ML is also being used to assist radiologists in examining other diagnostic images such as X-rays, MRIs, and CAT scans. Having large quantities of images and metadata to train the algorithms that are the key to ML is one of the main challenges for ML adoption. To help address this problem, the National Institutes of Health recently released 90,000 X-ray plates tagged either with one of 14 diseases or tagged as being normal. Leading academic medical centers are using these images to build their neural networks and train their algorithms. With advanced analytics and ML, we can answer the hard questions such as “what is the next best action for my patient, the expected outcome, and the cost.”

The foundations for a great analytical layer

Let’s pick up from where we left off in Part 1. We have seen how providers can ingest data into AWS from their data centers and store it securely into different services depending on the type of data. For example:

  1. All object data is stored in Amazon S3, Amazon S3 Infrequent Access, or Amazon Glacier depending on how often they are used.
  2. Data from the provider’s database is either processed and stored as objects in Amazon S3 or aggregated into data marts on Amazon Redshift.
  3. Metadata of the objects on Amazon S3 are maintained in the DynamoDB database.
  4. Amazon Athena is used to query the objects directly stored on Amazon S3 to address ad hoc requirements.

We will now look at two best practices that are key to building a robust analytical layer using these datasets.

  1. Separating storage and compute: You should not be compelled to scale compute resources just to store more data. The scaling rules of the two layers should be separate.
  2. Leverage the vast array of AWS big data services when it comes to building the analytical platforms instead of concentrating on just a few of them. Remember, one size does not fit all.

Technical overview

In this overview, we will demonstrate how we can leverage AWS big data and ML services to build a scalable analytical layer for our healthcare data. We will use a single source of data stored in Amazon S3 for performing ad hoc analysis using Amazon Athena, integrate it with a data warehouse on Amazon Redshift, build a visual dashboard for some metrics using Amazon QuickSight, and finally build a ML model to predict readmissions using Amazon SageMaker. By not moving the data around and just connecting to it using different services, we avoid building redundant copies of the same data. There are multiple advantages to this approach:

  1. We optimize our storage. Not having redundant copies reduces the amount of storage required.
  2. We keep the data secure with only authorized services having access to it. Keeping multiple copies of the data can result in higher security risk.
  3. We are able to scale the storage and compute separately as needed.
  4. It becomes easier to manage the data and monitor usage metrics centrally such as how often the data has been accessed, who has been accessing it, and what has been the growth pattern of the data over a period of time. These metrics can be difficult to aggregate if the data is duplicated multiple times.

Let’s build out this architecture using the following steps:

  1. Create a database in AWS Glue Data Catalog

We will do this using a Glue crawler. First create a JSON file that contains the parameters for the Glue crawler.

{
"Name": "readmissions",
"Role": "arn of the role for Glue",
"DatabaseName": "readmissions",
"Description": "glue data catalog for storing readmission data",
"Targets": {
"S3Targets": [
{
"Path": "s3://<bucket>/<prefix>"
},
{
"Path": "s3://<bucket>/<prefix>"
}
]
}
}

As you can see, the crawler will crawl two locations in Amazon S3 and save the resulting tables in a new database called “readmissions.” Replace the role ARN and Amazon S3 locations with your corresponding details. Save this in a file create_crawler.json. Then from the AWS CLI, call the following command to create the crawler:

aws glue create-crawler --cli-input-json file://create_crawler.json

Once the crawler is created, run it by calling the following command:

aws glue start-crawler --name readmissions

Log on to the AWS Glue console, navigate to the crawlers, and wait until the crawler completes running.

This will create two tables — phi and non-phi — in a database named “readmissions” in the AWS Glue Data Catalog as shown below.

  1. Query the data using Athena

The Amazon Glue Data Catalog is seamlessly integrated with Amazon Athena. For details on how to enable this, see Integration with AWS Glue.

As a result of this integration, the tables created using the Glue crawler can now be queried using Amazon Athena. Amazon Athena allows you to do ad hoc analysis on the dataset. You can do exploratory analysis on the data and also determine its structure and quality. This type of upfront ad hoc analysis is invaluable for ensuring the data quality in your downstream data warehouse or your ML algorithms that will make use of this data for training models. In the next few sections, we will explore these aspects in greater detail.

To query the data using Amazon Athena, navigate to the Amazon Athena console.

NOTE: Make sure the region is the same as the region you chose in the previous step. If it’s not the same, switch the region by using the drop-down menu on the top right-hand corner of the screen.

Once you arrive in the Amazon Athena console, you should already see the tables and databases you created previously, and you should be able to see the data in the two tables by writing Amazon Athena queries. Here is a list of the top 10 rows from the table readmissions.nonphi:

Now that we are able to query the dataset, we can run some queries for exploratory analysis. Here are just a few examples:

Analysis Amazon Athena Query
How many Patients have been discharged to home? SELECT count(*) from nonphi where discharge_disposition = ‘Discharged to home’
What’s the minimum and the maximum number of procedures carried out on a patient? SELECT min(num_procedures), max(num_procedures) from nonphi
How many patients were referred to this hospital by another physician? SELECT count(*) FROM nonphi group by admission_source having admission_source = ‘Physician Referral’
What were the top 5 specialties with positive readmissions?

SELECT count(readmission_result) as num_readmissions, medical_specialty from

(select readmission_result,medical_specialty from nonphi where readmission_result = ‘Yes’)

group by medical_specialty order by num_readmissions desc limit 5

Which payer was responsible for paying for treatments that involved more than 5 procedures? SELECT distinct payer_code from nonphi where num_procedures >5 and payer_code !='(null)’

While this information is valuable, you typically do not want to invest too much time and effort into building an ad hoc query platform like this because at this stage, you are not even sure if the data is of any value for your business-critical analytical applications. One benefit of using Amazon Athena for ad hoc analysis is that it requires little effort or time. It uses Schema-On-Read instead of schema on write, allowing you to work with various source data formats without worrying about the underlying structures. You can put the data on Amazon S3 and start querying immediately.

  1. Create an external table in Amazon Redshift Spectrum with the same data

Now that we are satisfied with the data quality and understand the structure of the data, we would like to integrate this with a data warehouse. We’ll use Amazon Redshift Spectrum to create external tables on the files in S3 and then integrate these external tables with a physical table in Amazon Redshift.

Amazon Redshift Spectrum allows you to run Amazon Redshift SQL queries against data on Amazon S3, extending the capabilities of your data warehouse beyond the physical Amazon Redshift clusters. You don’t need to do any elaborate ETL or move the data around. The data exists in one place in Amazon S3 and you interface with it using different services (Athena and Redshift Spectrum) to satisfy different requirements.

Before beginning, please look at this step by step guide to set up Redshift Spectrum.

After you have set up Amazon Redshift Spectrum, you can begin executing the steps below:

  1. Create an external schema called “readmissions.” Amazon Redshift Spectrum integrates with the Amazon Glue Data Catalog and allows you to create spectrum tables by referring the catalog. This feature allows you to build the external table on the same data that you analyzed with Amazon Athena in the previous step without the need for ETL. This can be achieved by the following:
create external schema readmissions
from data catalog
database 'readmissions'
iam_role 'arn for your redshift spectrum role '
region ‘region when the S3 data exists’;

NOTE: Make sure you select the appropriate role arn and region.

  1. Once the command executes successfully, you can confirm the schema was created by running the following:
select * from svv_external_schemas;

You should see a row similar to the one above with your corresponding region and role.

You can also see the external tables that were created by running the following command:

select * from SVV_EXTERNAL_TABLES;

  1. Let’s confirm we can see all the rows in the external table by counting the number of rows:
select count(*) from readmissions.phi;
select count(*) from readmissions.nonphi;

You should see 101,766 rows in both the tables, confirming that your external tables contain all the records that you read using the AWS Glue data crawler and analyzed using Athena.

  1. Now that we have all the external tables created, let’s create an aggregate fact table in the physical Redshift data warehouse. We can use the “As Select” clause of the Redshift create table query to do this:
create table readmissions_aggregate_fact as
select
readmission_result,admission_type,discharge_disposition,diabetesmed,
avg(time_in_hospital) as avg_time_in_hospital,
min(num_procedures) as min_procedures,
max(num_procedures) as max_procedures,
avg(num_procedures) as avg_num_procedures,
avg(num_medications) as avg_num_medications,
avg(number_outpatient) as avg_number_outpatient,
avg(number_emergency) as avg_number_emergency,
avg(number_inpatient) as avg_number_inpatient,
avg(number_diagnoses) as avg_number_diagnoses
from readmissions.nonphi
group by readmission_result,admission_type,discharge_disposition,diabetesmed

Once this query executes successfully, you can see a new table created in the physical public schema of your Amazon Redshift cluster. You can confirm this by executing the following query:

select distinct(tablename) from pg_table_def where schemaname = 'public'

  1. Build a QuickSight Dashboard from the aggregate fact

We can now create dashboards to visualize the data in our readmissions aggregate fact table using Amazon QuickSight. Here are some examples of reports you can generate using Amazon QuickSight on the readmission data.

For more details on Amazon QuickSight, refer to the service documentation.

  1. Build a ML model in Amazon SageMaker to predict readmissions

As a final step, we will create a ML model to predict the attribute readmission_result, which denotes if a patient was readmitted or not, using the non-PHI dataset.

  1. Create a notebook instance in Amazon SageMaker that is used to develop our code.
  2. The code reads non-PHI data from the Amazon S3 bucket as a data frame in Python. This is achieved using the pandas.readcsv function.

  1. Use the pandas.get_dummies function to encode categorical values into numeric values for use with the model.

  1. Split the data into two, 80% for training and 20% for testing, using the numpy.random.rand function.

  1. Form train_X, train_y and test_X, test_y corresponding to training features, training labels, testing features, and testing labels respectively.

  1. Use the Amazon SageMaker Linear learner algorithm to train our model. The implementation of the algorithm uses dense tensor format to optimize the training job. Use the function write_numpy_to_dense_tensor from the Amazon SageMaker library to convert the numpy array into the dense tensor format.

  1. Create the training job in Amazon SageMaker with appropriate configurations and run it.

  1. Once the training job completes, create an endpoint in Amazon SageMaker to host our model, using the linear.deploy function to deploy the endpoint.

  1. Finally, run a prediction by invoking the endpoint using the linear_predictor.predict function.

You can view the complete notebook here.

Data, analytics, and ML are strategic assets to help you manage your patients, staff, equipment, and supplies more efficiently. These technologies can also help you be more proactive in treating and preventing disease. Industry luminaries share this opinion: “By leveraging big data and scientific advancements while maintaining the important doctor-patient bond, we believe we can create a health system that will go beyond curing disease after the fact to preventing disease before it strikes by focusing on health and wellness,” writes Lloyd B. Minor, MD, dean of the Stanford School of Medicine.

ML and analytics offer huge value in helping achieve the quadruple aim : improved patient satisfaction, improved population health, improved provider satisfaction, and reduced costs. Technology should never replace the clinician but instead become an extension of the clinician and allow them to be more efficient by removing some of the mundane, repetitive tasks involved in prevention, diagnostics, and treatment of patients.

(1) “The Digital Future of Pathology.” The Medical Futurist, 28 May 2018, medicalfuturist.com/digital-future-pathology.

(2) Wang, Dayong, et al. “Deep Learning for Identifying Metastatic Breast Cancer.” Deep Learning for Identifying Metastatic Breast Cancer, 18 June 2016, arxiv.org/abs/1606.05718.

About the Author

Stephen Jepsen is a Global HCLS Practice Manager in AWS Professional Services.

 

Amazon Pinpoint Offers Customers a New Console Experience with Richer Segmentation Capabilities to Drive End-User Engagement

Post Syndicated from Zach Barbitta original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-pinpoint-offers-customers-a-new-console-experience-with-richer-segmentation-capabilities-to-drive-end-user-engagement/

Last night, we launched a significant update to the Amazon Pinpoint console. These updates make it easier to start sending campaigns with Amazon Pinpoint. In fact, you can start engaging with your customers in as few as three clicks!

This update also includes advanced segmentation capabilities, which enable you to filter based on user, endpoint, and channel attributes and bind those filters with certain conditions, such as AND, OR, and EQUAL TO operators. These features help you create finely-tuned segments, which result in better engagement and happier end-users. To learn more about creating segments in Amazon Pinpoint, see our tutorial in the Amazon Pinpoint User Guide.

Amazon Pinpoint’s new segmentation capabilities enable the following use cases:

  • Create new sub-segments from existing segments: You can now create sub-segments by selecting an existing segment and applying segmentation logic to it. For example, you could import a list of email addresses and filter out certain customers based on real-time endpoint attributes, such as the customer’s paid or free tier status. To learn more about creating segments, see Building Segments in the Amazon Pinpoint User Guide.
  • Advanced segmentation using smart operators: You can now create dynamic segments that incorporate certain operators, such as AND, OR, and EQUAL TO. For example, let’s say you classify your customers into tiers: tier 1 and tier 2. With today’s launch, you could create a new segment of customers whose tier status is equal to tier 1 OR tier 2, and then further refine your segment to include users who were not active in past 7 days.
  • Understand the impact of additional filters on the size of your segments, and see your audience size at a glance: The updated segmentation tool shows you the total size of your segment at all times. This size estimate is continuously updated as you add more filters to the segment, letting you see the impact of your filters in near real-time.

These new features are available to all Amazon Pinpoint users today at no additional cost.

Changes to our sending review process

Post Syndicated from Dustin Taylor original https://aws.amazon.com/blogs/messaging-and-targeting/changes-to-our-sending-review-process/

We’re changing some of the language we use in our sending review process to make our communications clearer and more helpful.

If you’re not familiar with the sending review process, it refers to the actions that we take when there are issues with the email sent from an Amazon SES or Amazon Pinpoint account. Usually, these issues are a result of senders making honest mistakes. However, when email providers receive problematic email from a sender, they can’t tell if the sender made a mistake, or if they’re doing something malicious. If an email provider detects a problem that’s severe enough, they might block all incoming email from the sender’s IP address. If that happens, email sent from other senders who use the same IP address is blocked as well.

For this reason, we look for certain patterns and behaviors that could cause deliverability problems, and then work with our customers to help resolve the issues with the email sent from their accounts. We used to call this our enforcement process, but we now refer to it as our sending review process. This name is a much better description of the process (not to mention a bit friendlier).

You might notice some other changes as well. When the reputation metrics for an account (such as the account’s bounce or complaint rate) exceed certain levels, or another issue occurs that could impact the reputation of that account, we’ll monitor the email sending behaviors of that account for a certain period of time. During this time, we make a note of whether the problem gets better or worse. Previously, this period was called a probation period; we now call it a review period.

If an account is under review, but the sender isn’t able to correct the issue before the end of the review period, we’ll temporarily disable the account’s ability to send any more email. We take this action to protect the reputation of the sender, and to ensure that other customers can send email without experiencing deliverability issues. We used to call this a suspension, but that name seemed very permanent and punitive. We now refer to these events as sending pauses, because in the majority of cases, they’re temporary and reversible.

Finally, if a sender disagrees with our decision to place a review period or sending pause on their account, they can contact us to let us know why we made this decision in error. This used to be known as an appeal, but we now call it a review.

If we ever change the status of your account, such as by implementing a review period or sending pause, we’ll contact you by email at the address associated with your AWS account. We recommend that you make sure that we have the right email address. For information about changing the email address associated with your AWS account, see Managing an AWS Account in the AWS Billing and Cost Management User Guide.

In addition to sending you a notification by email, we’ll also update the reputation dashboard in the Amazon SES console to show the current status of your account. To learn more about the reputation dashboard, see Using the Reputation Dashboard in the Amazon SES Developer Guide.

How to map out your migration of Oracle PeopleSoft to AWS

Post Syndicated from Ashok Shanmuga Sundaram original https://aws.amazon.com/blogs/architecture/how-to-map-out-your-migration-of-oracle-peoplesoft-to-aws/

Oracle PeopleSoft Enterprise is a widely used enterprise resource planning (ERP) application. Customers run production deployments of various PeopleSoft applications on AWS, including PeopleSoft Human Capital Management (HCM), Financials and Supply Chain Management (FSCM), Interactive Hub (IAH), and Customer Relationship Management (CRM).

We published a whitepaper on Best Practices for Running Oracle PeopleSoft on AWS in December 2017. It provides architectural guidance and outlines best practices for high availability, security, scalability, and disaster recovery for running Oracle PeopleSoft applications on AWS.

It also covers highly available, scalable, and cost-effective multi-region reference architectures for deploying PeopleSoft applications on AWS, like the one illustrated below.

While migrating your Oracle PeopleSoft applications to AWS, here are some things to keep in mind:

  • Multi-AZ deployments – Deploy your PeopleSoft servers and database across multiple Availability Zones (AZs) for high availability. AWS AZs allow you to operate production applications and databases that are more highly available, fault tolerant, and scalable than would be possible from a single data center.
  • Use Amazon Relational Database Service (Amazon RDS) to deploy your PeopleSoft databaseAmazon RDS makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, allowing you to focus on your applications and business. Deploying an RDS for Oracle Database in multiple AZs simplifies creating a highly available architecture because you’ll have built-in support for automated failover from your primary database to a synchronously replicated secondary database in an alternative AZ.
  • Migration of large databases – Migrating large databases to Amazon RDS within a small downtime window requires careful planning:
    • We recommend that you take a point-in-time export of your database, transfer it to AWS, import it into Amazon RDS, and then apply the delta changes from on-premises.
    • Use AWS Direct Connect or AWS Snowball to transfer the export dump to AWS.
    • Use AWS Database Migration Service to apply the delta changes and sync the on-premises database with the Amazon RDS instance.
  • AWS Infrastructure Event Management (IEM) – Take advantage of AWS IEM to mitigate risks and help ensure a smooth migration. IEM is a highly focused engagement where AWS experts provide you with architectural and operational guidance, assist you in reviewing and fine-tuning your migration plan, and provide real-time support for your migration.
  • Cost optimization – There are a number of ways you can optimize your costs on AWS, including:
    • Use reserved instances for environments that are running most of the time, like production environments. A Reserved Instance is an EC2 offering that provides you with a significant discount (up to 75%) on EC2 usage compared to On-Demand pricing when you commit to a one-year or three-year term.
    • Shut down resources that are not in use. For example, development and test environments are typically used for only eight hours a day during the work week. You can stop these resources when they are not in use for a potential cost savings of 75% (40 hours vs. 168 hours). Use the AWS Instance Scheduler to automatically start and stop your Amazon EC2 and Amazon RDS instances based on a schedule.

The Configuring Amazon RDS as an Oracle PeopleSoft Database whitepaper has detailed instructions on configuring a backend Amazon RDS database for your Oracle PeopleSoft deployment on AWS. After you read the whitepaper, I recommend these other resources as your next step:

  • For a real-world case study on migrating a large Oracle database to AWS, check out this blog post about how AFG migrated their mission-critical Oracle Siebel CRM system running on Oracle Exadata on-premises to Amazon RDS for Oracle.
  • For more information on running Oracle Enterprise Solutions on AWS, check out this re:Invent 2017 video.
  • You can find more Oracle on AWS resources here and here.

About the author

Ashok Shanmuga Sundaram is a partner solutions architect with the Global System Integrator (GSI) team at Amazon Web Services. He works with the GSIs to provide guidance on enterprise cloud adoption, migration and strategy.

Compute Abstractions on AWS: A Visual Story

Post Syndicated from Massimo Re Ferre original https://aws.amazon.com/blogs/architecture/compute-abstractions-on-aws-a-visual-story/

When I joined AWS last year, I wanted to find a way to explain, in the easiest way possible, all the options it offers to users from a compute perspective. There are many ways to peel this onion, but I want to share a “visual story” that I have created.

I define the compute domain as “anything that has CPU and Memory capacity that allows you to run an arbitrary piece of code written in a specific programming language.” Your mileage may vary in how you define it, but this is broad enough that it should cover a lot of different interpretations.

A key part of my story is around the introduction of different levels of compute abstractions this industry has witnessed in the last 20 or so years.

Separation of duties

The start of my story is a line. In a cloud environment, this line defines the perimeter between the consumer role and the provider role. In the cloud, there are things that AWS will do and things that the consumer will do. The perimeter of these responsibilities varies depending on the services you opt to use. If you want to understand more about this concept, read the AWS Shared Responsibility Model documentation.

The different abstraction levels

The reason why the line above is oblique is because it needs to intercept different compute abstraction levels. If you think about what happened in the last 20 years of IT, we have seen a surge of different compute abstractions that changed the way people consume CPU and Memory resources. It all started with physical (x86) servers back in the 80s, and then we have seen the industry adding abstraction layers over the years (for example, hypervisors, containers, functions).

The higher you go in the abstraction levels, the more the cloud provider can add value and can offload the consumer from non-strategic activities. A lot of these activities tend to be “undifferentiated heavy lifting.” We define this as something that AWS customers have to do but that don’t necessarily differentiate them from their competitors (because those activities are table-stakes in that particular industry).

What we found is that supporting millions of customers on AWS requires a certain degree of flexibility in the services we offer because there are many different patterns, use cases, and requirements to satisfy. Giving our customers choices is something AWS always strives for.

A couple of final notes before we dig deeper. The way this story builds up through the blog post is aligned to the progression of the launch dates of the various services, with a few noted exceptions. Also, the services mentioned are all generally available and production-grade. For full transparency, the integration among some of them may still be work-in-progress, which I’ll call out explicitly as we go.

The instance (or virtual machine) abstraction

This is the very first abstraction we introduced on AWS back in 2006. Amazon Elastic Compute Cloud (Amazon EC2) is the service that allows AWS customers to launch instances in the cloud. When customers intercept us at this level, they retain responsibility of the guest operating system and above (middleware, applications, etc.) and their lifecycle. AWS has the responsibility for managing the hardware and the hypervisor including their lifecycle.

At the very same level of the stack there is also Amazon Lightsail, which “is the easiest way to get started with AWS for developers, small businesses, students, and other users who need a simple virtual private server (VPS) solution. Lightsail provides developers compute, storage, and networking capacity and capabilities to deploy and manage websites and web applications in the cloud.”

And this is how these two services appear in our story:

The container abstraction

With the rise of microservices, a new abstraction took the industry by storm in the last few years: containers. Containers are not a new technology, but the rise of Docker a few years ago democratized access. You can think of a container as a self-contained environment with soft boundaries that includes both your own application as well as the software dependencies to run it. Whereas an instance (or VM) virtualizes a piece of hardware so that you can run dedicated operating systems, a container technology virtualizes an operating system so that you can run separated applications with different (and often incompatible) software dependencies.

And now the tricky part. Modern containers-based solutions are usually implemented in two main logical pieces:

  • A containers control plane that is responsible for exposing the API and interfaces to define, deploy, and lifecycle containers. This is also sometimes referred to as the container orchestration layer.
  • A containers data plane that is responsible for providing capacity (as in CPU/Memory/Network/Storage) so that those containers can actually run and connect to a network. From a practical perspective this is typically a Linux host or less often a Windows host where the containers get started and wired to the network.

Arguably, in a specific compute abstraction discussion, the data plane is key, but it is as important to understand what’s happening for the control plane piece.

In 2014, Amazon launched a production-grade containers control plane called Amazon Elastic Container Service (ECS), which “is a highly scalable, high performance container management service that supports Docker … Amazon ECS eliminates the need for you to install, operate, and scale your own cluster management infrastructure.”

In 2017, Amazon also announced the intention to release a new service called Amazon Elastic Container Service for Kubernetes (EKS) based on Kubernetes, a successful open source containers control plane technology. Amazon EKS was made generally available in early June 2018.

Just like for ECS, the aim for this service is to free AWS customers from having to manage a containers control plane. In the past, AWS customers would spin up EC2 instances and deploy/manage their own Kubernetes masters (masters is the name of the Kubernetes hosts running the control plane) on top of an EC2 abstraction. However, we believe many AWS customers will leave to AWS the burden of managing this layer by either consuming ECS or EKS, depending on their use cases. A comparison between ECS and EKS is beyond the scope of this blog post.

You may have noticed that what we have discussed so far is about the container control plane. How about the containers data plane? This is typically a fleet of EC2 instances managed by the customer. In this particular setup, the containers control plane is managed by AWS while the containers data plane is managed by the customer. One could argue that, with ECS and EKS, we have raised the abstraction level for the control plane, but we have not yet really raised the abstraction level for the data plane as the data plane is still comprised of regular EC2 instances that the customer has responsibility for.

There is more on that later on but, for now, this is how the containers control plane and the containers data plane services appear:

The function abstraction

At re:Invent 2014, AWS introduced another abstraction layer: AWS Lambda. Lambda is an execution environment that allows an AWS customer to run a single function. So instead of having to manage and run a full-blown OS instance to run your code, or having to track all software dependencies in a user-built container to run your code, Lambda allows you to upload your code and let AWS figure out how to run it at scale.

What makes Lambda so special is its event-driven model. Not only can you invoke Lambda directly (for example, via the Amazon API Gateway), but you can trigger a Lambda function upon an event in another AWS service (for example, an upload to Amazon S3 or a change in an Amazon DynamoDB table).

The key point about Lambda is that you don’t have to manage the infrastructure underneath the function you are running. No need to track the status of the physical hosts, no need to track the capacity of the fleet, no need to patch the OS where the function will be running. In a nutshell, no need to spend time and money on the undifferentiated heavy lifting.

And this is how the Lambda service appears:

The bare metal abstraction

Also known as the “no abstraction.”

As recently as re:Invent 2017, we announced (the preview of) the Amazon EC2 bare metal instances. We made this service generally available to the public in May 2018.

This announcement is part of Amazon’s strategy to provide choice to our customers. In this case, we are giving customers direct access to hardware. To quote from Jeff Barr’s post:

“…. (AWS customers) wanted access to the physical resources for applications that take advantage of low-level hardware features such as performance counters and Intel® VT that are not always available or fully supported in virtualized environments, and also for applications intended to run directly on the hardware or licensed and supported for use in non-virtualized environments.”

This is how the bare metal Amazon EC2 i3.metal instance appears:

As a side note, and also as alluded to by Jeff, i3.metal is the foundational EC2 instance type on top of which VMware created their own VMware Cloud on AWS service. We are now offering the ability to any AWS user to provision bare metal instances. This doesn’t necessarily mean you can load your hypervisor of choice out of the box, but you can certainly do things you wouldn’t be able to do with a traditional EC2 instance (note: this was just a Saturday afternoon hack).

More seriously, a question I get often asked is whether users could install ESXi on i3.metal on their own. Today this cannot be done, but I’d be interested in hearing your use case for this.

The full container abstraction (for lack of a better term)

Now that we covered all the abstractions, it is time to go back and see if there are other optimizations we can provide for AWS customers. When we discussed the container abstraction, we called out that while there are two different fully managed containers control planes (ECS and EKS), there wasn’t a managed option for the data plane.

Some customers were (and still are) happy about being in full control of said instances. Others have been very vocal that they wanted to get out of the (undifferentiated heavy-lifting) business of managing the lifecycle of that piece of infrastructure.

Enter AWS Fargate, a production-grade service that provides compute capacity to AWS containers control planes. Practically speaking, Fargate is making the containers data plane fall into the “Provider space” responsibility. This means the compute unit exposed to the user is the container abstraction, while AWS will manage transparently the data plane abstractions underneath.

This is how the Fargate service appears:

Now ECS has two “launch types”: one called “EC2” (where your tasks get deployed on a customer-managed fleet of EC2 instances), and the other one called “Fargate” (where your tasks get deployed on an AWS-managed fleet of EC2 instances).

For EKS, the strategy will be very similar, but as of this writing it was not yet available. If you’re interested in some of the exploration being done to make this happen, this is a good read.

Conclusions

We covered the spectrum of abstraction levels available on AWS and how AWS customers can intercept them depending on their use cases and where they sit on their cloud maturity journey. Customers with a “lift & shift” approach may be more akin to consume services on the left-hand side of the slide, whereas customers with a more mature cloud native approach may be more interested in consuming services on the right-hand side of the slide.

In general, customers tend to use higher-level services to get out of the business of managing non-differentiating activities. For example, I recently talked to a customer interested in using Fargate. The trigger there was the fact that Fargate is ISO, PCI, SOC and HIPAA compliant, which was a huge time and money saver for them because it’s easier to point to an AWS document during an audit than having to architect and document for compliance the configuration of a DIY containers data plane.

As a recap, here’s our visual story with all the abstractions available:

I hope you found it useful. Any feedback is greatly appreciated.

About the author

Massimo is a Principal Solutions Architect at AWS. For about 25 years, he specialized on the x86 ecosystem starting with operating systems and virtualization technologies, and lately he has been head down learning about cloud and how application architectures are evolving in that space. Massimo has a blog at www.it20.info and his Twitter handle is @mreferre.