All posts by Steve Roberts

Introducing a Public Registry for AWS CloudFormation

Post Syndicated from Steve Roberts original

AWS CloudFormation and the AWS Cloud Development Kit (CDK) provide scalable and consistent provisioning of AWS resources (for example, compute infrastructure, monitoring tools, databases, and more). We’ve heard from many customers that they’d like to benefit from the same consistency and scalability when provisioning resources from AWS Partner Network (APN) members, third-party vendors, and open-source technologies, regardless of whether they are using CloudFormation templates or have adopted the CDK to define their cloud infrastructure.

I’m pleased to announce a new public registry for CloudFormation, providing a searchable collection of extensions – resource types or modules – published by AWS, APN partners, third parties, and the developer community. The registry makes it easy to discover and provision these extensions in your CloudFormation templates and CDK applications in the same manner you use AWS-provided resources. Using extensions, you no longer need to create and maintain custom provisioning logic for resource types from third-party vendors. And, you are able to use a single infrastructure as code tool, CloudFormation, to provision and manage AWS and third-party resources, further simplifying the infrastructure provisioning process (the CDK uses CloudFormation under the hood).

Launch Partners
We’re excited to be joined by over a dozen APN Partners for the launch of the registry, with more than 35 extensions available for you to use today. Blog posts and announcements from the APN Partners who collaborated on this launch, along with AWS Quick Starts, can be found below (some will be added in the next few days).

Registries and Resource Types
In 2019, CloudFormation launched support for private registries. These enabled registration and use of resource providers (Lambda functions) in your account, including providers from AWS and third-party vendors. After you registered a provider you could use resource types, comprised of custom provisioning logic, from the provider in your CloudFormation templates. Resource types were uploaded by providers to an Amazon Simple Storage Service (Amazon S3) bucket, and you used the types by referencing the relevant S3 URL. The public registry provides consistency in the sourcing of resource types and modules, and you no longer need to use a collection of Amazon Simple Storage Service (Amazon S3) buckets.

Third-party resource types in the public registry also integrate with drift detection. After creating a resource from a third-party resource type, CloudFormation will detect changes to the resource from its template configuration, known as configuration drift, just as it would with AWS resources. You can also use AWS Config to manage compliance for third-party resources consumed from the registry. The resource types are automatically tracked as Configuration items when you have configured AWS Config to record them, and used CloudFormation to create, update, and delete them. Whether the resource types you use are third-party or AWS resources, you can view configuration history for them, in addition to being able to write AWS Config rules to verify configuration best practices.

The public registry also supports Type Configuration, enabling you to configure third-party resource types with API keys and OAuth tokens per account and region. Once set, the configuration is stored securely and can be updated. This also provides a centralized way to configure third-party resource types.

Publishing Extensions to the Public Registry
Extension publishers must be verified as AWS Marketplace sellers, or as GitHub or BitBucket users, and extensions are validated against best practices. To publish extensions (resource types or modules) to the registry, you must first register in an AWS Region, using one of the mentioned account types.

After you’ve registered, you next publish your extension to a private registry in the same Region. Then, you need to test that the extension meets publishing requirements. For a resource type extension, this means it must pass all the contract tests defined for the type. Modules are subject to different requirements, and you can find more details in the documentation. With testing complete, you can publish your extension to the public registry for your Region. See the user guide for detailed information on publishing extensions.

Using Extensions in the Public Registry
I decided to try a couple of extensions related to Kubernetes, contributed by AWS Quick Starts, to make configuration changes to a cluster. Personally, I don’t have a great deal of experience with Kubernetes and its API so this was a great chance to examine how extensions could save me significant time and effort. During the process of writing this post I learned from others that using the Kubernetes API (the usual way to achieve the changes I had in mind) would normally involve effort even for those with more experience.

For this example I needed a Kubernetes cluster, so I followed this tutorial to set one up in Amazon Elastic Kubernetes Service (EKS), using the Managed nodes – Linux node type. With my cluster ready, I want to make two configuration changes.

First, I want to add a new namespace to the cluster. A namespace is a partitioning construct that lets me deploy the same set of resources to different namespaces in the same cluster without conflict thanks to the isolation namespaces provide. Second, I want to set up and use Helm, a package manager for Kubernetes. I’ll use Helm to install the kube-state-metrics package from the Prometheus helm-charts repository for gathering cluster metrics. While I can use CloudFormation to provision clusters and compute resources, previously, to perform these two configuration tasks, I’d have had to switch to the API or various bespoke tool chains. With the registry, and these two extensions, I can now do everything using CloudFormation (and of course, as I mentioned earlier, I could also use the extensions with the CDK, which I’ll show later).

Before using an extension, it needs to be activated in my account. While activation is easy to do for single accounts using the console, as we’ll see in a moment, if I were using AWS Organizations and wanted to activate various third-party extensions across my entire organization, or for a specific organization unit (OU), I could achieve this using Service-Managed StackSets in CloudFormation. Using the resource type AWS::CloudFormation::TypeActivation in a template submitted to a Service-Managed StackSet, I can target an entire Organization, or a particular OU, passing the Amazon Resource Name (ARN) identifying the third-party extension to be activated. Activation of extensions is also very easy to achieve (whether using AWS Organizations or not) using the CDK with just a few lines of code, again making use of the aforementioned TypeActivation resource type.

To activate the extensions, I head to the CloudFormation console and click Public extensions from the navigation bar. This takes me to the Registry:Public extensions home page, where I switch to viewing third party resource type extensions.

Viewing third-party types in the registry

The extensions I want are AWSQS::Kubernetes::Resource and AWSQS::Kubernetes::Helm. The Resource extension is used to apply a manifest describing configuration changes to a cluster. In my case, the manifest requests a namespace be created. Clicking the name of the AWSQS::Kubernetes::Resource extension takes me to a page where I can view schema, configuration details, and versions for the extension.

Viewing details of the Resource extension

What happens if you deactivate an extension you’re using, or an extension is withdrawn by the publisher? If you deactivate an extension a stack depends on, any resources created from that extension won’t be affected, but you’ll be unable to perform further stack operations, such as Read, Update, Delete, and List (these will fail until the extension is re-activated). Publishers must request their extensions be withdrawn from the registry (there is no “delete” API). If the request is granted, customers who activated the extension prior to withdrawal can still perform Create/Read/Update/Delete/List operations, using what is effectively a snapshot of the extension in their account.

Clicking Activate takes me to a page where I need to specify the ARN of an execution role that CloudFormation will assume when it runs the code behind the extension. I create a role following this user guide topic, but the basic trust relationship is below for reference.

  "Version": "2012-10-17",
  "Statement": [
      "Effect": "Allow",
      "Principal": {
        "Service": ""
      "Action": "sts:AssumeRole"

I also add permissions for the resource types I’m using to my execution role. Details on the permissions needed for the types I chose can be found on GitHub, for Helm, and for Kubernetes (note the GitHub examples include the trust relationship too).

When activating an extension, I can elect to use the default name, which is how I will refer to the type in my templates or CDK applications, or I can enter a new name. The name chosen has to be unique within my account, so if I’ve enabled a version of an extension with its default name, and want to enable a different version, I must change the name. Once I’ve filled in the details, and chosen my versioning strategy (extensions use semantic versioning, and I can elect to accept automatic updates for minor version changes, or to “lock” to a specific version) clicking Activate extension completes the process.

Activating an extension from the registry

That completes the process for the first extension, and I follow the same steps for the AWSQS::Kubernetes::Helm extension. Navigating to Activated extensions I can view a list of all my enabled extensions.

Viewing the list of enabled extensions

I have one more set of permissions to update. Resource types make calls to the Kubernetes API on my behalf so I need to update the aws-auth ConfigMap for my cluster to reference the execution role I just used, otherwise the calls made by the resource types I’m using will fail. To do this, I run the command kubectl edit cm aws-auth -n kube-system at a command prompt. In the text editor that opens, I update the ConfigMap with a new group referencing my CfnRegistryExtensionExecRole, shown below (if you’re following along, be sure to change the account ID and role name to match yours).

apiVersion: v1
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::111122223333:role/myAmazonEKSNodeRole
      username: system:node:{{EC2PrivateDNSName}}
    - groups:
      - system:masters
      rolearn: arn:aws:iam::111122223333:role/CfnRegistryExtensionExecRole
      username: cfnresourcetypes
kind: ConfigMap
  creationTimestamp: "2021-06-04T20:44:24Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "6355"
  selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
  uid: dc91bfa8-1663-45d0-8954-1e841913b324

Now I’m ready to use the extensions to configure my cluster with a new namespace, Helm, and the kube-state-metrics package. I create a CloudFormation template that uses the extensions, adding parameters for the elements I want to specify when creating a stack: the name of the cluster to update, and the namespace name. The properties for the KubeStateMetrics resource reference the package I want Helm to install.

AWSTemplateFormatVersion: "2010-09-09"
    Type: String
    Type: String
    Type: AWSQS::Kubernetes::Helm
      ClusterID: !Ref ClusterName
      Name: kube-state-metrics
      Namespace: !GetAtt KubeNamespace.Name
      Chart: prometheus-community/kube-state-metrics
    Type: AWSQS::Kubernetes::Resource
      ClusterName: !Ref ClusterName
      Namespace: default
      Manifest: !Sub |
        apiVersion: v1
        kind: Namespace
          name: ${Namespace}
            name: ${Namespace}

On the Stacks page of the CloudFormation console, I click Create stack, upload my template, and then give my stack a name and the values for my declared parameters.

Launching a stack with my activated extensions

I click Next to proceed through the rest of the wizard, leaving other settings at their default values, and then Create stack to complete the process.

Once stack creation is complete, I verify my changes using the kubectl command line tool. I first check that the new namespace, newsblog-sample-namespace, is present with the command kubectl get namespaces. I then run the kubectl get all --namespace newsblog-sample-namespace command to verify the kube-state-metrics package is installed.

Verifying the extensions applied by changes

Extensions can also be used with the AWS Cloud Development Kit. To wrap up this exploration of using the new registry, I’ve included an example below of a CDK application snippet in TypeScript that achieves the same effect, using the same extensions, as the YAML template I showed earlier (I could also have written this using any of the languages supported by the CDK – C#, Java, or Python).

import {Stack, Construct, CfnResource} from '@aws-cdk/core';
export class UnoStack extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);
    const clusterName = 'newsblog-cluster';
    const namespace = 'newsblog-sample-namespace';

    const kubeNamespace = new CfnResource(this, 'KubeNamespace', {
      type: 'AWSQS::Kubernetes::Resource',
      properties: {
        ClusterName: clusterName,
        Namespace: 'default',
        Manifest: this.toJsonString({
          apiVersion: 'v1',
          kind: 'Namespace',
          metadata: {
            name: namespace,
            labels: {
              name: namespace,
    new CfnResource(this, 'KubeStateMetrics', {
      type: 'AWSQS::Kubernetes::Helm',
      properties: {
        ClusterID: clusterName,
        Name: 'kube-state-metrics',
        Namespace: kubeNamespace.getAtt('Name').toString(),
        Repository: '',
        Chart: 'prometheus-community/kube-state-metrics',

As mentioned earlier in this post, I don’t have much experience with the Kubernetes API, and Kubernetes in general. However, by making use of the resource types in the public registry, in conjunction with CloudFormation, I was able to easily configure my cluster using a familiar environment, without needing to resort to the API or bespoke tool chains.

Get Started with the CloudFormation Public Registry
Pricing for the public registry is the same as for the existing registry and private resource types. There is no additional charge for using native AWS resource types; for third-party resource types you will incur charges based on the number of handler operations (add, delete, list, etc.) you run per month. For details, see the AWS CloudFormation Pricing page. The new public registry is available today in the US East (N. Virginia, Ohio), US West (Oregon, N. California), Canada (Central), Europe (Ireland, Frankfurt, London, Stockholm, Paris, Milan), Asia Pacific (Hong Kong, Mumbai, Osaka, Singapore, Sydney, Seoul, Tokyo), South America (Sao Paulo), Middle East (Bahrain), and Africa (Cape Town) AWS Regions.

For more information, see the AWS CloudFormation User Guide and User Guide for Extension Development, and start publishing or using extensions today!

— Steve

Migrate Your Workloads with the Graviton Challenge!

Post Syndicated from Steve Roberts original

Today, Dave Brown, VP of Amazon EC2 at AWS, announced the Graviton Challenge as part of his session on AWS silicon innovation at the Six Five Summit 2021. We invite you to take the Graviton Challenge and move your applications to run on AWS Graviton2. The challenge, intended for individual developers and small teams, is based on the experiences of customers who’ve already migrated. It provides a framework of eight, approximately four-hour chunks to prepare, port, optimize, and finally deploy your application onto Graviton2 instances. Getting your application running on Graviton2, and enjoying the improved price performance, aren’t the only rewards. There are prizes and swag for those who complete the challenge!

AWS Graviton2 is a custom-built processor from AWS that’s based on the Arm64 architecture. It’s supported by popular Linux operating systems including Amazon Linux 2, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and Ubuntu. Compared to fifth-generation x86-based Amazon Elastic Compute Cloud (Amazon EC2) instance types, Graviton2 instance types have a 20% lower cost. Overall, customers who have moved applications to Graviton2 typically see up to 40% better price performance for a broad range of workloads including application servers, container-based applications, microservices, caching fleets, data analytics, video encoding, electronic design automation, gaming, open-source databases, and more.

Before I dive in to talk more about the challenge, check out the fun introductory video below from Jeff Barr, Chief Evangelist, AWS and Dave Brown, Vice President, EC2. As Jeff mentions in the video: same exact workload, same or better performance, and up to 40% better price performance!

After you complete the challenge, we invite you to tell us about your adoption journey and enter the contest. If you post on social media with the hashtag #ITookTheGravitonChallenge, you’ll earn a t-shirt. To earn a hoodie, include a short video with your post.

To enter the competition, you’ll need to create a 5 to 10-minute video that describes your project and the application you migrated, any hurdles you needed to overcome, and the price performance benefits you realized.

All valid contest entries will each receive a $500 AWS credit (limited to 500 quantity). A panel of judges will evaluate the content entries and award additional prizes across six categories. All category winners will receive an AWS re:Invent 2021 conference pass, flight, and hotel for one company representative, and winners will be able to meet with senior members of the Graviton2 team at the conference. Here are additional category-specific prizes:

  • Best adoption – enterprise
    Based on the performance gains, total cost savings, number of instances the workload is running on, and time taken to migrate the workload (faster is better), for companies with over 1000 employees. The winner will also receive a chance to present at the conference.
  • Best adoption – small/medium business
    Based on the performance gains, total cost savings, number of instances the workload is running on, and time taken to migrate the workload (faster is better), for companies with 100-1000 employees. The winner will also receive a chance to present at the conference.
  • Best adoption – startup
    Based on the performance gains, total cost savings, number of instances the workload is running on, and time taken to migrate the workload (faster is better), for companies with fewer than 100 employees. The winner will also receive a chance to present at the conference.
  • Best new workload adoption
    Awarded to a workload that’s new to EC2 (migrated to Graviton2 from on-premises, or other cloud) based on the performance gains, total cost savings, number of instances the workload is running on, and time taken to migrate the workload (faster is better). The winner will also receive a chance to participate in a video or written case study.
  • Most impactful adoption
    Awarded to the workload with the biggest social impact based on details provided about what the workload/application does. Applications in this category are related to fields such as sustainability, healthcare and life sciences, conservation, learning/education, justice/equity. The winner will also receive a chance to participate in a video or written case study.
  • Most innovative adoption
    Applications in this category solve unique problems for their customers, address new use cases, or are groundbreaking. The award will be based on the workload description, price performance gains, and total cost savings. The winner will also receive a chance to participate in a video or written case study.

Competition submissions open on June 22 and close August 31. Winners will be announced on October 1 2021.

Identifying a workload to migrate
Now that you know what’s possible with Graviton2, you’re probably eager to get started and identify a workload to tackle as part of the challenge. The ideal workload is one that already runs on Linux and uses open-source components. This means you’ll have full access to the source code of every component and can easily make any required changes. If you don’t have an existing Linux workload that is entirely open-source based, you can, of course, move other workloads. A robust ecosystem of ISVs and AWS services already support Graviton2. However, if you are using software from a vendor that does not support Arm64/Graviton2, reach out to the Graviton Challenge Slack channel for support.

What’s involved in the challenge?
The challenge includes eight steps performed over four days (but you don’t have to do the challenge in four consecutive days). If you need assistance from Graviton2 experts, a dedicated Slack channel is available and you can sign up for emails containing helpful tips and guidance. In addition to support on Slack and supporting emails, you also get $25 AWS credit to cover the cost of the taking the challenge. Graviton2-based burstable T4g instances also have a free trial, available until December 31 2021, that can be used to qualify your workloads.

You can download the complete whitepaper can be downloaded from the Graviton Challenge page, but here is an outline of the process.

Day 1: Learn and explore
The first day you’ll learn about Graviton2 and then assess your selected workload. I recommend that you start by checking out the 2020 AWS re:Invent session, Deep dive on AWS Graviton2 processor-powered EC2 instances. The Getting Started with AWS Graviton GitHub repository will be a useful reference as you work through the challenge.

Assessment involves identifying the application’s dependencies and requirements. As with all preparatory work, the more thorough you are at this stage, the better positioned you are for success. So, don’t skimp on this task!

Day 2: Create a plan and start porting
On the second day, you’ll create a Graviton2 environment. You can use EC2 virtual machine instances with AWS-provided images or build your own custom images. Alternatively, you can go the container route, because both Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (EKS) support Graviton2-based instances.

After you have created your environment, you’ll bootstrap the application. The Getting Started Guide on GitHub contains language-specific getting started information. If your application uses Java, Python, Node.js, .NET, or other high-level languages, then it might run as-is or need minimal changes. Other languages like C, C++, or Go will need to be compiled for the 64-bit Arm architecture. For more information, see the guides on GitHub.

Day 3: Debug and optimize
Now that the application is running on a Graviton2 environment, it’s time to test and verify its functionality. When you have a fully functional application, you can test performance and compare it to x86-64 environments. If you don’t observe the expected performance, reach out to your account team, or get support on the Graviton Challenge Slack channel. We’re here to help analyze and resolve any potential performance gaps.

Day 4: Update infrastructure and start deployments
It’s shipping day! You’ll update your infrastructure to add Graviton2-based instances, and then start deploying. We recommend that you use canary or blue-green deployments so that a portion of your traffic is redirected to the new environments. When you’re comfortable, you can transition all traffic.

At this point, you can celebrate completing the challenge, publish a post on social media using the #ITookTheGravitonChallenge hashtag, let us know about your success, and consider entering the competition. Remember, entries for the competition are due by August 31, 2021.

Start the challenge today!
Now that you have some details about the challenge and rewards, it’s time to start your (migration) engines. Download the whitepaper from the Graviton Challenge landing page, familiarize yourself with the details, and off you go! And, if you do decide to enter the competition, good luck!

In my role as a .NET Developer Advocate at AWS, I would be remiss if I failed to mention that this challenge is equally applicable to .NET applications using .NET Core or .NET 5 and later! In fact, .NET 5 includes ARM64-specific optimizations. For information about performance improvements my colleagues found for .NET applications running on AWS Graviton2, see the Powering .NET 5 with AWS Graviton2: Benchmarks blog post. There’s also a lab for .NET 5 on Graviton2. I invite you to check out the getting started material for .NET in the aws-graviton-getting-started GitHub repository and start migrating.

— Steve

New – AWS Systems Manager Consolidates Application Management

Post Syndicated from Steve Roberts original

A desire for consolidated, and simplified operational oversight isn’t limited to just cloud infrastructure. Increasingly, customers ask us for a “single pane of glass” approach for also monitoring and managing their application portfolios.

These customers tell us that detection and investigation of application issues takes additional time and effort, due to the typical use of multiple consoles, tools, and sources of information such as resource usage metrics, logs, and more, to enable their DevOps engineers to obtain context about the application issue under investigation. Here, an “application” means not just the application code but also the logical group of resources that act as a unit to host the application, along with ownership boundaries for operators, and environments such as development, staging, and production.

Today, I’m pleased to announce a new feature of AWS Systems Manager, called Application Manager. Application Manager aggregates operational information from multiple AWS services and Systems Manager capabilities into a single console, making it easier to view operational data for your applications.

To make it even more convenient, the service can automatically discover your applications. Today, auto-discovery is available for applications running in AWS CloudFormation stacks and Amazon Elastic Kubernetes Service (EKS) clusters, or launched using AWS Launch Wizard. Applications can also be discovered from Resource Groups.

A particular benefit of automated discovery is that application components and resources are automatically kept up-to-date on an ongoing basis, but you can also always revise applications as needed by adding or deleting components manually.

With applications discovered and consolidated into a single console, you can more easily diagnose operational issues and resolve them with minimal time and effort. Automated runbooks targeting an application component or resource can be run to help remediate operational issues. For any given application, you can select a resource and explore relevant details without needing to leave the console.

For example, the application can surface Amazon CloudWatch logs, operational metrics, AWS CloudTrail logs, and configuration changes, removing the need to engage with multiple tools or consoles. This means your on-call engineers can understand issues more quickly and reduce the time needed to resolve them.

Exploring an Application with Application Manager
I can access Application Manager from the Systems Manager home page. Once open, I get an overview of my discovered applications and can see immediately that there are some alarms, without needing to switch context to the Amazon CloudWatch console, and some operations items (“OpsItems”) that I might need to pay attention to. I can also switch to the Applications tab to view the collections of applications, or I can click the buttons in the Applications panel for the collection I’m interested in.

Screenshot of the <span title="">Application Manager</span> overview page

In the screenshot below, I’ve navigated to a sample application and again, have indicators showing that alarms have raised. The various tabs enable me to drill into more detail to view resources used by the application, config resource and rules compliance, monitoring alarms, logs, and automation runbooks associated with the application.

Screenshot of application components and overview

Clicking on the Alarm indicator takes me into the Monitoring tab, and it shows that the ConsumedWriteCapacityUnits alarm has been raised. I can change the timescale to zero in on when the event occurred, or I can use the View recent alarms dashboard link to jump into the Amazon CloudWatch Alarms console to view more detail.

Screenshot of alarms on the <span title="">Application Manager</span> Monitoring tab

The Logs tab shows me a consolidated list of log groups for the application, and clicking a log group name takes me directly to the CloudWatch Logs where I can inspect the log streams, and take advantage of Log Insights to dive deeper by querying the log data.

OpsItems shows me operational issues associated with the resources of my application, and enables me to indicate the current status of the issue (open, in progress, resolved). Below, I am marking investigation of a stopped EC2 instance as in progress.

Screenshot of <span title="">Application Manager</span> OpsItems tab

Finally, Runbooks shows me automation documents associated with the application and their execution status. Below, it’s showing that I ran the AWS-RestartEC2Instance automation document to restart the EC2 instance that was stopped, and I would now resolve the issue logged in the OpsItems tab.

Screenshot of <span title="">Application Manager</span>'s Runbooks tab

Consolidating this information into a single console gives engineers a single starting location to monitor and investigate issues arising with their applications, and automatic discovery of applications and resources makes getting started simple. AWS Systems Manager Application Manager is available today, at no extra charge, in all public AWS Regions where Systems Manager is available.

Learn more about Application Manager and get started at AWS Systems Manager.

— Steve

New – AWS Systems Manager Fleet Manager

Post Syndicated from Steve Roberts original

Organizations, and their systems administrators, routinely face challenges in managing increasingly diverse portfolios of IT infrastructure across cloud and on-premises environments. Different tools, consoles, services, operating systems, procedures, and vendors all contribute to complicate relatively common, and related, management tasks. As workloads are modernized to adopt Linux and open-source software, those same systems administrators, who may be more familiar with GUI-based management tools from a Windows background, have to continually adapt and quickly learn new tools, approaches, and skill sets.

AWS Systems Manager is an operational hub enabling you to manage resources on AWS and on-premises. Available today, Fleet Manager is a new console based experience in Systems Manager that enables systems administrators to view and administer their fleets of managed instances from a single location, in an operating-system-agnostic manner, without needing to resort to remote connections with SSH or RDP. As described in the documentation, managed instances includes those running Windows, Linux, and macOS operating systems, in both the AWS Cloud and on-premises. Fleet Manager gives you an aggregated view of your compute instances regardless of where they exist.

All that’s needed, whether for cloud or on-premises servers, is the Systems Manager agent installed on each server to be managed, some AWS Identity and Access Management (IAM) permissions, and AWS Key Management Service (KMS) enabled for Systems Manager‘s Session Manager. This makes it an easy and cost-effective approach for remote management of servers running in multiple environments without needing to pay the licensing cost of expensive management tools you may be using today. As noted earlier, it also works with instances running macOS. With the agent software and permissions set up, Fleet Manager enables you to explore and manage your servers from a single console environment. For example, you can navigate file systems, work with the registry on Windows servers, manage users, and troubleshoot logs (including viewing Windows event logs) and monitor common performance counters without needing the Amazon CloudWatch agent to be installed.

Exploring an Instance With Fleet Manager
To get started exploring my instances using Fleet Manager, I first head to the Systems Manager console. There, I select the new Fleet Manager entry on the navigation toolbar. I can also select the Managed Instances option – Fleet Manager replaces Managed Instances going forward, but the original navigation toolbar entry will be kept for backwards compatibility for a short while. But, before we go on to explore my instances, I need to take you on a brief detour.

When you select Fleet Manager, as with some other views in Systems Manager, a check is performed to verify that a role, named AmazonSSMRoleForInstancesQuickSetup, exists in your account. If you’ve used other components of Systems Manager in the past, it’s quite possible that it does. The role is used to permit Systems Manager to access your instances on your behalf and if the role exists, then you’re directed to the requested view. If however the role doesn’t exist, you’ll first be taken to the Quick Setup view. This in itself will trigger creation of the role, but you might want to explore the capabilities of Quick Setup, which you can also access any time from the navigation toolbar.

Quick Setup is a feature of Systems Manager that you can use to set up specific configuration items, such as the Systems Manager and CloudWatch agents on your instances (and keep them up-to-date), and also IAM roles permitting access to your resources for Systems Manager components. For this post, all the instances I’m going to use already have the required agent set up, including the role permissions, so I’m not going to discuss this view further but I encourage you to check it out. I also want to remind you that to take full advantage of Fleet Manager‘s capabilities you first need to have KMS encryption enabled for your instances and secondly, the role attached to your Amazon Elastic Compute Cloud (EC2) instances must have the kms:Decrypt role permission included, referencing the key you selected when you enabled KMS encryption. You can enable encryption, and select the KMS key, using the Preferences section of the Session Manager console, and of course you can set up the role permission in the IAM console.

That’s it for the diversion; if you have the role already, as I do, you’ll now be at the Managed instances list view. If you’re at Quick Setup instead, simply click the Fleet Manager navigation button once more.

The Managed instances view shows me all of my instances, in the cloud or on-premises, that I can access. Selecting an instance, in this case an EC2 Windows instance launched using AWS Elastic Beanstalk, and clicking Instance actions presents me with a menu of options. The options (less those specific to Windows) are available for my Amazon Linux instance too, and for instances running macOS I can use the View file system option.

Screenshot of <span title="">Fleet Manager</span>'s Managed instances view

The File system view displays a read-only view onto the file system of the selected instance. This can be particularly useful for viewing text-based log files, for example, where I can preview up to 10,000 lines of a log file and even tail it to view changes as the log updates. I used this to open and tail an IIS web server log on my Windows Server instance. Having selected the instance, I next select View file system from the Instance actions dropdown (or I can click the Instance ID to open a view onto that instance and select File system from the menu displayed on the instance view).

Having opened the file system view for my instance, I navigate to the folder on the instance containing the IIS web server logs.

Screenshot of <span title="">Fleet Manager</span>'s File system view

Selecting a log file, I then click Actions and select Tail file. This opens a view onto the log file contents, which updates automatically as new content is written.

Screenshot of tailing a log file in <span title="">Fleet Manager</span>

As I mentioned, the File system view is also accessible for macOS-based instances. For example, here is a screenshot of viewing the Applications folder on an EC2 macOS instance.

Screenshot of macOS file system view in <span title="">Fleet Manager</span>

Next, let’s examine the Performance counters view, which is available for both Windows and Linux instances. This view displays CPU, memory, network traffic, and disk I/O and will be familiar to Windows users from Task Manager. The metrics shown reflect the guest OS metrics, whereas EC2 instance metrics you may be used to relate to the hypervisor. On this particular instance I’ve deployed an ASP.NET Core 5 application, which generates a varying length collection of Fibonacci numbers on page refresh. Below is a snapshot of the counters, after I’ve put the instance under a small amount of load. The view updates automatically every 5 seconds.

Screenshot of <span title="">Fleet Manager</span>'s Performance Counters view

There are more views available than I have space for in this post. Using the Windows Registry view, I can view and edit the registry on the selected Windows instance. Windows event logs gives me access to the Application and Service logs, and common Windows logs such as System, Setup, Security, etc. With Users and groups I can manage users or groups, including assignment of users to groups (again for both Windows and Linux instances). For all views, Fleet Manager enables me to use a single and convenient console.

Getting Started
AWS Systems Manager Fleet Manager is available today for use with managed instances running Windows, Linux, and macOS. Information on pricing, for this and other Systems Manager features, can be found at this page.

Learn more, and get started with Fleet Manager today, at AWS Systems Manager.

— Steve

AWS Audit Manager Simplifies Audit Preparation

Post Syndicated from Steve Roberts original

Gathering evidence in a timely manner to support an audit can be a significant challenge due to manual, error-prone, and sometimes, distributed processes. If your business is subject to compliance requirements, preparing for an audit can cause significant lost productivity and disruption as a result. You might also have trouble applying traditional audit practices, which were originally designed for legacy on-premises systems, to your cloud infrastructure.

To satisfy complex and evolving sets of regulation and compliance standards, including the General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPAA), and Payment Card Industry Data Security Standard (PCI DSS), you’ll need to gather, verify, and synthesize evidence.

You’ll also need to constantly reevaluate how your AWS usage maps to those evolving compliance control requirements. To satisfy requirements you may need to show data encryption was active, and log files showing server configuration changes, diagrams showing application high availability, transcripts showing required training was completed, spreadsheets showing that software usage did not exceed licensed amounts, and more. This effort, sometimes involving dozens of staff and consultants, can last several weeks.

Available today, AWS Audit Manager is a fully managed service that provides prebuilt frameworks for common industry standards and regulations, and automates the continual collection of evidence to help you in preparing for an audit. Continuous and automated gathering of evidence related to your AWS resource usage helps simplify risk assessment and compliance with regulations and industry standards and helps you maintain a continuous, audit-ready posture to provide a faster, less disruptive preparation process.

Built-in and customizable frameworks map usage of your cloud resources to controls for different compliance standards, translating evidence into an audit-ready, immutable assessment report using auditor-friendly terminology. You can also search, filter, and upload additional evidence to include in the final assessment, such as details of on-premises infrastructure, or procedures such as business continuity plans, training transcripts, and policy documents.

Given that audit preparation typically involves multiple teams, a delegation workflow feature lets you assign controls to subject-matter experts for review. For example, you might delegate reviewing evidence of network security to a network security engineer.

The finalized assessment report includes summary statistics and a folder containing all the evidence files, organized in accordance with the exact structure of the associated compliance framework. With the evidence collected and organized into a single location, it’s ready for immediate review, making it easier for audit teams to verify the evidence, answer questions, and add remediation plans.

Getting started with Audit Manager
Let’s get started by creating and configuring a new assessment. From Audit Manager‘s console home page, clicking Launch AWS Audit Manager takes me to my Assessments list (I can also reach here from the navigation toolbar to the left of the console home). There, I click Create assessment to start a wizard that walks me through the settings for the new assessment. First, I give my assessment a name, optional description, and then specify an Amazon Simple Storage Service (S3) bucket where the reports associated with the assessment will be stored.

Next, I choose the framework for my assessment. I can select from a variety of prebuilt frameworks, or a custom framework I have created myself. Custom frameworks can be created from scratch or based on an existing framework. Here, I’m going to use the prebuilt PCI DSS framework.

Screenshot of framework selectionAfter clicking Next, I can select the AWS accounts to be included in my assessment (Audit Manager is also integrated with AWS Organizations). Since I have a single account, I select it and click Next, moving on to select the AWS services that I want to be included in evidence gathering. I’m going to include all the suggested services (the default) and click Next to continue.

Screenshot of service selection for assessmentNext I need to select the owners of the assessment, who have full permission to manage it (owners can be AWS Identity and Access Management (IAM) users or roles). You must select at least one owner, so I select my account and click Next to move to the final Review and create page. Finally, clicking Create assessment starts the gathering of evidence for my new assessment. This can take a while to complete, so I’m going to switch to another assessment to examine what kinds of evidence I can view and choose to include in my assessment report.

Back in the Assessments list view, clicking on the assessment name takes me to details of the assessment, a summary of the controls for which evidence is being collected, and a list of the control sets into which the controls are grouped. Total evidence tells me the number of events and supporting documents that are included in the assessment. The additional tabs can be used to give me insight into the evidence I select for the final report, which accounts and services are included in the assessment, who owns it, and more. I can also navigate to the S3 bucket in which the evidence is being collected.

Screenshot of assessment home pageExpanding a control set shows me the related controls, with links to dive deeper on a given control, together with the status (Under review, Reviewed, and Inactive), whom the control has been delegated to for review, the amount of evidence gathered for that control, and whether the control and evidence have been added to the final report. If I change a control to be Inactive, meaning automated evidence gathering will cease for that control, this is logged.

Screenshot of assessment controlLet’s take a closer look at a control to show how the automated evidence gathering can help identify compliance issues before I start compiling the audit report. Expanding Default control set, I click control 8.1.2 For a sample of privileged user IDs… which takes me to a view giving more detailed information on the control and how it is tested. Scrolling down, there is a set of evidence folders listed and here I notice that there are some issues. Clicking the issue link in the Compliance check column summarizes where the data came from. Here, I can also select the evidence that I want included in my final report.

Screenshot of issue summaryGoing further, I can click on the evidence folder to note that there was a failure, and in turn clicking on the time of the failure takes me to a detailed summary of the issues for this control, and how to remediate.

Screenshot of evidence for a controlScreenshot of evidence detailWith the evidence gathered, it’s a simple task to select sufficient controls and appropriate evidence to include in my assessment report that can then be passed to my auditors. For the purposes of this post I’ve gone ahead and selected evidence for a handful of controls into my report. Then, I selected the Assessment report selection tab, where I review my evidence selections, and clicked Generate assessment report. In the dialog that appeared I gave my report a name, and then clicked Generate assessment report. When the dialog closes I am taken to the Assessment reports view and, when my report is ready, I can select it and download a zip file containing the report and the selected evidence. Alternatively, I can open the S3 bucket associated with the assessment (from the assessment’s details page) and view the report details and evidence there, as shown in the screenshot below. The overall report is listed (as a PDF file) and if I drill into the evidence folders, I can also view PDF files related to the specific items of evidence I selected.

Screenshot of assessment report output in S3And to close, below is a screenshot of the beginning of the assessment report PDF file showing the number of selected controls and evidence, and services that I selected to be in scope when I created the assessment. Further pages go into more details.

Screenshot of assessment reportAudit Manager is available today in 10 AWS Regions: US East (Northern Virginia, Ohio), US West (Northern California, Oregon), Asia Pacific (Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, London).

Get all the details about AWS Audit Manager and get started today.

— Steve

Using Amazon CloudWatch Lambda Insights to Improve Operational Visibility

Post Syndicated from Steve Roberts original

To balance costs, while at the same time ensuring the service levels needed to meet business requirements are met, some customers elect to continuously monitor and optimize their AWS Lambda functions. They collect and analyze metrics and logs to monitor performance, and to isolate errors for troubleshooting purposes. Additionally, they also seek to right-size function configurations by measuring function duration, CPU usage, and memory allocation. Using various tools and sources of data to do this can be time-consuming, and some even go so far as to build their own customized dashboards to surface and analyze this data.

We announced Amazon CloudWatch Lambda Insights as a public preview this past October for customers looking to gain deeper operational oversight and visibility into the behavior of their Lambda functions. Today, I’m pleased to announce that CloudWatch Lambda Insights is now generally available. CloudWatch Lambda Insights provides clearer and simpler operational visibility of your functions by automatically collating and summarizing Lambda performance metrics, errors, and logs in prebuilt dashboards, saving you from time-consuming, manual work.

Once enabled on your functions, CloudWatch Lambda Insights automatically starts collecting and summarizing performance metrics and logs, and, from a convenient dashboard, provides you with a one-click drill-down into metrics and errors for Lambda function requests, simplifying analysis and troubleshooting.

Exploring CloudWatch Lambda Insights
To get started, I need to enable Lambda Insights on my functions. In the Lambda console, I navigate to my list of functions, and then select the function I want to enable for Lambda Insights by clicking on its name. From the function’s configuration view I then scroll to the Monitoring tools panel, click Edit, enable Enhanced monitoring, and click Save. If you want to enable enhanced monitoring for many functions, you may find it more convenient to use AWS Command Line Interface (CLI), AWS Tools for PowerShell, or AWS CloudFormation approaches instead. Note that once enhanced monitoring has been enabled, it can take a few minutes before data begins to surface in CloudWatch.

Screenshot showing enabling of <span title="">Lambda Insights</span> In the Amazon CloudWatch Console, I start by selecting Performance monitoring beneath Lambda Insights in the navigation panel. This takes me to the Multi-function view. Metrics for all functions on which I have enabled Lambda Insights are graphed in the view. At the foot of the page there’s also a table listing the functions, summarizing some data in the graphs and adding Cold starts. The table gives me the ability to sort the data based on the metric I’m interested in.

Screenshot of metric graphs on the <span title="">Lambda Insights</span> Multi-function viewScreenshot of the <span title="">Lambda Insights</span> Multi-function view summary listAn interesting graph on this page, especially if you are trying to balance cost with performance, is Function Cost. This graph shows the direct cost of your functions in terms of megabyte milliseconds (MB-MS), which is how Lambda computes the financial charge of a function’s invocation. Hovering over the graph at a particular point in time shows more details.

Screenshot of function cost graphLet’s examine my ExpensiveFunction further. Moving to the summary list at the bottom of the page I click on the function name which takes me to the Single function view (from here I can switch to my other functions using the controls at the top of the page, without needing to return to the multiple function view). The graphs show me metrics for invocations and errors, duration, any throttling, and memory, CPU, and network usage on the selected function and to add to the detail available, the most recent 1000 invocations are also listed in a table which I can sort as needed.

Clicking View in the Trace column of a request in the invocations list takes me to the Service Lens trace view, showing where my function spent its time on that particular invocation request. I could use this to determine if changes to the business logic of the function might improve performance by reducing function duration, which will have a direct effect on cost. If I’m troubleshooting, I can view the Application or Performance logs for the function using the View logs button. Application logs are those that existed before Lambda Insights was enabled on the function, whereas Performance logs are those that Lambda Insights has collated across all my enabled functions. The log views enable me to run queries and in the case of the Performance logs I can run queries across all enabled functions in my account, for example to perform a top-N analysis to determine my most expensive functions, or see how one function compares to another.

Here’s how I can make use of Lambda Insights to check if I’m ‘moving the needle’ in the correct direction when attempting to right-size a function, by examining the effect of changes to memory allocation on function cost. The starting point for my ExpensiveFunction is 128MB. By moving from 128MB to 512MB, the data shows me that function cost, duration, and concurrency are all reduced – this is shown at (1) in the graphs. Moving from 512MB to 1024MB, (2), has no impact on function cost, but it further reduces duration by 50% and also affected the maximum concurrency. I ran two further experiments, first moving from 1024MB to 2048MB, (3), which resulted in a further reduction in duration but the function cost started to increase so the needle is starting to swing in the wrong direction. Finally, moving from 2048MB to 3008MB, (4), significantly increased the cost but had no effect on duration. With the aid of Lambda Insights I can infer that the sweet spot for this function (assuming latency is not a consideration) lies between 1024MB and 2048MB. All these experiments are shown in the graphs below (the concurrency graph lags slightly, as earlier invocations are finishing up as configuration changes are made).

Screenshot of function cost experiments

CloudWatch Lambda Insights gives simple and convenient operational oversight and visibility into the behavior of my AWS Lambda functions, and is available today in all regions where AWS Lambda is present.

Learn more about Amazon CloudWatch Lambda Insights in the documentation and get started today.

— Steve

Coming Soon – Amazon EC2 G4ad Instances Featuring AMD GPUs for Graphics Workloads

Post Syndicated from Steve Roberts original

Customers with high performance graphic workloads, such as game streaming, animation, and video rendering for example, are always looking for higher performance with less cost. Today, I’m happy to announce new Amazon Elastic Compute Cloud (EC2) instances in the G4 instance family are in the works and will be available soon, to improve performance and reduce cost for graphics-intensive workloads. The new G4ad instances feature AMD’s latest Radeon Pro V520 GPUs and 2nd generation EPYC processors, and are the first in EC2 to feature AMD GPUs.

G4dn instances, released in 2019 and featuring NVIDIA T4 GPUs, were previously the most cost-effective GPU-based instances in EC2. G4dn instances are ideal for deploying machine learning models in production and also graphics-intensive applications. However, when compared to G4dn the new G4ad instances enable up to 45% better price performance for graphics-intensive workloads, including the aforementioned game streaming, remote graphics workstations, and rendering scenarios. Compared to an equally-sized G4dn instance, G4ad instances offer up to 40% improvement in performance.

G4dn instances will continue to be the best option for small-scale machine learning (ML) training and GPU-based ML inference due to included hardware optimizations like Tensor Cores. Additionally, G4dn instances are still best suited for graphics applications that need access to NVIDIA libraries such as CUDA, CuDNN, and NVENC. However, when there is no dependency on NVIDIA’s libraries, we recommend customers try the G4ad instances to benefit from the improved price and performance.

AMD Radeon Pro V520 GPUs in G4ad instances support DirectX 11/12, Vulkan 1.1, and OpenGL 4.5 APIs. For operating systems, customers can choose from Windows Server 2016/2019, Amazon Linux 2, Ubuntu 18.04.3, and CentOS 7.7. Instances using G4ad can be purchased as On-Demand, Savings Plan, Reserved Instances, or Spot Instances. Three instance sizes are available, from G4ad.4xlarge, with 1 GPU, to G4ad.16xlarge with 4 GPUs, as shown below.

Instance Size GPUs GPU Memory (GB) vCPUs Memory (GB) Storage EBS Bandwidth (Gbps) Network Bandwidth (Gbps)
g4ad.4xlarge 1 8 16 64 600 Up to 3 Up to 10
g4ad.8xlarge 2 16 32 128 1200 3 15
g4ad.16xlarge 4 32 64 256 2400 6 25

The new G4ad instances will be available soon in US East (N. Virginia), US West (Oregon), and Europe (Ireland).

Learn more about G4ad instances.

— Steve

Announcing a second Local Zone in Los Angeles

Post Syndicated from Steve Roberts original

In December 2019, Jeff Barr published this post announcing the launch of a new Local Zone in Los Angeles, California. A Local Zone extends existing AWS regions closer to end-users, providing single-digit millisecond latency to a subset of AWS services in the zone. Local Zones are attached to parent regions – in this case US West (Oregon) – and access to services and resources is performed through the parent region’s endpoints. This makes Local Zones transparent to your applications and end-users. Applications running in a Local Zone have access to all AWS services, not just the subset in the zone, via Amazon’s redundant and very high bandwidth private network backbone to the parent region.

At the end of the post, Jeff wrote (and I quote) – “In the fullness of time (as Andy Jassy often says), there could very well be more than one Local Zone in any given geographic area. In 2020, we will open a second one in Los Angeles (us-west-2-lax-1b), and are giving consideration to other locations.”

Well – that time has come! I’m pleased to announce that following customer requests, AWS today launched a second Local Zone in Los Angeles to further help customers in the area (and Southern California generally) to achieve even greater high-availability and fault tolerance for their applications, in conjunction with very low latency. These customers have workloads that require very low latency, such as artist workstations, local rendering, gaming, financial transaction processing, and more.

The new Los Angeles Local Zone contains a subset of services, such as Amazon Elastic Compute Cloud (EC2), Amazon Elastic Block Store (EBS), Amazon FSx for Windows File Server and Amazon FSx for Lustre, Elastic Load Balancing, Amazon Relational Database Service (RDS), and Amazon Virtual Private Cloud, and remember, applications running in the zone can access all AWS services and other resources through the zone’s association with the parent region.

Enabling Local Zones
Once you opt-in to use one or more Local Zones in your account, they appear as additional Availability Zones for you to use when deploying your applications and resources. The original zone, launched last December, was us-west-2-lax-1a. The additional zone, available to all customers now, is us-west-2-lax-1b. Local Zones can be enabled using the new Settings section of the EC2 Management Console, as shown below

As noted in Jeff’s original post, you can also opt-in (or out) of access to Local Zones with the AWS Command Line Interface (CLI) (aws ec2 modify-availability-zone-group command), AWS Tools for PowerShell (Edit-EC2AvailabilityZoneGroup cmdlet), or by calling the ModifyAvailabilityZoneGroup API from one of the AWS SDKs.

Using Local Zones for Highly-Available, Low-Latency Applications
Local Zones can be used to further improve high availability for applications, as well as ultra-low latency. One scenario is for enterprise migrations using a hybrid architecture. These enterprises have workloads that currently run in existing on-premises data centers in the Los Angeles metro area and it can be daunting to migrate these application portfolios, many of them interdependent, to the cloud. By utilizing AWS Direct Connect in conjunction with a Local Zone, these customers can establish a hybrid environment that provides ultra-low latency communication between applications running in the Los Angeles Local Zone and the on-premises installations without needing a potentially expensive revamp of their architecture. As time progresses, the on-premises applications can be migrated to the cloud in an incremental fashion, simplifying the overall migration process. The diagram below illustrates this type of enterprise hybrid architecture.

Anecdotally, we have heard from customers using this type of hybrid architecture, combining Local Zones with AWS Direct Connect, that they are achieving sub-1.5ms latency communication between the applications hosted in the Local Zone and those in the on-premises data center in the LA metro area.

Virtual desktops for rendering and animation workloads is another scenario for Local Zones. For these workloads, latency is critical and the addition of a second Local Zone for Los Angeles gives additional failover capability, without sacrificing latency, should the need arise.

As always, the teams are listening to your feedback – thank you! – and are working on adding Local Zones in other locations, along with the availability of additional services, including Amazon ECS Amazon Elastic Kubernetes Service, Amazon ElastiCache, Amazon Elasticsearch Service, and Amazon Managed Streaming for Apache Kafka. If you’re an AWS customer based in Los Angeles or Southern California, with a requirement for even greater high-availability and very low latency for your workloads, then we invite you to check out the new Local Zone. More details and pricing information can be found at Local Zone.

— Steve

Announcing the Porting Assistant for .NET

Post Syndicated from Steve Roberts original

.NET Core is the future of .NET! Version 4.8 of the .NET Framework is the last major version to be released, and Microsoft has stated it will receive only bug-, reliability-, and security-related fixes going forward. For applications where you want to continue to take advantage of future investments and innovations in the .NET platform, you need to consider porting your applications to .NET Core. Also, there are additional reasons to consider porting applications to .NET Core such as benefiting from innovation in Linux and open source, improved application scaling and performance, and reducing licensing spend. Porting can, however, entail significant manual effort, some of which is undifferentiated such as updating references to project dependencies.

When porting .NET Framework applications, developers need to search for compatible NuGet packages and update those package references in the application’s project files, which also need to be updated to the .NET Core project file format. Additionally, they need to discover replacement APIs since .NET Core contains a subset of the APIs available in the .NET Framework. As porting progresses, developers have to wade through long lists of compile errors and warnings to determine the best or highest priority places to continue chipping away at the task. Needless to say, this is challenging, and the added friction could be a deterrent for customers with large portfolios of applications.

Today we announced the Porting Assistant for .NET, a new tool that helps customers analyze and port their .NET Framework applications to .NET Core running on Linux. The Porting Assistant for .NET assesses both the application source code and the full tree of public API and NuGet package dependencies to identify those incompatible with .NET Core and guides developers to compatible replacements when available. The suggestion engine for API and package replacements is designed to improve over time as the assistant learns more about the usage patterns and frequency of missing packages and APIs.

The Porting Assistant for .NET differs from other tools in that it is able to assess the full tree of package dependencies, not just incompatible APIs. It also uses solution files as the starting point, which makes it easier to assess monolithic solutions containing large numbers of projects, instead of having to analyze and aggregate information on individual binaries. These and other abilities gives developers a jump start in the porting process.

Analyzing and porting an application
Getting started with porting applications using the Porting Assistant for .NET is easy, with just a couple of prerequisites. First, I need to install the .NET Core 3.1 SDK. Secondly I need a credential profile (compatible with the AWS Command Line Interface (CLI), although the CLI is not used or required). The credential profile is used to collect compatibility information on the public APIs and packages (from NuGet and core Microsoft packages) used in your application and public NuGet packages that it references. With those prerequisites taken care of, I download and run the installer for the assistant.

With the assistant installed, I check out my application source code and launch the Porting Assistant for .NET from the Start menu. If I’ve previously assessed some solutions, I can view and open those from the Assessed solutions screen, enabling me to pick up where I left off. Or I can select Get started, as I’ll do here, from the home page to begin assessing my application’s solution file.

I’m asked to select the credential profile I want to use, and here I can also elect to opt-in to share my telemetry data. Sharing this data helps to further improve on suggestion accuracy for all users as time goes on, and is useful in identifying issues, so we hope you consider opting-in.

I click Next, browse to select the solution file that I want, and then click Assess to begin the analysis. For this post I’m going to use the open source NopCommerce project.

When analysis is complete I am shown the overall results – the number of incompatible packages the application depends on, APIs it uses that are incompatible, and an overall Portability score. This score is an estimation of the effort required to port the application to .NET Core, based on the number of incompatible APIs it uses. If I’m working on porting multiple applications I can use this to identify and prioritize the applications I want to start on first.

Let’s dig into the assessment overview to see what was discovered. Clicking on the solution name takes me to a more detailed dashboard and here I can see the projects that make up the application in the solution file, and for each the numbers of incompatible package and API dependencies, along with the portability score for each particular project. The current port status of each project is also listed, if I’ve already begun porting the application and have reopened the assessment.

Note that with no project selected in the Projects tab the data shown in the Project references, NuGet packages, APIs, and Source files tabs is solution-wide, but I can scope the data if I wish by first selecting a project.

The Project references tab shows me a graphical view of the package dependencies and I can see where the majority of the dependencies are consumed, in this case the Npp.Core, Npp.Services, and Npp.Web.Framework projects. This view can help me decide where I might want to start first, so as to get the most ‘bang for my buck’ when I begin. I can also select projects to see the specific dependencies more clearly.

The NuGet packages tab gives me a look at the compatible and incompatible dependencies, and suggested replacements if available. The APIs tab lists the incompatible APIs, what package they are in, and how many times they are referenced. Source files lists all of the source files making up the projects in the application, with an indication of how many incompatible API calls can be found in each file. Selecting a source file opens a view showing me where the incompatible APIs are being used and suggested package versions to upgrade to, if they exist, to resolve the issue. If there is no suggested replacement by simply updating to a different package version then I need to crack open a source editor and update the code to use a different API or approach. Here I’m looking at the report for DependencyRegistrar.cs, that exists in the Nop.Web project, and uses the Autofac NuGet package.

Let’s start porting the application, starting with the Nop.Core project. First, I navigate back to the Projects tab, select the project, and then click Port project. During porting the tool will help me update project references to NuGet packages, and also updates the project files themselves to the newer .NET Core formats. I have the option of either making a copy of the application’s solution file, project files, and source files, or I can have the changes made in-place. Here I’ve elected to make a copy.

Clicking Save copies the application source code to the selected location, and opens the Port projects view where I can set the new target framework version (in this case netcoreapp3.1), and the list of NuGet dependencies for the project that I need to upgrade. For each incompatible package the Porting Assistant for .NET gives me a list of possible version upgrades and for each version, I am shown the number of incompatible APIs that will either remain, or will additionally become incompatible. For the package I selected here there’s no difference but for cases where later versions potentially increase the number of incompatible APIs that I would then need to manually fix up in the source code, this indication helps me to make a trade-off decision around whether to upgrade to the latest versions of a package, or stay with an older one.

Once I select a version, the Deprecated API calls field alongside the package will give me a reminder of what I need to fix up in a code editor. Clicking on the value summarizes the deprecated calls.

I continue with this process for each package dependency and when I’m ready, click Port to have the references updated. Using my IDE, I can then go into the source files and work on replacing the incompatible API calls using the Porting Assistant for .NET‘s source file and deprecated API list views as a reference, and follow a similar process for the other projects in my application.

Improving the suggestion engine
The suggestion engine behind the Porting Assistant for .NET is designed to learn and give improved results over time, as customers opt-in to sharing their telemetry. The data models behind the engine, which are the result of analyzing hundreds of thousands of unique packages with millions of package versions, are available on GitHub. We hope that you’ll consider helping improve accuracy and completeness of the results by contributing your data. The user guide gives more details on how the data is used.

The Porting Assistant for .NET is free to use and is available now.

— Steve

Software Package Management with AWS CodeArtifact

Post Syndicated from Steve Roberts original

Software artifact repositories and their associated package managers are an essential component of development. Downloading and referencing pre-built libraries of software with a package manager, at the point in time the libraries are needed, simplifies both development and build processes. A variety of package repositories can be used, for example Maven Central, npm public registry, and PyPi (Python Package Index), among others. Working with a multitude of artifact repositories can present some challenges to organizations that want to carefully control both versions of, and access to, the software dependencies of their applications. Any changes to dependencies need to be controlled, to try and prevent undetected and exploitable vulnerabilities creeping into the organization’s applications. By using a centralized repository, it becomes easier for organizations to manage access control and version changes, and gives teams confidence that when updating package versions, the new versions have been approved for use by their IT leaders. Larger organizations may turn to traditional artifact repository software to solve these challenges, but these products can introduce additional challenges around installation, configuration, maintenance, and scaling. For smaller organizations, the price and maintenance effort of traditional artifact repository software may be prohibitive.

Generally available today, AWS CodeArtifact is a fully managed artifact repository service for developers and organizations to help securely store and share the software packages used in their development, build, and deployment processes. Today, CodeArtifact can be used with popular build tools and package managers such as Maven and Gradle (for Java), npm and yarn (for Javascript), and pip and twine (for Python), with more to come. As new packages are ingested, or published to your repositories, CodeArtifact automatically scales, and as a fully managed service, CodeArtifact requires no infrastructure installation or maintenance on your part. Additionally, CodeArtifact is a polyglot artifact repository, meaning it can store artifact packages of any supported type. For example, a single CodeArtifact repository could be configured to store packages from Maven, npm and Python repositories side by side in one location.

CodeArtifact repositories are organized into a domain. We recommend that you use a single domain for your organization, and then add repositories to it. For example you might choose to use different repositories for different teams. To publish packages into your repositories, or ingest packages from external repositories, you simply use the package manager tools your developers are used to. Let’s take a look at the process of getting started.

Getting started with CodeArtifact
To get started with CodeArtifact, I first need to create a domain for my organization, which will aggregate my repositories. Domains are used to perform the actual storage of packages and metadata, even though I consume them from a repository. This has the advantage that a single package asset, for example a given npm package, would be stored only once per domain no matter how many repositories it may appear to be in. From the CodeArtifact console, I can select Domains from the left-hand navigation panel, or instead create a domain as part of creating my first repository, which I’ll do here by clicking Create repository.

First, I give my repository a name and optional description, and I then have the option to connect my repository to several upstream repositories. When requests are made for packages not present in my repository, CodeArtifact will pull the respective packages from these upstream repositories for me, and cache them into my CodeArtifact repository. Note that a CodeArtifact repository can also act as an upstream for other CodeArtifact repositories. For the example here, I’m going to pull packages from the npm public registry and PyPi. CodeArtifact will refer to the repositories it creates on my behalf to manage these external connections as npm-store and pypi-store.

Clicking Next, I then select, or create, a domain which I do by choosing the account that will own the domain and then giving the domain a name. Note that CodeArtifact encrypts all assets and metadata in a domain using a single AWS Key Management Service (KMS) key. Here, I’m going to use a key that will be created for me by the service, but I can elect to use my own.

Clicking Next takes me to the final step to review my settings, and I can confirm the package flow from my selected upstream repositories is as I expect. Clicking Create repository completes the process, and in this case creates the domain, my repository, and two additional repositories representing the upstreams.

After using this simple setup process, my domain and its initial repository, configured to pull upstream from npm and PyPi, are now ready to hold software artifact packages, and I could also add additional repositories if needed. However my next step for this example is to configure the package managers for my upstream repositories, npm and pip, with access to the CodeArtifact repository, as follows.

Configuring package managers
The steps to configure various package managers can be found in the documentation, but conveniently the console also gives me the instructions I need when I select my repository. I’m going to start with npm, and I can access the instructions by first selecting my npm-pypi-example-repository and clicking View connection instructions.

In the resulting dialog I select the package manager I want to configure and I am shown the relevant instructions. I have the choice of using the AWS Command Line Interface (CLI) to manage the whole process (for npm, pip, and twine), or I can use a CLI command to get the token and then run npm commands to attach the token to the repository reference.

Regardless of the package manager, or the set of instructions I follow, the commands simply attach an authorization token, which is valid for 12 hours, to the package manager configuration for the repository. So that I don’t forget to refresh the token, I have taken the approach of adding the relevant command to my startup profile so that my token is automatically refreshed at the start of each day.

Following the same guidance, I similarly configure pip, again using the AWS CLI approach:

C:\> aws codeartifact login --tool pip --repository npm-pypi-example-repository --domain my-example-domain --domain-owner ACCOUNT_ID
Writing to C:\Users\steve\AppData\Roaming\pip\pip.ini
Successfully logged in to codeartifact for pypi

That’s it! I’m now ready to start using the single repository for dependencies in my Node.js and Python applications. Any dependency I add which is not already in the repository will be fetched from the designated upstream repositories and added to my CodeArtifact repository.

Let’s try some simple tests to close out the post. First, after changing to an empty directory, I execute a simple npm install command, in this case to install the AWS Cloud Development Kit.

npm install -g aws-cdk

Selecting the repository in the CodeArtifact console, I can see that the packages for the AWS Cloud Development Kit, and its dependencies, have now been downloaded from the upstream npm public registry repository, and added to my repository.

I mentioned earlier that CodeArtifact repositories are polyglot, and able to store packages of any supported type. Let’s now add a Python package, in this case Pillow, a popular image manipulation library.

> pip3 install Pillow
Looking in indexes: https://aws:****
Collecting Pillow
  Downloading (2.0 MB)
     |████████████████████████████████| 2.0 MB 819 kB/s
Installing collected packages: Pillow
Successfully installed Pillow-7.1.2

In the console, I can see the Python package sitting alongside the npm packages I added earlier.

Although I’ve used the console to verify my actions, I could equally well use CLI commands. For example, to list the repository packages I could have run the following command:

aws codeartifact list-packages --domain my-example-domain --repository npm-pypi-example-repository

As you might expect, additional commands are available to help with work with domains, repositories, and the packages they contain.

AWS CodeArtifact is now generally available in the Frankfurt, Ireland, Mumbai, N.Virginia, Ohio, Oregon, Singapore, Sweden, Sydney, and Tokyo regions. Tune in on June 12th at noon (PST) to or LinkedIn Live, where we will be showing how you can get started with CodeArtifact.

— Steve

Simplified Time-Series Analysis with Amazon CloudWatch Contributor Insights

Post Syndicated from Steve Roberts original

Inspecting multiple log groups and log streams can make it more difficult and time consuming to analyze and diagnose the impact of an issue in real time. What customers are affected? How badly? Are some affected more than others, or are outliers? Perhaps you performed deployment of an update using a staged rollout strategy and now want to know if any customers have hit issues or if everything is behaving as expected for the target customers before continuing further. All of the data points to help answer these questions is potentially buried in a mass of logs which engineers query to get ad-hoc measurements, or build and maintain custom dashboards to help track.

Amazon CloudWatch Contributor Insights, generally available today, is a new feature to help simplify analysis of Top-N contributors to time-series data in CloudWatch Logs that can help you more quickly understand who or what is impacting system and application performance, in real-time, at scale. This saves you time during an operational problem by helping you understand what is contributing to the operational issue and who or what is most affected. Amazon CloudWatch Contributor Insights can also help with ongoing analysis for system and business optimization by easily surfacing outliers, performance bottlenecks, top customers, or top utilized resources, all at a glance. In addition to logs, Amazon CloudWatch Contributor Insights can also be used with other products in the CloudWatch portfolio, including Metrics and Alarms.

Amazon CloudWatch Contributor Insights can analyze structured logs in either JSON or Common Log Format (CLF). Log data can be sourced from Amazon Elastic Compute Cloud (EC2) instances, AWS CloudTrail, Amazon Route 53, Apache Access and Error Logs, Amazon Virtual Private Cloud (VPC) Flow Logs, AWS Lambda Logs, and Amazon API Gateway Logs. You also have the choice of using structured logs published directly to CloudWatch, or using the CloudWatch Agent. Amazon CloudWatch Contributor Insights will evaluate these log events in real-time and display reports that show the top contributors and number of unique contributors in a dataset. A contributor is an aggregate metric based on dimensions contained as log fields in CloudWatch Logs, for example account-id, or interface-id in Amazon Virtual Private Cloud Flow Logs, or any other custom set of dimensions. You can sort and filter contributor data based on your own custom criteria. Report data from Amazon CloudWatch Contributor Insights can be displayed on CloudWatch dashboards, graphed alongside CloudWatch metrics, and added to CloudWatch alarms. For example customers can graph values from two Amazon CloudWatch Contributor Insights reports into a single metric describing the percentage of customers impacted by faults, and configure alarms to alert when this percentage breaches pre-defined thresholds.

Getting Started with Amazon CloudWatch Contributor Insights
To use Amazon CloudWatch Contributor Insights I simply need to define one or more rules. A rule is simply a snippet of data that defines what contextual data to extract for metrics reported from CloudWatch Logs. To configure a rule to identify the top contributors for a specific metric I supply three items of data – the log group (or groups), the dimensions for which the top contributors are evaluated, and filters to narrow down those top contributors. To do this, I head to the Amazon CloudWatch console dashboard and select Contributor Insights from the left-hand navigation links. This takes me to the Amazon CloudWatch Contributor Insights home where I can click Create a rule to get started.

To get started quickly, I can select from a library of sample rules for various services that send logs to CloudWatch Logs. You can see above that there are currently a variety of sample rules for Amazon API Gateway, Amazon Route 53 Query Logs, Amazon Virtual Private Cloud Flow Logs, and logs for container services. Alternatively, I can define my own rules, as I’ll do in the rest of this post.

Let’s say I have a deployed application that is publishing structured log data in JSON format directly to CloudWatch Logs. This application has two API versions, one that has been used for some time and is considered stable, and a second that I have just started to roll out to my customers. I want to know as early as possible if anyone who has received the new version, targeting the new api, is receiving any faults and how many faults are being triggered. My stable api version is sending log data to one log group and my new version is using a different group, so I need to monitor multiple log groups (since I also want to know if anyone is experiencing any error, regardless of version).

The JSON to define my rule, to report on 500 errors coming from any of my APIs, and to use account ID, HTTP method, and resource path as dimensions, is shown below.

  "Schema": {
    "Name": "CloudWatchLogRule",
    "Version": 1
  "AggregateOn": "Count",
  "Contribution": {
    "Filters": [
        "Match": "$.status",
        "EqualTo": 500
    "Keys": [
  "LogFormat": "JSON",
  "LogGroupNames": [

I can set up my rule using either the Wizard tab, or I can paste the JSON above into the Rule body field on the Syntax tab. Even though I have the JSON above, I’ll show using the Wizard tab in this post and you can see the completed fields below. When selecting log groups I can either select them from the drop down, if they already exist, or I can use wildcard syntax in the Select by prefix match option (MyApplicationLogsV* for example).

Clicking Create saves the new rule and makes it immediately start processing and analyzing data (unless I elect to create it in disabled state of course). Note that Amazon CloudWatch Contributor Insights processes new log data created once the rule is active, it does not perform historical inspection, so I need to build rules for operational scenarios that I anticipate happening in future.

With the rule in place I need to start generating some log data! To do that I’m going to use a script, written using the AWS Tools for PowerShell, to simulate my deployed application being invoked by a set of customers. Of those customers, a select few (let’s call them the unfortunate ones) will be directed to the new API version which will randomly fail on HTTP POST requests. Customers using the old API version will always succeed. The script, which runs for 5000 iterations, is shown below. The cmdlets being used to work with CloudWatch Logs are the ones with CWL in the name, for example Write-CWLLogEvent.

# Set up some random customer ids, and select a third of them to be our unfortunates
# who will experience random errors due to a bad api update being shipped!
$allCustomerIds = @( 1..15 | % { Get-Random })
$faultingIds = $allCustomerIds | Get-Random -Count 5

# Setup some log groups
$group1 = 'MyApplicationLogsV1'
$group2 = 'MyApplicationLogsV2'
$stream = "MyApplicationLogStream"

# When writing to a log stream we need to specify a sequencing token
$group1Sequence = $null
$group2Sequence = $null

$group1, $group2 | % {
    if (!(Get-CWLLogGroup -LogGroupName $_)) {
        New-CWLLogGroup -LogGroupName $_
        New-CWLLogStream -LogGroupName $_ -LogStreamName $stream
    } else {
        # When the log group and stream exist, we need to seed the sequence token to
        # the next expected value
        $logstream = Get-CWLLogStream -LogGroupName $_ -LogStreamName $stream
        $token = $logstream.UploadSequenceToken
        if ($_ -eq $group1) {
            $group1Sequence = $token
        } else {
            $group2Sequence = $token

# generate some log data with random failures for the subset of customers
1..5000 | % {

    Write-Host "Log event iteration $_" # just so we know where we are progress-wise

    $customerId = Get-Random $allCustomerIds

    # first select whether the user called the v1 or the v2 api
    $useV2Api = ((Get-Random) % 2 -eq 1)
    if ($useV2Api) {
        $resourcePath = '/api/v2/some/resource/path/'
        $targetLogGroup = $group2
        $nextToken = $group2Sequence
    } else {
        $resourcePath = '/api/v1/some/resource/path/'
        $targetLogGroup = $group1
        $nextToken = $group1Sequence

    # now select whether they failed or not. GET requests for all customers on
    # all api paths succeed. POST requests to the v2 api fail for a subset of
    # customers.
    $status = 200
    $errorMessage = ''
    if ((Get-Random) % 2 -eq 0) {
        $httpMethod = "GET"
    } else {
        $httpMethod = "POST"
        if ($useV2Api -And $faultingIds.Contains($customerId)) {
            $status = 500
            $errorMessage = 'Uh-oh, something went wrong...'

    # Write an event and gather the sequence token for the next event
    $nextToken = Write-CWLLogEvent -LogGroupName $targetLogGroup -LogStreamName $stream -SequenceToken $nextToken -LogEvent @{
        TimeStamp = [DateTime]::UtcNow
        Message = (ConvertTo-Json -Compress -InputObject @{
            requestId = [Guid]::NewGuid().ToString("D")
            httpMethod = $httpMethod
            resourcePath = $resourcePath
            status = $status
            protocol = "HTTP/1.1"
            accountId = $customerId
            errorMessage = $errorMessage

    if ($targetLogGroup -eq $group1) {
        $group1Sequence = $nextToken
    } else {
        $group2Sequence = $nextToken

    Start-Sleep -Seconds 0.25

I start the script running, and with my rule enabled, I start to see failures show up in my graph. Below is a snapshot after several minutes of running the script. I can clearly see a subset of my simulated customers are having issues with HTTP POST requests to the new v2 API.

From the Actions pull down in the Rules panel, I could now go on to create a single metric from this report, describing the percentage of customers impacted by faults, and then configure an alarm on this metric to alert when this percentage breaches pre-defined thresholds.

For the sample scenario outlined here I would use the alarm to halt the rollout of the new API if it fired, preventing the impact spreading to additional customers, while investigation of what is behind the increased faults is performed. Details on how to set up metrics and alarms can be found in the user guide.

Amazon CloudWatch Contributor Insights is available now to users in all commercial AWS Regions, including China and GovCloud.

— Steve

New – Multi-Attach for Provisioned IOPS (io1) Amazon EBS Volumes

Post Syndicated from Steve Roberts original

Starting today, customers running Linux on Amazon Elastic Compute Cloud (EC2) can take advantage of new support for attaching Provisioned IOPS (io1) Amazon Elastic Block Store (EBS) volumes to multiple EC2 instances. Each EBS volume, when configured with the new Multi-Attach option, can be attached to a maximum of 16 EC2 instances in a single Availability Zone. Additionally, each Nitro-based EC2 instance can support the attachment of multiple Multi-Attach enabled EBS volumes. Multi-Attach capability makes it easier to achieve higher availability for applications that provide write ordering to maintain storage consistency.

Applications can attach Multi-Attach volumes as non-boot data volumes, with full read and write permission. Snapshots can be taken of volumes configured for Multi-Attach, just as with regular volumes, but additionally the snapshot can be initiated from any instance that the volume is attached to, and Multi-Attach volumes also support encryption. Multi-Attach enabled volumes can be monitored using Amazon CloudWatch metrics, and to monitor performance per instance, you can use the Linux iostat tool.

Getting Started with Multi-Attach EBS Volumes
Configuring and using Multi-Attach volumes is a simple process for new volumes using either the AWS Command Line Interface (CLI) or the AWS Management Console. In a simple example for this post I am going to create a volume, configured for Multi-Attach, and attach it to two Linux EC2 instances. From one instance I will write a simple text file, and from the other instance I will read the contents. Let’s get started!

In the AWS Management Console I first navigate to the EC2 homepage, select Volumes from the navigation panel and then click Create Volume. Choosing Provisioned IOPS SSD (io1) for Volume Type, I enter my desired size and IOPS and then check the Multi-Attach option.

To instead do this using the AWS Command Line Interface (CLI) I simply use the ec2 create-volume command, with the --multi-attach-enabled option, as shown below.

aws ec2 create-volume --volume-type io1 --multi-attach-enabled --size 4 --iops 100 --availability-zone us-east-1a

I can verify that Multi-Attach is enabled on my volume from the Description tab when the volume is selected. The volume table also contains a column, Multi-Attach Enabled that displays a simple ‘yes/no’ value, enabling me to check if Multi-Attach is enabled across multiple volumes at a glance.

With the volume created and ready for use, I next launch two T3 EC2 instances running Linux. Remember, Multi-Attach needs an AWS Nitro System based instance type and the instances have to be created in the same Availability Zone as my volume. My instances are running Amazon Linux 2, and have been placed into the us-east-1a Availability Zone, matching the placement of my new Multi-Attach enabled volume.

Once the instances are running, it’s time to attach my volume to both of them. I click Volumes from the EC2 dashboard, then select the Multi-Attach volume I created. From the Actions menu, I click Attach Volume. In the screenshot below you can see that I have already attached the volume to one instance, and am attaching to the second.

If I’m using the AWS Command Line Interface (CLI) to attach the volume, I make use of the ec2 attach-volume command, as I would for any other volume type:

aws ec2 attach-volume --device /dev/sdf --instance-id i-0c44a... --volume-id vol-012721da...

For a given volume, the AWS Management Console shows me which instances it is attached to, or those currently being attached, when I select the volume:

With the volume attached to both instances, let’s make use of it with a simple test. Selecting my first instance in the Instances view of the EC2 dashboard, I click Connect and then open a shell session onto the instance using AWS Systems Manager‘s Session Manager. Following the instructions here, I created a file system on the new volume attached as /dev/sdf, mounted it as /data, and using vi I write some text to a file.

sudo mkfs -t xfs /dev/sdf
sudo mkdir /data
sudo mount /dev/sdf /data
cd /data
sudo vi file.txt

Selecting my second instance in the AWS Management Console, I repeat the connection steps. I don’t need to create a file system this time but I do again mount the /dev/sdf volume as /data (although I could use a different mount point if I chose). On changing directory to /data, I see that the file I wrote from my first instance exists, and contains the text I expect.

Creating and working with Multi-Attach volumes is simple! Just remember, they need to be attached to and be in the same Availability Zone as the instances they are to be attached to. This post obviously made use of a simple use case, but for any real-world application usage you might also want to consider implementing some form of write ordering, so as to ensure consistency is maintained.

Using Delete-on-Termination with Multi-Attach Volumes
If you prefer to make use of the option to delete attached volumes on EC2 instance termination then we recommend you have a consistent setting of the option across all of the instances that a Multi-Attach volume is attached to – use either all delete, or all retain, to allow for predictable termination behavior. If you attach the volume to a set of instances that have differing values for Delete-on-Termination then deletion of the volume depends on whether the last instance to detach is set to delete or not. A consistent setting obviously avoids any doubt!

For more information see the Amazon Elastic Block Store (EBS) technical documentation. Multi-Attach for Provisioned IOPS (io1) volumes on Amazon Elastic Block Store (EBS) is available today at no extra charge to customers in the US East (N. Virginia & Ohio), US West (Oregon), EU (Ireland), and Asia Pacific (Seoul) regions.

— Steve

Announcing UltraWarm (Preview) for Amazon Elasticsearch Service

Post Syndicated from Steve Roberts original

Today, we are excited to announce UltraWarm, a fully managed, low-cost, warm storage tier for Amazon Elasticsearch Service. UltraWarm is now available in preview and takes a new approach to providing hot-warm tiering in Amazon Elasticsearch Service, offering up to 900TB of storage, at almost a 90% cost reduction over existing options. UltraWarm is a seamless extension to the Amazon Elasticsearch Service experience, enabling you to query and visualize across both hot and UltraWarm data, all from your familiar Kibana interface. UltraWarm data can be queried using the same APIs and tools you use today, and also supports popular Amazon Elasticsearch Service features like encryption at rest and in flight, integrated alerting, SQL querying, and more.

A popular use case for our customers of Amazon Elasticsearch Service is to ingest and analyze high (and increasingly growing) volumes of machine-generated log data. However, those customers tell us that they want to perform real-time analysis on more of this data, so they can use it to help quickly resolve operational and security issues. Storage and analysis of months, or even years, of data has been cost prohibitive for them at scale, causing some to turn to use multiple analytics tools, while others simply delete valuable data, missing out on insights. UltraWarm, with its cost-effective storage backed by Amazon Simple Storage Service (S3), helps solve this problem, enabling customers to retain years of data for analysis.

With the launch of UltraWarm, Amazon Elasticsearch Service supports two storage tiers, hot and UltraWarm. The hot tier is used for indexing, updating, and providing the fastest access to data. UltraWarm complements the hot tier to add support for high volumes of older, less-frequently accessed, data to enable you to take advantage of a lower storage cost. As I mentioned earlier, UltraWarm stores data in S3 and uses custom, highly-optimized nodes, built on the AWS Nitro System, to cache, pre-fetch, and query that data. This all contributes to providing an interactive experience when querying and visualizing data.

The UltraWarm preview is now available to all customers in the US East (N. Virginia) and US West (Oregon) Regions. The UltraWarm tier is available with a pay-as-you-go pricing model, charging for the instance hours for your node, and utilized storage. The UltraWarm preview can be enabled on new Amazon Elasticsearch Service version 6.8 domains. To learn more, visit the technical documentation.

— Steve


Automate OS Image Build Pipelines with EC2 Image Builder

Post Syndicated from Steve Roberts original

Earlier in my career, I can recall being assigned the task of creating and maintaining operating system (OS) images for use by my development team. This was a time-consuming process, sometimes error-prone, needing me to manually re-create and re-snapshot images frequently. As I’m sure you can imagine, it also involved a significant amount of manual testing!

Today, customers still need to keep their images up to date and they do so either by manually updating and snapshotting VMs, or they have teams that build automation scripts to maintain the images, both of which can still be time consuming, resource intensive, and error-prone. I’m excited to announce the availability of EC2 Image Builder, a service that makes it easier and faster to build and maintain secure OS images for Windows Server and Amazon Linux 2, using automated build pipelines. The images created by EC2 Image Builder can be used with Amazon Elastic Compute Cloud (EC2) and on-premises, and can be secured and hardened to help comply with applicable InfoSec regulations. AWS provides security hardening policies that you can use as a starting point to meet the “Security Technical Implementation Guide (STIG)” standard needed to operate in regulated industries.

The pipelines that you can configure for EC2 Image Builder include the image recipe, infrastructure configuration, distribution, and test settings, to produce the resulting images. This includes the ability to automatically provision images as new software updates, including security patches, become available. As new images are created by the pipelines, you can additionally configure automated tests to be run to validate the image, before then distributing it to AWS regions that you specify. EC2 Image Builder can be used with EC2 VM Import/Export to build images in multiple formats for on-premises use, including VMDK, VHDX, and OVF. When testing you can use a combination of AWS-provided tests and custom tests that you have authored yourself.

Let’s take a look at how to get started using EC2 Image Builder.

Creating an OS Image Build Pipeline
From the console homepage I can quickly get started by clicking Create image pipeline. Here, I’m going to construct a pipeline that will build a custom Amazon Linux 2 image. The first step is to define the recipe which involves selecting the source image to start from, the build components to apply to the image being created, and the tests to be run.

Starting with the source image, I’m going to select a managed image provided by EC2 Image Builder. Note that I can also choose other images that either I have created, or that have been shared with me, or specify a custom AMI ID.

Next I select the build components to include in the recipe – in other words, the software I want to be installed onto the new image. From within the wizard I have the option to create a new build component by clicking Create build component. Build components have a name (and optional description), a target operating system, an optional AWS Key Management Service (KMS) key to encrypt the component, and a YAML document that specifies the set of customization steps for the component. Build components can also be versioned, so I have a lot of flexibility in customizing the software to apply to my image. I can create, and select, multiple build components and don’t have to do all my customization from one component.

For this post however I’ve clicked Browse build components and selected some Amazon-provided components for Amazon Corretto, Python 3 and PowerShell Core.

The final step for the recipe is to select tests to be applied to the image to validate it. Just as with build components, I can create and specify tests within the wizard, and I have the same capabilities for defining a test as I do a build component. Again though, I’m going to keep this simple and click Browse tests to select an Amazon-provided test that the image will reboot successfully (note that I can also select multiple tests).

That completes my recipe, so I click Next and start to define my pipeline. First, I give the pipeline a name and also select an AWS Identity and Access Management (IAM) role to associate with the EC2 instance to build the new image. EC2 Image Builder will use this role to create Amazon Elastic Compute Cloud (EC2) instances in my account to perform the customization and testing steps. Pipeline builds can be performed manually, or I can elect to run them on a schedule. I have the flexibility to specify my schedule using simple Day/Week/Month period and time-of-day selectors, or I can use a CRON expression.

I selected a managed IAM policy (EC2InstanceProfileForImageBuilder) with just enough permissions to use common AWS-provided build components and and run tests. When you start to use Image Builder yourself, you will need to set up a role that has enough permissions to perform your customizations, run your tests, and write troubleshooting logs to S3. As a starting point for setting up the proper permissions, I recommend that you attach the AmazonSSMManagedInstanceCore IAM policy to the IAM role attached to the instance.

Finally for the pipeline I can optionally specify some settings for the infrastructure that will be launched on my behalf, for example the size of instance type used when customizing my image, and an Amazon Simple Notification Service (SNS) topic that notifications can be forwarded to. I can also take control of Amazon Virtual Private Cloud related settings should I wish.

If the operating system of the image I am building is associated with a license, I can specify that next (or create a new license configuration on-the-fly), along with a name for my new image and also the AWS regions into which the new image will be shared, either publicly or privately.

Clicking Review, I can review all of my settings and finally click Create Pipeline to complete the process.

Even though when I configured my pipeline I asked for it to run daily at 06:00 hours UTC, I can still run it whenever I wish. Selecting the pipeline, I click Actions and then Run pipeline.

Once the build has completed, the AMI will be ready to launch from the Amazon EC2 console!

Thinking back to my earlier career years and the tasks assigned to me, this would have saved me so much time and effort! EC2 Image Builder is provided at no cost to customers and is available in all commercial AWS Regions. You are charged only for the underlying AWS resources that are used to create, store, and share the images.

— Steve

A New, Simplified, Bring-Your-Own-License Experience for Microsoft Windows Server and SQL Server

Post Syndicated from Steve Roberts original

Customers have asked us to provide an easier way to bring, and manage, their existing licenses for Microsoft Windows Server and SQL Server to AWS. Today we are excited to announce a new, simpler, bring-your-own-license (BYOL) experience.

When launching Windows Server or SQL Server instances, customers can use licenses from AWS with a pay-as-you-go model or they can bring their own existing licenses. When using software licenses obtained from AWS, customers get a fully-compliant, pay-as-you-go licensing model that doesn’t require managing complex licensing terms and conditions. The new BYOL experience launched today enables those customers who want to use their existing Windows Server or SQL Server licenses to seamlessly create virtual machines in EC2, while AWS takes care of managing their licenses to help ensure compliance to licensing rules specified by the customer.

Previously, when bringing their own server-bound licenses, customers needed to write additional automation to manage capacity, and ensure effective utilization of the dedicated hosts that are required for BYOL. This process made managing hosts non-EC2 like, and certainly unlike the simple and easy-to-use experience when using licenses provided by AWS. The new BYOL experience simplifies the host management experience for customers, by automating the key host management tasks such as allocation and release of hosts, managing host capacity, and enabling capabilities such as auto scaling and auto recovery of hosts. As a result, you can simply create a BYOL instance on Dedicated Host the way you create any other EC2 instance.

Let’s take a walk-through of the new, simplified, experience to bring a licensed image to the AWS Cloud. In this post I am going to use a Windows Server license but the experience is the same if I wanted to bring Windows Server and SQL Server licenses.

Setting up License Management
With my licensed image imported and available as a private AMI in my account in Amazon Elastic Compute Cloud (EC2), it’s time to head to the AWS License Manager console and get started by first creating a license configuration, which I do by clicking Create license configuration from the License Configurations page. I fill out a name for the configuration, an optional description, and select the License type, which in this case I set to Cores. Note that I can set an upper limit for cores, which we recommend customers set according to the number of licenses they are bringing, in addition to enabling Enforce license limit. For this walk-through I’m going to leave both unset. Clicking Submit creates the new license configuration, which will then be shown on the License configurations page.

The next step is to associate the AMI containing my licensed operating system image with the configuration. Selecting the configuration I just created, from Actions I select Associate AMI. This takes me to a page listing my custom images, where I select the image I want and click Associate (note that I could select and associate multiple images, for example I could also add my Windows Server and SQL Server image also shown below).

With my license configuration setup completed for my Windows Server – don’t forget, I could also have set up a configuration to bring my Windows Server with SQL Server licenses too – I next need to create a Host resource group and associate it with the license configuration. A Host resource group is a collection of Dedicated Hosts that can be managed together as a single entity using your specified preferences. I start by selecting Host resource groups from the navigation panel, and then clicking Create host resource group.

I give my Host resource group a name, and select the license configurations I want to associate with this group (I can select more than one), then click Create. Note the options for EC2 Dedicated Host management. Leaving these selected means I do not need to manage capacity and Dedicated Hosts myself. AWS License Manager will automatically manage my hosts for me when I launch instances into a Host resource group, and also ensure I stay within my license count limits if so configured.

That’s it for setting up my license configuration and Host resource group, so now I can head to the Amazon EC2 console and launch some instances. Before I do, I want to bring to your attention one more feature of the new BYOL experience – integration with AWS Organizations. If my account is the master account for an Organization, an additional option is shown when creating Host resource groups – Share host resource group with all my member accounts. As the name suggests, selecting this option will cause the Host resource group to be shared with the member accounts in my Organization. When member accounts launch instances, the hosts get allocated automatically in my master account. This enables me to use the same host across multiple accounts, and not require hosts in every member account, increasing host utilization.

Launching an Automatically Managed BYOL Image
In the Amazon EC2 console I click Launch instance and select the AMI that contains my licensed version of Windows Server (or Windows Server with SQL Server, if that is what I am bringing), on the My AMIs tab in the launch wizard, then select the instance type I want to launch, before arriving at the Configure Instance Details page.

In the launch wizard, as I have selected an image associated with a license configuration, Tenancy is already preset to the required Dedicated host – Launch this instance on a Dedicated host setting. I toggle the option under Host resource group to Launch instance into a host resource group and can then select the appropriate Host resource group. In the screenshot below I’ve selected my group, however, if I have only one group, EC2 will automatically launch the instance in the mapped Host resource group based on the license configuration attached, without me needing to select it in the launch wizard.

In order to maximize utilization, the new experience also provides for heterogeneous instance support on Dedicated Hosts. What that means is if I launch different sized instances of the same family, an attempt is made to place them together on the same host (previously you could only create instances of the same size and family on one host).

At this point, I can click Review and launch, although you likely have more settings you want to specify for storage, tags, and security group, before proceeding. Once the instance(s) have launched, on selecting an instance in the console the Details tab shows me it has been associated with a Host resource group. I can optionally also click the Host ID to view details of the host.

BYOL, Auto Scaling and Dedicated Hosts
Automatic management of instances launched through Auto Scaling onto Dedicated Hosts is an additional benefit of the new BYOL support, and is easy to set up and use. In the launch wizard shown above, I could have created and specified a new Auto Scaling group, but instead I’ve chosen to create a Launch Template and used that to define settings for my Auto Scaling group. In the snapshots below, you can see that all I need do in the launch template is select my licensed image and then, under Advanced Details, my license configuration.

Next I simply specify the template when creating an Auto Scaling group, and I’m done! As new instances are created in the group during launch, or a scaling event, they will be mapped automatically onto a Dedicated Host, something that has not been possible when using BYOL before today.

Availability of the New BYOL Experience

The new BYOL experience is available for your eligible Windows Server and SQL Server licenses. As a reminder, customers can also bring their own SQL Server licenses with Active Software Assurance to default (shared) tenant EC2 through License Mobility benefits. This new experience gives our SQL Server customers another way to bring their existing licenses. For more information on licensing, please visit the Windows on AWS FAQ. Additional information on Microsoft licensing options on AWS can be found here. Be sure to also check out AWS License Manager, to learn more about management preferences.

The simpler BYOL experience is available now to customers in the US-East (Northern Virginia) and US-West (Oregon) regions, and will be available in other regions in the coming days.

— Steve

The Next Evolution in AWS Single Sign-On

Post Syndicated from Steve Roberts original

Efficiently managing user identities at scale requires new solutions that connect the multiple identity sources that many organizations use today. Customers want to establish a single identity and access strategy across their own apps, 3rd party apps (SaaS), and AWS cloud environments.

Today we announced the next evolution of AWS Single Sign-On, enabling enterprises that use Azure AD to leverage their existing identity store with AWS Single Sign-On. Additionally, automatic synchronization of user identities, and groups, from Azure AD is also supported. Users can now sign into the multiple accounts and applications that make up their AWS environments using their existing Azure AD identity – no need to remember additional usernames and passwords – and they will use the sign-in experience they are familiar with. Additionally, administrators can also focus on managing a single source of truth for user identities in Azure AD while having the convenience of configuring access to all AWS accounts and apps centrally.

Let’s imagine that I am an administrator for an enterprise that already uses Azure AD for user identities and now wants to enable simple and easy use of our AWS environments for my users using their existing identities. I do not want to duplicate my Azure AD group and user membership setup by hand in AWS Single Sign-On, and maintain two identity systems, so I am also going to enable automatic synchronization. My users will sign in to the AWS environments using the experience they are already familiar with in Azure AD. You can read more about enabling single sign-on to applications in Azure AD here. To learn more about managing permissions to AWS accounts and applications, refer to the AWS Single Sign-On User Guide.

Connecting Azure AD as an Identity Source for AWS Single Sign-On
My first step is to connect Azure AD with AWS Single Sign-On. First, I sign into the Azure Portal for my account and navigate to the Azure Active Directory dashboard. From the left-hand navigation panel I then select Enterprise Applications.

Next, I click + New application, and select Non-gallery application. In the Add your own application page that opens, I enter AWS SSO in the Name field – you can use whatever name you like – and click Add. After a few seconds my new application will be created and I am redirected to the settings overview for the application.

This is where I will configure the settings to enable single sign-on and exchange federation metadata between Azure AD and AWS Single Sign-On. I select Single sign-on from the navigation panel and then click the SAML option. From the settings page that then opens, I need to download the Federation Metadata XML file, which is shown in the SAML Signing Certificate panel. This file, which will be named AWS SSO.xml by default, will be used to configure AWS Single Sign-On in the next steps.

Having downloaded the file, I open another browser tab (I leave the Azure AD tab open, as I need to come back to it), and sign into the AWS Single Sign-On Console. From the dashboard home, I click the Enable AWS SSO button. After a few seconds, my account is enabled for single sign-on and I can proceed to configure the connection with Azure AD.

Clicking Settings in the navigation panel, I first set the Identity source by clicking the Change link and selecting External identity provider from the list of options. I now have two things to do. First, I need to download the AWS SSO SAML metadata file using the link on the page – I will need this back in the Azure AD portal. Second, I browse to and select the XML file I downloaded from Azure AD in the Identity provider metadata section.

I click Next: Review, enter CONFIRM in the provided field, and finally click Change identity source to complete the AWS Single Sign-On side of the process.

Returning to the tab I left open to my Azure AD Set up Single Sign-On with SAML settings page, I click the Upload metadata file button at the top of the page, navigate to and select the file I downloaded from the AWS SSO SAML metadata link in the AWS Single Sign-On settings and then, in the Basic SAML Configuration fly-out that opens, click Save. At this point, if I already have a user account in AWS Single Sign-On with a username that matches to a user in Azure AD, I can click the Test button to verify my setup.

Customers with a relatively small number of users may prefer to continue maintaining the user accounts in AWS Single Sign-On rather than rely on automatic provisioning and can stop here, as it is possible to use just the sign-in aspect. We do however recommend enabling automatic provisioning for convenience. New users you add to an Azure AD group automatically get access with no additional action needed, making it convenient for administration and productive for the end user. Users who get removed from a group in Azure AD will automatically lose access to associated permissions in AWS Single Sign-On, a security benefit.

Enabling Automatic Provisioning
Now that my Azure AD is configured for single sign-on for my users to connect using AWS Single Sign-On I’m going to enable automatic provisioning of user accounts. As new accounts are added to my Azure AD, and assigned to the AWS SSO application I created, when those users sign into AWS, a corresponding AWS Single Sign-On user will be created automatically. As an administrator, I do not need to do any work to configure a corresponding account in AWS to map to the Azure AD user.

From the AWS Single Sign-On Console I navigate to Settings and then click the Enable identity synchronization link. This opens a dialog containing the values for the SCIM endpoint and an OAuth bearer Access token (hidden by default). I need both of these values to use in the Azure AD application settings so either make note of both values, or use multiple browser tabs and copy/paste as you go.

Switching back to my Azure AD browser tab and the AWS SSO application, I click Provisioning in the navigation panel and set Provisioning Mode to Automatic. This triggers display of the fields I need to complete with values from the dialog in the AWS Single Sign-On Console. First, I paste the SCIM endpoint value into the Tenant URL field. Then I paste the Access token into the Secret Token field.

I complete the settings by entering a value for Notification Email, and opt in to receive an email when provisioning failures occur, then click the Test Connection button to verify everything is working as expected. Assuming everything is good, I click Save in the page toolbar and then just a couple of small steps remain to configure mapping of attributes and I am done!

Expanding Mappings, I click the Synchronize Azure Active Directory Users to customappsso link (your link may read ‘…to AWS SSO‘). That takes me to a page where I control attribute mappings. In that section I deleted the facsimileTelephoneNumber and mobile attributes as I won’t be using them, and I click on and edit the mailNickname attribute, changing the Source attribute to be objectId. I click Save to return to the Settings screen and I have one more step remaining, to turn on synchronization, which I do by clicking On in Provisioning Status and clicking Save one more time.

Note that the first sync will take longer than subsequent ones, which happen around every 40 minutes. To check progress I can either view the Synchronization Details or the Audit Logs in Azure AD, or in the AWS Single Sign-On Console I can select Users from the navigation panel.

I Configured Single Sign-On, Now What?
Azure AD will now be my single source of truth for my user identities and their assignment into groups, and periodic synchronization will automatically create corresponding user identities in AWS Single Sign-On, enabling my users to sign into their AWS accounts and applications with their Azure AD credentials and experience, and not have to remember an additional username and password. However, as things stand my users will only have access to sign in. To manage permissions in terms of what they can access once signed into AWS, I need to set up permissions in AWS Single Sign-On which I do using the AWS Single Sign-On Console. AWS Single Sign-On uses a concept of permission sets for assignments. A permission set is essentially a standard role definition to which I attach AWS Identity and Access Management (IAM) policies. Once I define a permission set, I then assign a group, or user, to the permission set in a specified account. AWS Single Sign-On then creates the underlying AWS Identity and Access Management (IAM) role in the designated account, including all the right information that grants access to that group or user. You can read more about permissions sets in the AWS Single Sign-On User Guide.

In the screenshots below, you can see the effect of automatic synchronization. In Azure AD I have created three groups, and assigned my user account into two of them (for the sake of this blog post). Switching to the AWS Single Sign-On Console once synchronization completes, I see the three groups have been automatically created, with my user account assigned into the ones I chose.

Using Permission Sets, available from the AWS Accounts and Applications links in the navigation panel, I can associate one or more access control policies (custom policies I have created, or AWS Identity and Access Management (IAM) managed policies), to both groups and users, thus controlling permissions once users sign in. Sign in is accomplished using the familiar Azure AD experience, and users will be able to choose the account(s) and role(s) to assume. Sign in can also be done using the AWS console and CLI. Details on using single sign-on with the version 2 of the AWS Command Line Interface (CLI) is detailed in this blog post.

In this post I showed how you can take advantage of the new AWS Single Sign-On capabilities to to link Azure AD user identity into AWS accounts and applications centrally for single sign-on, and make use of the new automatic provisioning support to reduce complexity when managing and using identities. Administrators can now use a single source of truth for managing their users, and users no longer need to manage an additional identity and password to sign into their AWS accounts and applications.

This next evolution is now available to all users in all AWS Single Sign-On supported regions. You can check the regional availability of AWS Single Sign-On here.

— Steve

Visualize and Monitor Highly Distributed Applications with Amazon CloudWatch ServiceLens

Post Syndicated from Steve Roberts original

Increasingly distributed applications, with thousands of metrics and terabytes of logs, can be a challenge to visualize and monitor. Gaining an end-to-end insight of the applications and their dependencies to enable rapid pinpointing of performance bottlenecks, operational issues, and customer impact quite often requires the use of multiple dedicated tools each presenting their own particular facet of information. This in turn leads to more complex data ingestion, manual stitching together of the various insights to determine overall performance, and increased costs from maintaining multiple solutions.

Amazon CloudWatch ServiceLens, announced today, is a new fully managed observability solution to help with visualizing and analyzing the health, performance, and availability of highly distributed applications, including those with dependencies on serverless and container-based technologies, all in one place. By enabling you to easily isolate endpoints and resources that are experiencing issues, together with analysis of correlated metrics, logs, and application traces, CloudWatch ServiceLens helps reduce Mean Time to Resolution (MTTR) by consolidating all of this data in a single place using a service map. From this map you can understand the relationships and dependencies within your applications, and dive deep into the various logs, metrics, and traces from a single tool to help you quickly isolate faults. Crucial time spent correlating metrics, logs, and trace data from across various tools is saved, thus reducing any downtime incurred by end users.

Getting Started with Amazon CloudWatch ServiceLens
Let’s see how we can take advantage of Amazon CloudWatch ServiceLens to diagnose the root cause of an alarm triggered from an application. My sample application reads and writes transaction data to a Amazon DynamoDB table using AWS Lambda functions. An Amazon API Gateway is my application’s front-end, with resources for GET and PUT requests, directing traffic to the corresponding GET and PUT lambda functions. The API Gateway resources and the Lambda functions have AWS X-Ray tracing enabled, and the API calls to DynamoDB from within the Lambda functions are wrapped using the AWS X-Ray SDK. You can read more about how to instrument your code, and work with AWS X-Ray, in the Developer Guide.

An error condition has triggered an alarm for my application, so my first stop is the Amazon CloudWatch Console, where I click the Alarm link. I can see that there is some issue with availability with one or more of my API Gateway resources.

Let’s drill down to see what might be going on. First I want to get an overview of the running application so I click Service Map under ServiceLens in the left-hand navigation panel. The map displays nodes representing the distributed resources in my application. The relative size of the nodes represents the amount of request traffic that each is receiving, as does the thickness of the links between them. I can toggle the map between showing Requests modes or Latency mode. Using the same drop-down I can also toggle the option to change relative sizing of the nodes. The data shown for Request mode or Latency mode helps me isolate nodes that I need to triage first. Clicking View connections can also be used to aid in the process, since it helps me understand incoming and outgoing calls, and their impact on the individual nodes.

I’ve closed the map legend in the screenshot so we can get a good look at all the nodes, for reference here it is below.

From the map I can immediately see that my front-end gateway is the source of the triggered alarm. The red indicator on the node is showing me that there are 5xx faults associated with the resource, and the length of the indicator relative to the circumference gives me some idea of how many requests are faulting compared to successful requests. Secondly, I can see that the Lambda functions that are handling PUT requests through the API are showing 4xx errors. Finally I can see a purple indicator on the DynamoDB table, indicating throttling is occurring. At this point I have a pretty good idea of what might be happening, but let’s dig a little deeper to see what CloudWatch ServiceLens can help me prove.

In addition to Map view, I can also toggle List view. This gives me at-a-glance information on average latency, faults, and requests/min for all nodes and is specifically ordered by default to show the “worst” node first, using a sort order of descending by fault rate – descending by number of alarms in alarm – ascending by Node name.

Returning to Map view, hovering my mouse over the node representing my API front-end also gives me similar insight into traffic and faulting request percentage, specific to the node.

To see even more data, for any node, clicking it will open a drawer below the map containing graphed data for that resource. In the screenshot below I have clicked on the ApiGatewayPutLambdaFunction node.

The drawer, for each resource, enables me to jump to view logs for the resource (View logs), traces (View traces), or a dashboard (View dashboard). Below, I have clicked View dashboard for the same Lambda function. Scrolling through the data presented for that resource, I note that the duration of execution is not high, while all invokes are going into error in tandem.

Returning to the API front-end that is showing the alarm, I’d like to take look at the request traces so I click the node to open the drawer, then click View traces. I already know from the map that 5xx and 4xx status codes are being generated in the code paths selected by PUT requests coming into my application so I switch Filter type to be Status code, then select both 502 and 504 entries in the list, finally clicking Add to filter. The Traces view switches to show me the traces that resulted in those error codes, the response time distribution and a set of traces.

Ok, now we’re getting close! Clicking the first trace ID, I get a wealth of data including exception messages about that request – more than I can show in a single screenshot! For example, I can see the timelines of each segment that was traced as part of the request.

Scrolling down further, I can view exception details (below this I can also see log messages specific to the trace too) – and here lays my answer, confirming the throttling indicator that I saw in the original map. I can also see this exception message in the log data specific to the trace, shown at the foot of the page. Previously, I would have had to scan through logs for the entire application to hopefully spot this message, being able to drill down from the map is a significant time saver.

Now I know how to fix the issue and get the alarm resolved – increase the write provisioning for the table!

In conjunction with CloudWatch ServiceLens, Amazon CloudWatch has also launched a preview of CloudWatch Synthetics that helps to monitor endpoints using canaries that run 24×7, 365 days a year, so that I am quickly alerted of issues that my customers are facing. These are also visualized on the Service Map and just as I did above, I can drill down to the traces to view transactions that originated from a canary. The faster I can dive deep into a consolidated view of an operational failure or an alarm, the faster I can root cause the issue and help reduce time to resolution and mitigate the customer impact.

Amazon CloudWatch ServiceLens is available now in all commercial AWS Regions.

— Steve

Safe Deployment of Application Configuration Settings With AWS AppConfig

Post Syndicated from Steve Roberts original

A few years ago, we identified a need for an internal service to enable us to make configuration changes faster than traditional code deployments, but with the same operational scrutiny as code changes. So, we built a tool to address this need and now most teams across AWS, Retail, Kindle and Alexa use this configuration deployment system to perform changes dynamically in seconds, rather than minutes. The ability to deploy only configuration changes, separate from code, means we do not have to restart the applications or services that use the configuration, and changes take effect immediately. This service is used thousands of times a day inside of Amazon and AWS.

A common question we receive from customers is how they can operate like us, and so we decided to take this internal tooling and externalize it for our customers to use. Today, we announced AWS AppConfig, a feature of AWS Systems Manager. AWS AppConfig enables customers to quickly rollout application configuration changes, independent of code, across any size application hosted on Amazon Elastic Compute Cloud (EC2) instances, Containers, and Serverless applications and functions.

Configurations can be created and updated using the API or Console, and you can have the changes validated prior to deployment using either a defined schema template or an AWS Lambda function. AWS AppConfig also includes automated safety controls to monitor the deployment of the configuration changes, and to rollback to the previous configuration if issues occur. Deployments of configuration updates are available immediately to your running application, which uses a simple API to periodically poll and retrieve the latest available configuration.

Managing Application Configuration Settings with AWS AppConfig
Using AWS AppConfig to create and manage configuration settings for applications is a simple process. First I create an Application, and then for that application I define one or more Environments, Configuration profiles, and Deployment strategies. An environment represents a logical deployment group, such as Beta or Production environments, or it might be subcomponents of an application, for example Web, Mobile, and Back-end components. I can configure Amazon CloudWatch alarms for each environment, which will be monitored by AWS AppConfig and trigger a rollback if the alarm fires during deployment. A configuration profile defines the source of the configuration data to deploy, together with optional Validators that will ensure your configuration data is correct prior to deployment. A deployment strategy defines how the deployment is to be performed.

To get started, I go to the AWS Systems Manager dashboard and select AppConfig from the navigation panel. Clicking Create configuration data takes me to a page where I specify a name for my application, together with an optional description, and I can also apply tags to the application if I wish.

Once I have created my application, I am taken to a page with Environments and Configuration Profiles tabs. Let’s start with creating an environment to target my production environment for the application. With the Environments tab selected, I click Create environment. On the resulting page, I give my environment a name, and then, under Monitors, select Enable rollback on Cloudwatch alarms. I then supply an IAM role that enables AWS AppConfig to monitor the selected alarms during deployments, and click Add. Details about the permissions needed for the role can be found here.

Next, I set up a configuration profile. Returning to the details view for my application, I select the Configuration Profiles tab, and click Create configuration profile. The first details needed are a name for the configuration and a service role that AWS AppConfig will assume to access the configuration. I can choose to create a new role, or choose an existing role.

Clicking Next, I then select the source of the configuration data that will be deployed. I can choose from a Systems Manager Document, or a parameter in Parameter Store. For this walk-through, I’m going to select a parameter that when set to true, will enable the functionality related to that feature for users of this sample application.

Next, I can optionally add one or more validators to the profile, so that AWS AppConfig can verify that when I attempt to deploy an updated setting, the value is correct and not going to cause an application failure. Here, I am going to use a JSON schema, so I am using a syntactic check. If I want to run code to perform validation, a semantic check, I would configure a Lambda function. Once I have entered the schema, I click Create configuration profile to complete the process.

One more task remains, to configure a Deployment strategy that will govern the rollout. I can do this from the AWS AppConfig dashboard by selecting the Deployment Strategies tab and clicking Create deployment strategy. Here I have configured a strategy that will cause AWS AppConfig to deploy to one-fifth (20%) of my instances hosting the application at a time. Monitoring will be performed during deployments and a ‘bake’ time which occurs when all deployments complete. During deployments and the bake time, if anything goes wrong, the alarms associated with the environment will trigger and a rollback will occur. You can read more about the options you can set for a strategy here. Clicking Create deployment strategy completes this step and makes the strategy available for selection during deployment.

With an environment and a configuration profile configured for my application, and a deployment strategy in place – I can add more of each – I am ready to start a deployment. Navigating to my configuration profile, I click Start deployment.

I select the environment I want to deploy to, the version of the parameter that contains the configuration change I want to deploy, the deployment strategy, and I can enter an optional description. The parameter version I selected is 2. Version 1 of the parameter had the value { “featureX”: false }. Version 2 of the parameter has the value { “featureX”: true }, enabling the functionality associated with that feature for my users, as soon as it is deployed.

Clicking Start deployment starts the validation and deployment process. I know that the value of version 2 of the parameter is valid, so deployment will proceed after validation. As soon as the deployment becomes available to a target, in its next poll for configuration, the target will receive the updated configuration and the new features associated with featureX will be safely enabled for my users!

In this post I presented a very simple experience based around a feature toggle, enabling me to be able to instantly turn on features that might require a timely rollout (for example a new product launch or announcement) . AWS AppConfig can however be used for many different use cases, here are a couple more examples to consider:

  • A/B Testing: perform experiments on which versions of an application earns more revenue.
  • User Membership: allow Premium Subscribers to access an application’s paid content.

Note that the targets receiving deployments from AWS AppConfig do not not need to be configured with the Systems Manager Agent, or have an AWS Identity and Access Management (IAM) instance profile, which is required by other Systems Manager capabilities. This enables me to use AWS AppConfig with managed and unmanaged instances. You can read more about AWS AppConfig in the User Guide, and pricing information is available here. AWS AppConfig is available now to customers in all commercial AWS regions.

— Steve

Using Spatial Data with Amazon Redshift

Post Syndicated from Steve Roberts original

Today, Amazon Redshift announced support for a new native data type called GEOMETRY. This new type enables ingestion, storage, and queries against two-dimensional geographic data, together with the ability to apply spatial functions to that data. Geographic data (also known as georeferenced data) refers to data that has some association with a location relative to Earth. Coordinates, elevation, addresses, city names, zip (or postal) codes, administrative and socioeconomic boundaries are all examples of geographic data.

The GEOMETRY type enables us to easily work with coordinates such as latitude and longitude in our table columns, which can then be converted or combined with other types of geographic data using spatial functions. The type is abstract, meaning it cannot be directly instantiated, and polymorphic. The actual types supported for this data (and which will be used in table columns) are points, linestrings, polygons, multipoints, multilinestrings, multipolygons, and geometry collections. In addition to creating GEOMETRY-typed data columns in tables the new support also enables ingestion of geographic data from delimited text files using the existing COPY command. The data in the files is expected to be in hexadecimal Extended Well-Known Binary (EWKB) format which is a standard for representing geographic data.

To show the new type in action I imagined a scenario where I am working as a personal tour coordinator based in Berlin, Germany, and my client has supplied me with a list of attractions that they want to visit. My task is to locate accommodation for this client that is reasonably central to the set of attractions, and within a certain budget. Geographic data is ideal for solving this scenario. Firstly, the set of points representing the attractions combine to form one or more polygons which I can use to restrict my search for accommodation. In a single query I can then join the data representing those polygons with data representing a set of accommodations to arrive at the results. This spatial query is actually quite expensive in CPU terms yet Redshift is able to execute the query in less that one second.

Sample Scenario Data
To show my scenario in action I needed to first source various geographic data related to Berlin. Firstly I obtained the addresses, and latitude/longitude coordinates, of a variety of attractions in the city using several ‘top X things to see’ travel websites. For accommodation I used Airbnb data, licensed under the Creative Commons 1.0 Universal “Public Domain Dedication” from I then added to this zip code data for the city, licensed under Creative Commons Attribution 3.0 Germany (CC BY 3.0 DE). The provider for this data is Amt für Statistik Berlin-Brandenburg.

Any good tour coordinator would of course have a web site or application with an interactive map so as to be able to show clients the locations of the accommodation that matched their criteria. In real life, I’m not a tour coordinator (outside of my family!) so for this post I’m going to focus solely on the back-end processes – the loading of the data, and the eventual query to satisfy our client’s request using the Redshift console.

Creating a Redshift Cluster
My first task is to load the various sample data sources into database tables in a Redshift cluster. To do this I go to the Redshift console dashboard and select Create cluster. This starts a wizard that walks me through the process of setting up a new cluster, starting with the type and number of nodes that I want to create.

In Cluster details I fill out a name for my new cluster, set a password for the master user, and select an AWS Identity and Access Management (IAM) role that will give permission for Redshift to access one of my buckets in Amazon Simple Storage Service (S3) in read-only mode when I come to load my sample data later. The new cluster will be created in my default Amazon Virtual Private Cloud for the region, and I also opted to use the defaults for node types and number of nodes. You can read more about available options for creating clusters in the Management Guide. Finally I click Create cluster to start the process, which will take just a few minutes.

Loading the Sample Data
With the cluster ready to use I can load the sample data into my database, so I head to the Query editor and using the pop-up, connect to my default database for the cluster.

My sample data will be sourced from delimited text files that I’ve uploaded as private objects to an S3 bucket and loaded into three tables. The first, accommodations, will hold the Airbnb data. The second, zipcodes, will hold the zip or postal codes for the city. The final table, attractions, will hold the coordinates of the city attractions that my client can choose from. To create and load the accommodations data I paste the following statements into tabs in the query editor, one at a time, and run them. Note that schemas in databases have access control semantics and the public prefix shown on the table names below simply means I am referencing the public schema, accessible to all users, for the database in use.

To create the accommodations table I use:

CREATE TABLE public.accommodations (
  shape GEOMETRY,
  name VARCHAR(100),
  host_name VARCHAR(100),
  neighbourhood_group VARCHAR(100),
  neighbourhood VARCHAR(100),
  room_type VARCHAR(100),
  price SMALLINT,
  minimum_nights SMALLINT,
  number_of_reviews SMALLINT,
  last_review DATE,
  reviews_per_month NUMERIC(8,2),
  calculated_host_listings_count SMALLINT,
  availability_365 SMALLINT

To load the data from S3:

COPY public.accommodations
FROM 's3://my-bucket-name/redshift-gis/accommodations.csv'
CREDENTIALS 'aws_iam_role=arn:aws:iam::123456789012:role/RedshiftDemoRole';

Next, I repeat the process for the zipcodes table.

CREATE TABLE public.zipcode (
  ogc_field INTEGER,
  wkb_geometry GEOMETRY,
  gml_id VARCHAR,
  spatial_name VARCHAR,
  spatial_alias VARCHAR,
  spatial_type VARCHAR
COPY public.zipcode
FROM 's3://my-bucket-name/redshift-gis/zipcode.csv'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftDemoRole';

And finally I create the attractions table and load data into it.

CREATE TABLE public.berlin_attractions (
  name VARCHAR,
  address VARCHAR,
  gps_lat VARCHAR,
  gps_lon VARCHAR
COPY public.berlin_attractions
FROM 's3://my-bucket-name/redshift-gis/berlin-attraction-coordinates.txt'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftDemoRole';

Finding Somewhere to Stay!
With the data loaded, I can now put on my travel coordinator hat and select some properties for my client to consider for their stay in Berlin! Remember, in the real world this would likely be surfaced from a web or other application for the client. I’m simply going to make use of the query editor again.

My client has decided they want a trip to focus on the city museums and they have a budget of 200 EUR per night for accommodation. Opening a new tab in the editor, I paste in and run the following query.

WITH museums(name,loc) AS
(SELECT name, ST_SetSRID(ST_Point(lon,lat),4326) FROM public.berlin_attractions
WHERE name LIKE '%Museum%')
SELECT,a.price,avg(ST_DistanceSphere(m.loc,a.shape)) AS avg_distance
FROM museums m,public.accommodations a
WHERE a.price <= 200 GROUP BY,a.price ORDER BY avg_distance

The query finds the accommodation(s) that are “best located” to visit all the museums, and whose price is within the client’s budget. Here “best located” is defined as having the smallest average distance from all the selected museums. In the query you can see some of the available spatial functions, ST_SetSRID and ST_Point, operating on the latitude and longitude GEOMETRY columns for the attractions, and ST_DistanceSphere to determine distance.

This yields the following results.

Wrap a web or native application front-end around this and we have a new geographic data-based application that we can use to delight clients who have an idea of what they want to see in the city and also want convenient and in-budget accommodation best placed to enable that!

Let’s also consider another scenario. Imagine I have a client who wants to stay in the center of Berlin but isn’t sure what attractions or accommodations are present in the central district, and has a budget of 150 EUR per night. How can we answer that question? First we need the coordinates of what we might consider to be the center of Berlin – latitude 52.516667, longitude 13.388889. Using the zipcode table we can convert this coordinate location to a polygon enclosing that region of the city. Our query must then get all attractions within that polygon, plus all accommodations (within budget), ordered by average distance from the attractions. Here’s the query:

WITH center(geom) AS
(SELECT wkb_geometry FROM zipcode
  WHERE ST_Within(ST_SetSRID(ST_Point(13.388889, 52.516667), 4326), wkb_geometry)),
pois(name,loc) AS
(SELECT name, ST_SetSRID(ST_Point(lon,lat),4326) FROM public.berlin_attractions,center
  WHERE ST_Within(ST_SetSRID(ST_Point(lon,lat),4326), center.geom))
  AS avg_distance, LISTAGG(, ';') as pois
FROM pois p,public.accommodations a
WHERE a.price <= 150 GROUP BY,a.price ORDER BY avg_distance

When I run this in the query editor, I get the following results. You can see the list of attractions in the area represented by the zipcode in the pois column.

So there we have some scenarios for making use of geographic data in Amazon Redshift using the new GEOMETRY type and associated spatial functions, and I’m sure there are many more! The new type and functions are available now in all AWS Regions to all customers at no additional cost.

— Steve

Accelerate SQL Server Always On Deployments with AWS Launch Wizard

Post Syndicated from Steve Roberts original

Customers sometimes tell us that while they are experts in their domain, their unfamiliarity with the cloud can make getting started more challenging and take more time. They want to be able to quickly and easily deploy enterprise applications on AWS without needing prior tribal knowledge of the AWS platform and best practices, so as to accelerate their journey to the cloud.

Announcing AWS Launch Wizard for SQL Server
AWS Launch Wizard for SQL Server is a simple, intuitive and free to use wizard-based experience that enables quick and easy deployment of high availability SQL solutions on AWS. The wizard walks you through an end-to-end deployment experience of Always On Availability Groups using prescriptive guidance. By answering a few high-level questions about the application such as required performance characteristics the wizard will then take care of identifying, provisioning, and configuring matching AWS resources such as Amazon Elastic Compute Cloud (EC2) instances, Amazon Elastic Block Store (EBS) volumes, and an Amazon Virtual Private Cloud. Based on your selections the wizard presents you with a dynamically generated estimated cost of deployment – as you modify your resource selections, you can see an updated cost assessment to help you match your budget.

Once you approve, AWS Launch Wizard for SQL Server provisions these resources and configures them to create a fully functioning production-ready SQL Server Always On deployment in just a few hours. The created resources are tagged making it easy to identity and work with them and the wizard also creates AWS CloudFormation templates, providing you with a baseline for repeatable and consistent application deployments.

Subsequent SQL Server Always On deployments become faster and easier as AWS Launch Wizard for SQL Server takes care of dealing with the required infrastructure on your behalf, determining the resources to match your application’s requirements such as performance, memory, bandwidth etc (you can modify the recommended defaults if you wish). If you want to bring your own SQL Server licenses, or have other custom requirements for the instances, you can also select to use your own custom AMIs provided they meet certain requirements (noted in the service documentation).

Using AWS Launch Wizard for SQL Server
To get started with my deployment, in the Launch Wizard Console I click the Create deployment button to start the wizard and select SQL Server Always On.

The wizard requires an AWS Identity and Access Management (IAM) role granting it permissions to deploy and access resources in my account. The wizard will check to see if a role named AmazonEC2RoleForLaunchWizard exists in my account. If so it will be used, otherwise a new role will be created. The new role will have two AWS managed policies, AmazonSSMManagedInstanceCore and AmazonEC2RolePolicyforLaunchWizard, attached to it. Note that this one time setup process will be typically performed by an IAM Administrator for your organization. However, the IAM user does not have to be an Administrator and CreateRole, AttachRolePolicy, and GetRole permissions are sufficient to perform these operations. After the role is created, the IAM Administrator can delegate the application deployment process to another IAM user who, in turn, must have the AWS Launch Wizard for SQL Server IAM managed policy called AmazonLaunchWizardFullaccess attached to it.

With the application type selected I can proceed by clicking Next to start configuring my application settings, beginning with setting a deployment name and optionally an Amazon Simple Notification Service (SNS) topic that AWS Launch Wizard for SQL Server can use for notifications and alerts. In the connectivity options I can choose to use an existing Amazon Virtual Private Cloud or have a new one created. I can also specify the name of an existing key pair (or create one). The key pair will be used if I want to RDP into my instances or obtain the administrator password. For a new Virtual Private Cloud I can also configure the IP address or range to which remote desktop access will be permitted:
Instances launched by AWS Launch Wizard for SQL Server will be domain joined to an Active Directory. I can select either an existing AWS Managed AD, or an on-premises AD, or have the wizard create a new AWS Managed Directory for my deployment:

The final application settings relate to SQL Server. This is also where I can specify a custom AMI to be used if I want to bring my own SQL Server licenses or have other customization requirements. Here I’m just going to create a new SQL Server Service account and use an Amazon-provided image with license included. Note that if I choose to use an existing service account it should be part of the Managed AD in which you are deploying:

Clicking Next takes me to a page to define the infrastructure requirements of my application, in terms of CPU and network performance and memory. I can also select the type of storage (solid state vs magnetic) and required SQL Server throughput. The wizard will recommend the resource types to be launched but I can also override and select specific instance and volume types, and I can also set custom tags to apply to the resources that will be created:

The final section of this page shows me the cost estimate based on my selections. This data in this panel is dynamically generated based on my prior selections and I can go back and forth in the wizard, tuning my selections to match my budget:

When I am happy with my selections, clicking Next takes me to wizard’s final Review page where I can view a summary of my selections and acknowledge that AWS resources and AWS Identity and Access Management (IAM) permissions will be created on my behalf, along with the estimated cost as was shown in the estimator on the previous page. My final step is to click Deploy to start the deployment process. Status updates during deployment can be viewed on the Deployments page with a final notification to inform me on completion.

Post-deployment Management
Once my application has been deployed I can manage its resources easily. Firstly I can navigate to Deployments on the AWS Launch Wizard for SQL Server dashboard and using the Actions dropdown I can jump to the Amazon Elastic Compute Cloud (EC2) console where I can manage the EC2 instances, EBS volumes, Active Directory etc. Or, using the same Actions dropdown, I can access SQL Server via the remote desktop gateway instance. If I want to manage future updates and patches to my application using AWS Systems Manager another Actions option takes me to the Systems Manager dashboard for managing my application. I can also use the AWS Launch Wizard for SQL Server to delete deployments performed using the wizard and it will perform a roll-back of all AWS CloudFormation stacks that the service created.

Now Available
AWS Launch Wizard for SQL Server is generally available and you can use it in the following AWS Regions: US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), Canada (Central), South America (Sao Paulo), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Seoul), Asia Pacific (Tokyo), EU (Frankfurt), EU (Ireland), EU (London), and EU (Stockholm). Support for the AWS regions in China, and for the GovCloud Region, is in the works. There is no additional charge for using AWS Launch Wizard for SQL Server, only for the resources it creates.

— Steve