All posts by Jeff Barr

Amazon S3 Path Deprecation Plan – The Rest of the Story

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

Last week we made a fairly quiet (too quiet, in fact) announcement of our plan to slowly and carefully deprecate the path-based access model that is used to specify the address of an object in an S3 bucket. I spent some time talking to the S3 team in order to get a better understanding of the situation in order to write this blog post. Here’s what I learned…

We launched S3 in early 2006. Jeff Bezos’ original spec for S3 was very succinct – he wanted malloc (a key memory allocation function for C programs) for the Internet. From that starting point, S3 has grown to the point where it now stores many trillions of objects and processes millions of requests per second for them. Over the intervening 13 years, we have added many new storage options, features, and security controls to S3.

Old vs. New
S3 currently supports two different addressing models: path-style and virtual-hosted style. Let’s take a quick look at each one. The path-style model looks like either this (the global S3 endpoint):

https://s3.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg
https://s3.amazonaws.com/jeffbarr-public/classic_amazon_door_desk.png

Or this (one of the regional S3 endpoints):

https://s3-us-east-2.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg
https://s3-us-east-2.amazonaws.com/jeffbarr-public/classic_amazon_door_desk.png

In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /classic_amazon_door_desk.png are object keys.

Even though the objects are owned by distinct AWS accounts and are in different S3 buckets (and possibly in distinct AWS regions), both of them are in the DNS subdomain s3.amazonaws.com. Hold that thought while we look at the equivalent virtual-hosted style references (although you might think of these as “new,” they have been around since at least 2010):

https://jbarr-public.s3.amazonaws.com/images/ritchie_and_thompson_pdp11.jpeg
https://jeffbarr-public.s3.amazonaws.com/classic_amazon_door_desk.png

These URLs reference the same objects, but the objects are now in distinct DNS subdomains (jbarr-public.s3.amazonaws.com and jeffbarr-public.s3.amazonaws.com, respectively). The difference is subtle, but very important. When you use a URL to reference an object, DNS resolution is used to map the subdomain name to an IP address. With the path-style model, the subdomain is always s3.amazonaws.com or one of the regional endpoints; with the virtual-hosted style, the subdomain is specific to the bucket. This additional degree of endpoint specificity is the key that opens the door to many important improvements to S3.

Out with the Old
In response to feedback on the original deprecation plan that we announced last week, we are making an important change. Here’s the executive summary:

Original Plan – Support for the path-style model ends on September 30, 2020.

Revised Plan – Support for the path-style model continues for buckets created on or before September 30, 2020. Buckets created after that date must be referenced using the virtual-hosted model.

We are moving to virtual-hosted references for two reasons:

First, anticipating a world with billions of buckets homed in many dozens of regions, routing all incoming requests directly to a small set of endpoints makes less and less sense over time. DNS resolution, scaling, security, and traffic management (including DDoS protection) are more challenging with this centralized model. The virtual-hosted model reduces the area of impact (which we call the “blast radius” internally) when problems arise; this helps us to increase availability and performance.

Second, the team has a lot of powerful features in the works, many of which depend on the use of unique, virtual-hosted style subdomains. Moving to this model will allow you to benefit from these new features as soon as they are announced. For example, we are planning to deprecate some of the oldest security ciphers and versions (details to come later). The deprecation process is easier and smoother (for you and for us) if you are using virtual-hosted references.

In With the New
As just one example of what becomes possible when using virtual-hosted references, we are thinking about providing you with increased control over the security configuration (including ciphers and cipher versions) for each bucket. If you have ideas of your own, feel free to get in touch.

Moving Ahead
Here are some things to know about our plans:

Identifying Path-Style References – You can use S3 Access Logs (look for the Host Header field) and AWS CloudTrail Data Events (look for the host element of the requestParameters entry) to identify the applications that are making path-style requests.

Programmatic Access – If your application accesses S3 using one of the AWS SDKs, you don’t need to do anything, other than ensuring that your SDK is current. The SDKs already use virtual-hosted references to S3, except if the bucket name contains one or more “.” characters.

Bucket Names with Dots – It is important to note that bucket names with “.” characters are perfectly valid for website hosting and other use cases. However, there are some known issues with TLS and with SSL certificates. We are hard at work on a plan to support virtual-host requests to these buckets, and will share the details well ahead of September 30, 2020.

Non-Routable Names – Some characters that are valid in the path component of a URL are not valid as part of a domain name. Also, paths are case-sensitive, but domain and subdomain names are not. We’ve been enforcing more stringent rules for new bucket names since last year. If you have data in a bucket with a non-routable name and you want to switch to virtual-host requests, you can use the new S3 Batch Operations feature to move the data. However, if this is not a viable option, please reach out to AWS Developer Support.

Documentation – We are planning to update the S3 Documentation to encourage all developers to build applications that use virtual-host requests. The Virtual Hosting documentation is a good starting point.

We’re Here to Help
The S3 team has been working with some of our customers to help them to migrate, and they are ready to work with many more.

Our goal is to make this deprecation smooth and uneventful, and we want to help minimize any costs you may incur! Please do not hesitate to reach out to us if you have questions, challenges, or concerns.

Jeff;

PS – Stay tuned for more information on tools and other resources.

New – The Next Generation (I3en) of I/O-Optimized EC2 Instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-the-next-generation-i3en-of-i-o-optimized-ec2-instances/

Amazon’s Customer Obsession leadership principle says:

Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.

Starting from the customer and working backwards means that we do not invent in a vacuum. Instead, we speak directly to our customers (both external and internal), ask detailed questions, and pay attention to what we learn. On the AWS side, we often hear about new use cases that help us to get a better understanding of what our customers are doing with AWS. For example, large-scale EC2 users provide us with another set of interesting data points, often expressed in terms of ratios between dollars, vCPUs, memory size, storage size, and networking throughput.

We launched the I3 instances (Now Available – I3 Instances for Demanding, I/O Intensive Workloads) just about two years ago. Our customers use them to host distributed file systems, relational & NoSQL databases, in-memory caches, key-value stores, data warehouses, and MapReduce clusters. Because our customers are always (in Jeff Bezos’ words) “divinely discontent”, they want I/O-optimized instances with even more power & storage. To be specific, they have asked us for:

  • A lower price per TB of storage
  • Increased storage density to allow consolidation of workloads and scale-up processing
  • A higher ratio of network bandwidth and instance storage to vCPUs

The crucial element here is that our customers were able to express their needs in a detailed and specific manner. Simply asking for something to be better, faster, and cheaper does not help us to make well-informed decisions.

New I3en Instances
Today I am happy to announce the I3en instances. Designed to meet these needs and to do an even better job of addressing the use cases that I mentioned above, these instances are powered by AWS-custom Intel Xeon Scalable (Skylake) processors with 3.1 GHz sustained all-core turbo performance, up to 60 TB of fast NVMe storage, and up to 100 Gbps of network bandwidth. Here are the specs:

Instance NamevCPUs
MemoryLocal Storage
(NVMe SSD)
Random Read IOPS
(4 K Block)
Read Throughput
(128 K Block)
EBS-Optimized BandwidthNetwork Bandwidth
i3en.large216 GiB1 x 1.25 TB42.5 K325 MB/sUp to 3,500 MbpsUp to 25 Gbps
i3en.xlarge432 GiB1 x 2.50 TB85 K650 MB/sUp to 3,500 MbpsUp to 25 Gbps
i3en.2xlarge864 GiB2 x 2.50 TB170 K1.3 GB/sUp to 3,500 MbpsUp to 25 Gbps
i3en.3xlarge1296 GiB1 x 7.5 TB250 K2 GB/sUp to 3,500 MbpsUp to 25 Gbps
i3en.6xlarge24192 GiB2 x 7.5 TB500 K4 GB/s3,500 Mbps25 Gbps
i3en.12xlarge48384 GiB4 x 7.5 TB1 M8 GB/s7,000 Mbps50 Gbps
i3en.24xlarge96768 GiB8 x 7.5 TB2 M16 GB/s14,000 Mbps100 Gbps

In comparison to the I3 instances, the I3en instances offer:

  • A cost per GB of SSD instance storage that is up to 50% lower
  • Storage density (GB per vCPU) that is roughly 2.6x greater
  • Ratio of network bandwidth to vCPUs that is up to 2.7x greater

You will need HVM AMIs with the NVMe 1.0e and ENA drivers. You can also make use of the new Elastic Fabric Adapter (EFA) if you are using the i3en.24xlarge (read my recent post to learn more).

Now Available
You can launch I3en instances today in the US East (N. Virginia), US West (Oregon), and Europe (Ireland) Regions in On-Demand and Spot form. Reserved Instances, Dedicated Instances, and Dedicated Hosts are available.

Jeff;

 

 

SAP on AWS Update – Customer Case Studies, Scale-Up, Scale-Out, and More

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/sap-on-aws-update-customer-case-studies-scale-up-scale-out-and-more/

SAP SAPPHIRE NOW 2019 takes place this week in Florida! Many of my AWS colleagues will be there, and they would love to talk to you. Today, I would like to share some customer success stories and give you a brief update on the work that we have been doing to make sure that AWS is the best place for you to run your SAP-powered OLTP and OLAP enterprise workloads.

Customer Update
Let’s start with a quick recap of some recent customer success stories. Here are just a few of the many customers that are using SAP on AWS in production today:

Fiat Chrysler Automotive – After exploring multiple options and vendors, FIAT decided to deploy SAP on AWS with Capgemini as a partner:

Engie – Read the case study to learn how this international energy provider has been able to Transform and Streamline their Financial Processes and drastically reduced the ramp-up time for new users from three days to one day by running SAP S/4HANA on AWS:


AIG – Watch the video to learn how AIG migrated 13 SAP landscapes from an on-premises environment to SAP HANA on AWS in 13 months, while reducing their infrastructure cost by $8M:

Sumitomo Chemical – Read this case study to learn how Sumitomo Chemical runs a huge number of SAP ERP batch jobs on AWS, cutting job processing time by around 40%:

There are additional SAP on AWS Case Studies for your reading pleasure!

AWS customers are making great use of the 6, 9, and 12 TB EC2 High Memory instances that we launched last year. They are building many different SAP Solutions on AWS, taking advantage of the SAP Rapid Migration Test Program, and working with members of the SAP Competency Partner Network.

What’s New
Our customers are building ever-larger SAP installations, using both scale-up (larger instances) or scale-out (more instances) models. We have been working with SAP to certify two additional scale-out options:

48 TB Scale-Out (S/4HANA) – When we launched the EC2 High Memory instances with 12 TB of memory last year, they were certified by SAP to run OLTP and OLAP HANA workloads in scale-up configurations. These instances now support additional configuration choices for your OLTP workloads. You can now use up to four of these 12 TB High Memory instances to run an OLTP S/4HANA solution in scale-out mode, while meeting all SAP requirements.

This is the first ever SAP-certified scale-out certification of S/4HANA on cloud instances. SAP recommends (SAP OSS Note 2408419) the use of bare metal platforms with a minimum of 8 CPUs and 6 TB of memory for running S/4HANA in scale-out. Since the EC2 High Memory instances with 12 TB memory is an EC2 bare metal instance that combines the benefits of the cloud with the performance characteristics of a bare metal platform, it is able to support SAP-certified scale-out configurations for S/4HANA in the cloud. To learn more, read Announcing support for extremely large S/4HANA deployments on AWS and review the certification.

100 TB Scale-Out (BW4/HANA, BW on HANA, Datamart) – You can now use up to 25 x1e.32xlarge EC2 instances (thanks to TDI Phase 5) to create an OLAP solution that scales to 100 TB, again while meeting all SAP requirements. You can start with as little as 244 GB of memory and scale out to 100 TB; review the certification to learn more.

The 48 TB OLTP solution and the 100 TB OLAP solution are the largest SAP-certified solutions available from any cloud provider.

We also have a brand-new S/4HANA Quick Start to help you get going in minutes. It sets up a VPC that spans two Availability Zones, each with a public and private subnet, and a pair of EC2 instances. One instance hosts the primary copy of S/4HANA and the other hosts the secondary. Read the Quick Start to learn more:

What’s Next
Ok, still with me? I hope so, since I have saved the biggest news for last!

We are getting ready to extend our lineup of EC2 High Memory instances, and will make them available with 18 TB and 24 TB of memory in the fall of 2019. The instances will use second-generation Intel® Xeon® Scalable processors, and will be available in bare metal form. Like the existing EC2 High Memory instances, you will be able to run them in the same Virtual Private Cloud (VPC) that hosts your cloud-based business applications, and you will be able to make use of important EC2 features such as Elastic Block Store and advanced networking. You can launch, manage, and even resize these EC2 instances using the AWS Command Line Interface (CLI) and the AWS SDKs.

Here are screen shots of SAP HANA Studio running on 18 TB and 24 TB instances that are currently in development:

And here is the output from top on those instances:

Here is a handy reference to all of your scale-up and scale-out SAP HANA on AWS options:

If you want to learn more or you want to gain early access to the new instances, go ahead and contact us.

Jeff;

 

New – Amazon S3 Batch Operations

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-s3-batch-operations/

AWS customers routinely store millions or billions of objects in individual Amazon Simple Storage Service (S3) buckets, taking advantage of S3’s scale, durability, low cost, security, and storage options. These customers store images, videos, log files, backups, and other mission-critical data, and use S3 as a crucial part of their data storage strategy.

Batch Operations
Today, I would like to tell you about Amazon S3 Batch Operations. You can use this new feature to easily process hundreds, millions, or billions of S3 objects in a simple and straightforward fashion. You can copy objects to another bucket, set tags or access control lists (ACLs), initiate a restore from Glacier, or invoke an AWS Lambda function on each one.

This feature builds on S3’s existing support for inventory reports (read my S3 Storage Management Update post to learn more), and can use the reports or CSV files to drive your batch operations. You don’t have to write code, set up any server fleets, or figure out how to partition the work and distribute it to the fleet. Instead, you create a job in minutes with a couple of clicks, turn it loose, and sit back while S3 uses massive, behind-the-scenes parallelism to take care of the work. You can create, monitor, and manage your batch jobs using the S3 Console, the S3 CLI, or the S3 APIs.

A Quick Vocabulary Lesson
Before we get started and create a batch job, let’s review and introduce a couple of important terms:

Bucket – An S3 bucket holds a collection of any number of S3 objects, with optional per-object versioning.

Inventory Report – An S3 inventory report is generated each time a daily or weekly bucket inventory is run. A report can be configured to include all of the objects in a bucket, or to focus on a prefix-delimited subset.

Manifest – A list (either an Inventory Report, or a file in CSV format) that identifies the objects to be processed in the batch job.

Batch Action – The desired action on the objects described by a Manifest. Applying an action to an object constitutes an S3 Batch Task.

IAM Role – An IAM role that provides S3 with permission to read the objects in the inventory report, perform the desired actions, and to write the optional completion report. If you choose Invoke AWS Lambda function as your action, the function’s execution role must grant permission to access the desired AWS services and resources.

Batch Job – References all of the items above. Each job has a status and a priority; higher priority (numerically) jobs take precedence over those with lower priority.

Running a Batch Job
Ok, let’s use the S3 Console to create and run a batch job! In preparation for this blog post I enabled inventory reports for one of my S3 buckets (jbarr-batch-camera) earlier this week, with the reports routed to jbarr-batch-inventory:

I select the desired inventory item, and click Create job from manifest to get started (I can also click Batch operations while browsing my list of buckets). All of the relevant information is already filled in, but I can choose an earlier version of the manifest if I want (this option is only applicable if the manifest is stored in a bucket that has versioning enabled). I click Next to proceed:

I choose my operation (Replace all tags), enter the options that are specific to it (I’ll review the other operations later), and click Next:

I enter a name for my job, set its priority, and request a completion report that encompasses all tasks. Then I choose a bucket for the report and select an IAM Role that grants the necessary permissions (the console also displays a role policy and a trust policy that I can copy and use), and click Next:

Finally, I review my job, and click Create job:

The job enters the Preparing state. S3 Batch Operations checks the manifest and does some other verification, and the job enters the Awaiting your confirmation state (this only happens when I use the console). I select it and click Confirm and run:

I review the confirmation (not shown) to make sure that I understand the action to be performed, and click Run job. The job enters the Ready state, and starts to run shortly thereafter. When it is done it enters the Complete state:

If I was running a job that processed a substantially larger number of objects, I could refresh this page to monitor status. One important thing to know: After the first 1000 objects have been processed, S3 Batch Operations examines and monitors the overall failure rate, and will stop the job if the rate exceeds 50%.

The completion report contains one line for each of my objects, and looks like this:

Other Built-In Batch Operations
I don’t have enough space to give you a full run-through of the other built-in batch operations. Here’s an overview:

The PUT copy operation copies my objects, with control of the storage class, encryption, access control list, tags, and metadata:

I can copy objects to the same bucket to change their encryption status. I can also copy them to another region, or to a bucket owned by another AWS account.

The Replace Access Control List (ACL) operation does exactly that, with control over the permissions that are granted:

And the Restore operation initiates an object-level restore from the Glacier or Glacier Deep Archive storage class:

Invoking AWS Lambda Functions
I have saved the most general option for last. I can invoke a Lambda function for each object, and that Lambda function can programmatically analyze and manipulate each object. The Execution Role for the function must trust S3 Batch Operations:

Also, the Role for the Batch job must allow Lambda functions to be invoked.

With the necessary roles in place, I can create a simple function that calls Amazon Rekognition for each image:

import boto3
def lambda_handler(event, context):
    s3Client = boto3.client('s3')
    rekClient = boto3.client('rekognition')
    
    # Parse job parameters
    jobId = event['job']['id']
    invocationId = event['invocationId']
    invocationSchemaVersion = event['invocationSchemaVersion']

    # Process the task
    task = event['tasks'][0]
    taskId = task['taskId']
    s3Key = task['s3Key']
    s3VersionId = task['s3VersionId']
    s3BucketArn = task['s3BucketArn']
    s3Bucket = s3BucketArn.split(':')[-1]
    print('BatchProcessObject(' + s3Bucket + "/" + s3Key + ')')
    resp = rekClient.detect_labels(Image={'S3Object':{'Bucket' : s3Bucket, 'Name' : s3Key}}, MaxLabels=10, MinConfidence=85)
    
    l = [lb['Name'] for lb in resp['Labels']]
    print(s3Key + ' - Detected:' + str(sorted(l)))

    results = [{
        'taskId': taskId,
        'resultCode': 'Succeeded',
        'resultString': 'Succeeded'
    }]
    
    return {
        'invocationSchemaVersion': invocationSchemaVersion,
        'treatMissingKeysAs': 'PermanentFailure',
        'invocationId': invocationId,
        'results': results
    }

With my function in place, I select Invoke AWS lambda function as my operation when I create my job, and choose my BatchProcessObject function:

Then I create and confirm my job as usual. The function will be invoked for each object, taking advantage of Lambda’s ability to scale and allowing this moderately-sized job to run to completion in less than a minute:

I can find the “Detected” messages in the CloudWatch Logs Console:

As you can see from my very simple example, the ability to easily run Lambda functions on large numbers of S3 objects opens the door to all sorts of interesting applications.

Things to Know
I am looking forward to seeing and hearing about the use cases that you discover for S3 Batch Operations! Before I wrap up, here are some final thoughts:

Job Cloning – You can clone an existing job, fine-tune the parameters, and resubmit it as a fresh job. You can use this to re-run a failed job or to make any necessary adjustments.

Programmatic Job Creation – You could attach a Lambda function to the bucket where you generate your inventory reports and create a fresh batch job each time a report arrives. Jobs that are created programmatically do not need to be confirmed, and are immediately ready to execute.

CSV Object Lists – If you need to process a subset of the objects in a bucket and cannot use a common prefix to identify them, you can create a CSV file and use it to drive your job. You could start from an inventory report and filter the objects based on name or by checking them against a database or other reference. For example, perhaps you use Amazon Comprehend to perform sentiment analysis on all of your stored documents. You can process inventory reports to find documents that have not yet been analyzed and add them to a CSV file.

Job Priorities – You can have multiple jobs active at once in each AWS region. Your jobs with a higher priority take precedence, and can cause existing jobs to be paused momentarily. You can select an active job and click Update priority in order to make changes on the fly:

Learn More
Here are some resources to help you learn more about S3 Batch Operations:

Documentation – Read about Creating a Job, Batch Operations, and Managing Batch Operations Jobs.

Tutorial Videos – Check out the S3 Batch Operations Video Tutorials to learn how to Create a Job, Manage and Track a Job, and to Grant Permissions.

Now Available
You can start using S3 Batch Operations in all commercial AWS regions except Asia Pacific (Osaka) today. S3 Batch Operations is also available in both of the AWS GovCloud (US) regions.

Jeff;

Use AWS Transit Gateway & Direct Connect to Centralize and Streamline Your Network Connectivity

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-aws-transit-gateway-direct-connect-to-centralize-and-streamline-your-network-connectivity/

Last year I showed you how to Use an AWS Transit Gateway to Simplify Your Network Architecture. As I said at the time:

You can connect your existing VPCs, data centers, remote offices, and remote gateways to a managed Transit Gateway, with full control over network routing and security, even if your VPCs, Active Directories, shared services, and other resources span multiple AWS accounts. You can simplify your overall network architecture, reduce operational overhead, and gain the ability to centrally manage crucial aspects of your external connectivity, including security. Last but not least, you can use Transit Gateways to consolidate your existing edge connectivity and route it through a single ingress/egress point.

In that post I also promised you support for AWS Direct Connect, and I’m happy to announce that this support is available today for use in the US East (N. Virginia), US East (Ohio), US West (N. California), and US West (Oregon) Regions. The applications that you run in the AWS Cloud can now communicate with each other, and with your on-premises applications, at speeds of up to 10 Gbps per Direct Connect connection. You can set it up in minutes (assuming that you already have a private or hosted connection running at 1 Gbps or more) and start using it right away.

Putting it all together, you get a lot of important benefits from today’s launch:

Simplification – You can simplify your network architecture and your network management overhead by creating a hub-and-spoke model that spans multiple VPCs, regions, and AWS accounts. If you go this route, you may also be in a position to cut down on the number of AWS VPN connections that you use.

Consolidation – You have the opportunity to reduce the number of private or hosted connections, saving money and avoiding complexity in the process. You can consolidate your connectivity so that it all flows across the same BGP session.

Connectivity – You can reach your Transit Gateway using your connections from any of the 90+ AWS Direct Connect locations (except from AWS Direct Connect locations in China).

Using Transit Gateway & Direct Connect
I will use the freshly updated Direct Connect Console to set up my Transit Gateway for use with Direct Connect. The menu on the left lets me view and create the resources that I will need:

My AWS account already has access to a 1 Gbps connection (MyConnection) to TierPoint in Seattle:

I create a Direct Connect Gateway (MyDCGateway):

I create a Virtual Interface (VIF) with type Transit:

I reference my Direct Connect connection (MyConnection) and my Direct Connect Gateway (MyDCGateway) and click Create virtual interface:

When the state of my new VIF switches from pending to down I am ready to proceed:

Now I am ready to create my transit gateway (MyTransitGW). This is a VPC component; clicking on Transit gateways takes me to the VPC console. I enter a name, description, and ASN (which must be distinct from the one that I used for the Direct Connect Gateway), leave the other values as-is, and click Create Transit Gateway:

The state starts out as pending, and transitions to available:

With all of the resources ready, I am ready to connect them! I return to the Direct Connect Console, find my Transit Gateway, and click Associate Direct Connect gateway:

I associate the Transit Gateway with a Direct Connect Gateway in my account (using another account requires the ID of the gateway and the corresponding AWS account number), and list the network prefixes that I want to advertise to the other side of the Direct Connect connection. Then I click Associate Direct Connect gateway to make it so:

The state starts out as associating and transitions to associated. This can take some time, so I will take Luna for a walk:

By the time we return, the Direct Connect Gateway is associated with the Transit Gateway, and we are good to go!

In a real-world situation you would spend more time planning your network topology and addressing, and you would probably use multiple AWS accounts.

Available Now
You can use this new feature today to interface with your Transit Gateways hosted in four AWS regions.

Jeff;

New – Amazon Managed Blockchain – Create & Manage Scalable Blockchain Networks

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-managed-blockchain-create-manage-scalable-blockchain-networks/

Trust is a wonderful thing, and is the basis for almost every business and personal relationship or transaction. In some cases, trust is built up over an extended period of time, reinforced with each successful transaction and seen as an integral part of the relationship. In other situations, there’s no time to accumulate trust and other mechanisms must be used instead. The parties must find a way to successfully complete the transaction in the absence of trust. Today, emerging blockchain technologies such as Hyperledger Fabric and Ethereum fill this important need, allowing parties to come to consensus regarding the validity of a proposed transaction and create an unalterable digital record (commonly known as a ledger) of each transaction in the absence of trust.

Amazon Managed Blockchain
We announced Amazon Managed Blockchain at AWS re:Invent 2018 and invited you to sign up for a preview. I am happy to announce that the preview is complete and that Amazon Managed Blockchain is now available for production use in the US East (N. Virginia) Region. You can use it to create scalable blockchain networks that use the Hyperledger Fabric open source framework, with Ethereum in the works. As you will see in a minute, you can create your network in minutes. Once created, you can easily manage and maintain your blockchain network. You can manage certificates, invite new members, and scale out peer node capacity in order to process transactions more quickly.

The blockchain networks that you create with Amazon Managed Blockchain can span multiple AWS accounts so that a group of members can execute transactions and share data without a central authority. New members can easily launch and configure peer nodes that process transaction requests and store a copy of the ledger.

Using Amazon Managed Blockchain
I can create my own scalable blockchain network from the AWS Management Console, AWS Command Line Interface (CLI) (aws managedblockchain create-network), or API (CreateNetwork). To get started, I open the Amazon Managed Blockchain Console and click Create a network:

I need to choose the edition (Starter or Standard) for my network. The Starter Edition is designed for test networks and small production networks, with a maximum of 5 members per network and 2 peer nodes per member. The Standard Edition is designed for scalable production use, with up to 14 members per network and 3 peer nodes per member (check out the Amazon Managed Blockchain Pricing to learn more about both editions). I also enter a name and a description for my network:

Then I establish the voting policy for my network, and click Next to move ahead (read Work with Proposals to learn more about creating and voting on proposals):

Now, I need to create the first member of my network. Each member is a distinct identity within the network, and is visible within the network. I also set up a user name and password for my certificate authority, and click Next:

I review my choices, and click Create network and member:

My network enters the Creating status, and I take a quick break to walk my dog! When I return, my network is Available:

Inviting Members
Now that my network is available, I can invite members by clicking the Members tab:

I can see the current members of my network, both those I own and those owned by others. I click on Propose invitation to invite a new member:

Then I enter the AWS account number of the proposed member and click Create:

This creates a proposal (visible to me and to the other members of the network). I click on the ID to proceed:

I review the proposal, select my identity (block-wizard), and then click Yes to vote:

After enough Yes votes have been received to pass the threshold that I specified when I created the network, the invitation will be extended to the new member, and will be visible in the Invitations section:

If you are building a blockchain network for testing purposes and don’t have access to multiple AWS accounts, you can even invite your own account. After you do this (and vote to let yourself in), you will end up with multiple members in the same account.

Using the Network
Now that the network is running, and has some members, the next step is to create an endpoint in the Virtual Private Cloud (VPC) where I will run my blockchain applications (this feature is powered by AWS PrivateLink). Starting from the detail page for my network, I click Create VPC endpoint:

I choose the desired VPC and the subnets within it, pick a security group, and click Create:

My applications can use the VPC endpoint to communicate with my blockchain network:

The next step is to build applications that make use of the blockchain. To learn how to do this, read Build and deploy an application for Hyperledger Fabric on Amazon Managed Blockchain. You can also read Get Started Creating a Hyperledger Fabric Blockchain Network Using Amazon Managed Blockchain.

Things to Know
As usual, we have a healthy roadmap for this new service. Stay tuned to learn more!

Jeff;

PS – Check out the AWS Blockchain Pub to see a novel use for Amazon Managed Blockchain and AWS DeepLens.

 

The AWS DeepRacer League Virtual Circuit is Now Open – Train Your Model Today!

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/the-aws-deepracer-league-virtual-circuit-is-now-open-train-your-model-today/

AWS DeepRacer is a 1/18th scale four-wheel drive car with a considerable amount of onboard hardware and software. Starting at re:Invent 2018 and continuing with the AWS Global Summits, you have the opportunity to get hands-on experience with a DeepRacer. At these events, you can train a model using reinforcement learning, and then race it around a track. The fastest racers and their laptimes for each summit are shown on our leaderboards.

New DeepRacer League Virtual Circuit
Today we are launching the AWS DeepRacer League Virtual Circuit. You can build, train, and evaluate your reinforcement learning models online and compete online for some amazing prizes, all from the comfort of the DeepRacer Console!

We’ll add a new track each month, taking inspiration from famous race tracks around the globe, so that you can refine your models and broaden your skill set. The top entrant in the leaderboard each month will win an expenses-paid package to AWS re:Invent 2019, where they will take part in the League Knockout Rounds, with a chance to win the Championship Cup!

New DeepRacer Console
We are making the DeepRacer Console available today in the US East (N. Virginia) Region. You can use it to build and train your DeepRacer models and to compete in the Virtual Circuit, while gaining practical, hands-on experience with Reinforcement Learning. Following the steps in the DeepRacer Lab that is used at the hands-on DeepRacer workshops, I open the console and click Get started:

The console provides me with an overview of the model training process, and then asks to create the AWS resources needed to train and evaluate my models. I review the info and click Create resources to proceed:

The resources are created in minutes (I can click Learn RL to learn more about reinforcement learning while this is happening). I click Create model to move ahead:

I enter a name and a description for for my model:

Then I pick a track (more tracks will appear throughout the duration of the Virtual League):

Now I define the Action space for my model. This is a set of discrete actions that my model can perform. Choosing values that increase the number of options will generally enhance the quality of my model, at the cost of additional training time:

Next, I define the reward function for my model. This function evaluates the current state of the vehicle throughout the training process and returns a reward value to indicate how well the model is performing (higher rewards signify better performance). I can use one of three predefined models (available by clicking Reward function examples) as-is, customize them, or build one from scratch. I’ll use Prevent zig-zag, a sample reward function that penalizes zig-zap behavior, to get started:

The reward function is written in Python 3, and has access to parameters (track_width, distance_from_center, all_wheels_on_track, and many more) that describe the position and state of the car, and also provide information about the track.

I also control a set of hyperparameters that affect the overall training performance. Since I don’t understand any of these (just being honest here), I will accept all of the defaults:

To learn more about hyperparameters, read Systematically Tune Hyperparameters.

Finally, I specify a time limit for my training job, and click Start training. In general, simple models will converge in 90 to 120 minutes, but this is highly dependent on the maximum speed and the reward function.

The training job is initialized (this takes about 6 minutes), and I can track progress in the console:

The training job makes use of AWS RoboMaker so I can also monitor it from the RoboMaker Console. For example, I can open the Gazebo window, see my car, and watch the training process in real time:

One note of caution: changing the training environment (by directly manipulating Gazebo) will adversely affect the training run, most likely rendering it useless.

As the training progresses, the Reward graph will go up and to the right (as we often say at Amazon) if the car is learning how to stay on the track:

If the graph flattens out or declines precipitously and stays there, your reward function is not rewarding the desired behavior or some other setting is getting in the way. However, patience is a virtue, and there will be the occasional regression on the way to the top. After the training is complete, there’s a short pause while the new model is finalized and stored, and then it is time for me to evaluate my model by running it in a simulation. I click Start evaluation to do this:

I can evaluate the model on any of the available tracks. Using one track for training and a different one for evaluation is a good way to make sure that the model is general, and has not been overfit so that it works on just one track. However, using the same track for training and testing is a good way to get started, and that’s what I will do. I select the Oval Track and 3 trials, and click Start evaluation:

The RoboMaker simulator launches, with an hourly cost for the evaluation, as noted above. The results (lap times) are displayed when the simulation is complete:

At this point I can evaluate my model on another track, step back and refine my model and evaluate it again, or submit my model to the current month’s Virtual Circuit track to take part in the DeepRacer League. To do that, I click Submit to virtual race, enter my racer name, choose a model, agree to the Ts and C’s, and click Submit model:

After I submit, my model will be evaluated on the pre-season track and my lap time will be used to place me in the Virtual Circuit Leaderboard.

Things to Know
Here are a couple of things to know about the AWS DeepRacer and the AWS DeepRacer League:

AWS ResourcesAmazon SageMaker is used to train models, which are then stored in Amazon Simple Storage Service (S3). AWS RoboMaker provides the virtual track environment, which is used for training and evaluation. An AWS CloudFormation stack is used to create a Amazon Virtual Private Cloud, complete with subnets, routing tables, an Elastic IP Address, and a NAT Gateway.

Costs – You can use the DeepRacer console at no charge. As soon as you start training your first model, you will get service credits for SageMaker and RoboMaker to give you 10 hours of free training on these services. The credits are applied at the end of the month and are available for 30 days, as part of the AWS Free Tier. The DeepRacer architecture uses a NAT Gateway that carries an availability charge. Your account will automatically receive service credits to offset this charge, showing net zero on your account.

DeepRacer Cars – You can preorder your DeepRacer car now! Deliveries to addresses in the United States will begin in July 2019.

Jeff;

Now Available – Elastic Fabric Adapter (EFA) for Tightly-Coupled HPC Workloads

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-available-elastic-fabric-adapter-efa-for-tightly-coupled-hpc-workloads/

We announced Elastic Fabric Adapter (EFA) at re:Invent 2018 and made it available in preview form at the time. During the preview, AWS customers put EFA through its paces on a variety of tightly-coupled HPC workloads, providing us with valuable feedback and helping us to fine-tune the final product.

Now Available
Today I am happy to announce that EFA is now ready for production use in multiple AWS regions. It is ready to support demanding HPC workloads that need lower and more consistent network latency, along with higher throughput, than is possible with traditional TCP communication. This launch lets you apply the scale, flexibility, and elasticity of the AWS Cloud to tightly-coupled HPC apps and I can’t wait to hear what you do with it. You can, for example, scale up to thousands of compute nodes without having to reserve the hardware or the network ahead of time.

All About EFA
An Elastic Fabric Adapter is an AWS Elastic Network Adapter (ENA) with added capabilities (read my post, Elastic Network Adapter – High Performance Network Interface for Amazon EC2, to learn more about ENA). An EFA can still handle IP traffic, but also supports an important access model commonly called OS bypass. This model allows the application (most commonly through some user-space middleware) access the network interface without having to get the operating system involved with each message. Doing so reduces overhead and allows the application to run more efficiently. Here’s what this looks like (source):

The MPI Implementation and libfabric layers of this cake play crucial roles:

MPI – Short for Message Passing Interface, MPI is a long-established communication protocol that is designed to support parallel programming. It provides functions that allow processes running on a tightly-coupled set of computers to communicate in a language-independent way.

libfabric – This library fits in between several different types of network fabric providers (including EFA) and higher-level libraries such as MPI. EFA supports the standard RDM (reliable datagram) and DGRM (unreliable datagram) endpoint types; to learn more, check out the libfabric Programmer’s Manual. EFA also supports a new protocol that we call Scalable Reliable Datagram; this protocol was designed to work within the AWS network and is implemented as part of our Nitro chip.

Working together, these two layers (and others that can be slotted in instead of MPI), allow you to bring your existing HPC code to AWS and run it with little or no change.

You can use EFA today on c5n.18xlarge and p3dn.24xlarge instances in all AWS regions where those instances are available. The instances can use EFA to communicate within a VPC subnet, and the security group must have ingress and egress rules that allow all traffic within the security group to flow. Each instance can have a single EFA, which can be attached when an instance is started or while it is stopped.

You will also need the following software components:

EFA Kernel Module – The EFA Driver is in the Amazon GitHub repo, and in the Amazon Linux & Amazon Linux 2 AMIs. We are working to add it to AMIs for other Linux distributions.

Libfabric Network Stack – You will need to use an AWS-custom version (already present in the Amazon Linux and Amazon Linux 2 AMIs) for now. We are working to get our changes into the next release (1.8) of libfabric.

MPI or NCCL Implementation – You can use Open MPI 3.1.3 (or later) or NCCL (2.3.8 or later) plus the OFI driver for NCCL. We also also working on support for the Intel MPI library.

You can launch an instance and attach an EFA using the CLI, API, or the EC2 Console, with CloudFormation support coming in a couple of weeks. If you are using the CLI, you need to include the subnet ID and ask for an EFA, like this (be sure to include the appropriate security group):

$ aws ec2 run-instances ... \
  --network-interfaces DeleteOnTermination=true,DeviceIndex=0,SubnetId=SUBNET,InterfaceType=efa

After your instance has launched, run lspci | grep efa0 to verify that the EFA device is attached. You can (but don’t have to) launch your instances in a Cluster Placement Group in order to benefit from physical adjacency when every light-foot counts. When used in this way, EFA can provide one-way MPI latency of 15.5 microseconds.

You can also create a Launch Template and use it to launch EC2 instances (either directly or as part of an EC2 Auto Scaling Group) in On-Demand or Spot Form, launch Spot Fleets, and to run compute jobs on AWS Batch.

Learn More
To learn more about EFA, and to see some additional benchmarks, be sure to watch this re:Invent video (Scaling HPC Applications on EC2 w/ Elastic Fabric Adapter):

 

 

AWS Customer CFD Direct maintains the popular OpenFOAM platform for Computational Fluid Dynamics (CFD) and also produces CFD Direct From the Cloud (CFDDC), an AWS Marketplace offering that makes it easy for you to run OpenFOAM on AWS. They have been testing and benchmarking EFA and recently shared their measurements in a blog post titled OpenFOAM HPC with AWS EFA. In the post, they report on a pair of simulations:

External Aerodynamics Around a Car – This simulation scales extra-linearly to over 200 cores, gradually declining to linear scaling at 1000 cores (about 100K simulation cells per core).

Flow Over a Weir with Hydraulic Jump – This simulation (1000 cores and 100M cells) scales at between 67% and 72.6%, depending on a “data write” setting.

Read the full post to learn more and to see some graphs and visualizations.

In the Works
We plan to add EFA support to additional EC2 instance types over time. In general, we plan to provide EFA support for the two largest sizes of “n” instances of any given type, and also for bare metal instances.

Jeff;

 

Now Open – AWS Asia Pacific (Hong Kong) Region

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-open-aws-asia-pacific-hong-kong-region/

The AWS Region in Hong Kong SAR is now open and you can start using it today. The official name is Asia Pacific (Hong Kong) and the API name is ap-east-1. The AWS Asia Pacific (Hong Kong) Region is the eighth active AWS Region in Asia Pacific and mainland China along with Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, and, Tokyo. With this launch, AWS now spans 64 Availability Zones within 21 geographic regions around the world. We have also announced plans for 12 more Availability Zones and four more AWS Regions in Bahrain, Cape Town, Jakarta, and Milan.

Instances and Services
Applications running in this 3-AZ region can use C5, C5d, D2, I3, M5, M5d, R5, R5d, and T3 instances, and can make use of a long list of AWS services including Amazon API Gateway, Application Auto Scaling, AWS Certificate Manager (ACM), AWS Artifact, AWS CloudFormation, Amazon CloudFront, AWS CloudTrail, Amazon CloudWatch, CloudWatch Events, Amazon CloudWatch Logs, AWS CodeDeploy, AWS Config, AWS Config Rules, AWS Database Migration Service, AWS Direct Connect, Amazon DynamoDB, EC2 Auto Scaling, EC2 Dedicated Hosts, Amazon Elastic Block Store (EBS), Amazon Elastic Compute Cloud (EC2), Elastic Container Registry, Amazon ECS, Application Load Balancers (Classic, Network, and Application), Amazon EMR, Amazon ElastiCache, Amazon Elasticsearch Service, Amazon Glacier, AWS Identity and Access Management (IAM), Amazon Kinesis Data Streams, AWS Key Management Service (KMS), AWS Lambda, AWS Marketplace, AWS Organizations, AWS Personal Health Dashboard, AWS Resource Groups, Amazon Redshift, Amazon Relational Database Service (RDS), Amazon Aurora, Amazon Route 53 (including Private DNS for VPCs), AWS Shield, AWS Server Migration Service, AWS Snowball, AWS Snowball Edge, Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), Amazon Simple Storage Service (S3), Amazon Simple Workflow Service (SWF), AWS Step Functions, AWS Support API, Amazon EC2 Systems Manager (SSM), AWS Trusted Advisor, Amazon Virtual Private Cloud, and VM Import/Export.

AWS Elastic Beanstalk, Amazon Elastic Container Service for Kubernetes, and AWS X-Ray are scheduled for deployment next month, with other services to follow. We are also working to enable cross-region delivery from SNS topics hosted in other regions to SQS queues hosted in the new region.

Using the Asia Pacific (Hong Kong) Region
As we announced last month, you need to explicitly enable this region for your AWS account in order to be able to create and manage resources within it. Enabling or disabling a region requires the account:EnableRegion, account:DisableRegion, and account:ListRegions permissions. Here’s a sample IAM policy that grants these permissions for the new region:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "aws-portal:ViewAccount",
            "account:ListRegions"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "account:EnableRegion",
            "account:DisableRegion"
         ],
         "Resource":"*",
         "Condition":{
            "StringEquals":{
               "account:TargetRegion":"ap-east-1"
            }
         }
      }
   ]
}

Log in to the AWS Management Console as a user that has these appropriate permissions and click My Account:

Scroll down to the AWS Regions section, find the new region, and click Enable:

Then confirm your action by clicking Enable region:

The region is enabled immediately, and will be ready for use shortly thereafter.

You can also enable the region by selecting it from the menu:

And then confirming your action:

Connectivity, Edge Locations, and Latency
Hong Kong SAR is already home to three Amazon CloudFront edge locations (the first one opened way back in 2008). There are also more than thirty other edge locations and eleven regional edge caches in Asia; see the Amazon CloudFront Infrastructure page for a full list.

The region offers low-latency connections to other cities and AWS regions in the area. Here are the latest numbers:

There are now two Hong Kong AWS Direct Connect locations: the existing one at iAdvantage Mega-I and a new one at Equinix HK1. Both locations have direct connectivity to the Asia Pacific (Hong Kong) Region. If you already connect to AWS at iAdvantage, you can use your existing Direct Connect connection to access the new region via Direct Connect Gateway.

Investing in the Future
Before I wrap up I would like to tell you about some of work that we are doing to support startups and to educate developers:

AWS Activate – This global program provides startups with credits, training, and support so that they can build their businesses on AWS.

AWS Educate – This global program teaches students about cloud computing. It provides AWS credits to educators and students, along with discounts on training, access to curated content, personalized learning pathways, and collaboration tools. Dozens of Hong Kong universities and business schools are already participating

AWS Academy – This global program is designed to bridge the gap between academia and industry by giving students the knowledge that they need to have in order to qualify for jobs that require cloud skills. The program is built around hands-on experience, and includes an AWS-authored curriculum, access to AWS certification, accreditation for educators.

Training and Certification – This global program helps developers to build cloud skills using digital or classroom training and to validate those skills by earning an industry-recognized credential. It includes learning paths for Cloud Practitioners, Architects, Developers, and Operations.

Jeff;

 

Now Available – AMD EPYC-Powered Amazon EC2 T3a Instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-available-amd-epyc-powered-amazon-ec2-t3a-instances/

The AMD EPYC-powered T3a instances that I promised you last year are available now and you can start using them today! Like the recently announced M5ad and R5ad instances, the T3a instances are built on the AWS Nitro System and give you an opportunity to balance your instance mix based on cost and performance.

T3a Instances
These instances deliver burstable, cost-effective performance and are a great fit for workloads that do not need high sustained compute power but experience temporary spikes in usage. You get a generous and assured baseline amount of processing power and the ability to transparently scale up to full core performance when you need more processing power, for as long as necessary. To learn more about the burstable compute model common to the T3 and the T3a, read New T3 Instances – Burstable, Cost-Effective Performance.

You can launch T3a instances today in seven sizes in the US East (N. Virginia), US West (Oregon), Europe (Ireland), US East (Ohio), and Asia Pacific (Singapore) Regions in On-Demand, Spot, and Reserved Instance form. Here are the specs:

Instance NamevCPUsRAMEBS-Optimized BandwidthNetwork Bandwidth
t3a.nano
20.5 GiBUp to 1.5 GbpsUp to 5 Gbps
t3a.micro
21 GiBUp to 1.5 GbpsUp to 5 Gbps
t3a.small
22 GiBUp to 1.5 GbpsUp to 5 Gbps
t3a.medium
24 GiBUp to 1.5 GbpsUp to 5 Gbps
t3a.large
28 GiBUp to 2.1 GbpsUp to 5 Gbps
t3a.xlarge
416 GiBUp to 2.1 GbpsUp to 5 Gbps
t3a.2xlarge
832 GiBUp to 2.1 GbpsUp to 5 Gbps

The T3 and the T3a instances are available in the same sizes and can use the same AMIs, making it easy for you to try both and find the one that is the best match for you application.

Pricing is 10% lower than the equivalent existing T3 instances; see the On-Demand, Spot, and Reserved Instance pricing pages for more info.

Jeff;

New – Query for AWS Regions, Endpoints, and More Using AWS Systems Manager Parameter Store

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-query-for-aws-regions-endpoints-and-more-using-aws-systems-manager-parameter-store/

In response to requests from AWS customers, I have been asking our service teams to find ways to make information about our regions and services available programmatically. Today I am happy to announce that this information is available in the AWS Systems Manager Parameter Store, and that you can easily access it from your scripts and your code. You can get a full list of active regions, find out which services are available with them, and much more.

Running Queries
I’ll use the AWS Command Line Interface (CLI) for most of my examples; you can also use the AWS Tools for Windows PowerShell or any of the AWS SDKs. As is the case with all of the CLI commands, you can request output in JSON, tab-delimited text, or table format. I’ll use JSON, and will make liberal use of the jq utility to show the more relevant part of the output from each query.

Here’s how to query for the list of active regions:

$ aws ssm get-parameters-by-path \
  --path /aws/service/global-infrastructure/regions --output json | \
  jq .Parameters[].Name
"/aws/service/global-infrastructure/regions/ap-northeast-1"
"/aws/service/global-infrastructure/regions/eu-central-1"
"/aws/service/global-infrastructure/regions/eu-north-1"
"/aws/service/global-infrastructure/regions/eu-west-1"
"/aws/service/global-infrastructure/regions/eu-west-3"
"/aws/service/global-infrastructure/regions/sa-east-1"
"/aws/service/global-infrastructure/regions/us-east-2"
"/aws/service/global-infrastructure/regions/us-gov-east-1"
"/aws/service/global-infrastructure/regions/us-gov-west-1"
"/aws/service/global-infrastructure/regions/us-west-1"
"/aws/service/global-infrastructure/regions/ap-northeast-2"
"/aws/service/global-infrastructure/regions/ap-northeast-3"
"/aws/service/global-infrastructure/regions/ap-south-1"
"/aws/service/global-infrastructure/regions/ap-southeast-1"
"/aws/service/global-infrastructure/regions/ap-southeast-2"
"/aws/service/global-infrastructure/regions/ca-central-1"
"/aws/service/global-infrastructure/regions/cn-north-1"
"/aws/service/global-infrastructure/regions/cn-northwest-1"
"/aws/service/global-infrastructure/regions/eu-west-2"
"/aws/service/global-infrastructure/regions/us-west-2"
"/aws/service/global-infrastructure/regions/us-east-1"

Here’s how to display a complete list of all available AWS services, sort them into alphabetical order, and display the first 10 (out of 155, as I write this):

$ aws ssm get-parameters-by-path \
  --path /aws/service/global-infrastructure/services --output json | \
  jq .Parameters[].Name | sort | head -10
"/aws/service/global-infrastructure/services/acm"
"/aws/service/global-infrastructure/services/acm-pca"
"/aws/service/global-infrastructure/services/alexaforbusiness"
"/aws/service/global-infrastructure/services/apigateway"
"/aws/service/global-infrastructure/services/application-autoscaling"
"/aws/service/global-infrastructure/services/appmesh"
"/aws/service/global-infrastructure/services/appstream"
"/aws/service/global-infrastructure/services/appsync"
"/aws/service/global-infrastructure/services/athena"
"/aws/service/global-infrastructure/services/autoscaling"

Here’s how to get the list of services that are available in a given region (again, first 10, sorted):

$ aws ssm get-parameters-by-path \
  --path /aws/service/global-infrastructure/regions/us-east-1/services --output json | \
  jq .Parameters[].Name | sort | head -10
"/aws/service/global-infrastructure/regions/us-east-1/services/acm"
"/aws/service/global-infrastructure/regions/us-east-1/services/acm-pca"
"/aws/service/global-infrastructure/regions/us-east-1/services/alexaforbusiness"
"/aws/service/global-infrastructure/regions/us-east-1/services/apigateway"
"/aws/service/global-infrastructure/regions/us-east-1/services/application-autoscaling"
"/aws/service/global-infrastructure/regions/us-east-1/services/appmesh"
"/aws/service/global-infrastructure/regions/us-east-1/services/appstream"
"/aws/service/global-infrastructure/regions/us-east-1/services/appsync"
"/aws/service/global-infrastructure/regions/us-east-1/services/athena"
"/aws/service/global-infrastructure/regions/us-east-1/services/autoscaling"

Here’s how to get the list of regions where a service (Amazon Athena, in this case) is available:

$ aws ssm get-parameters-by-path \
  --path /aws/service/global-infrastructure/services/athena/regions --output json | \
  jq .Parameters[].Value
"ap-northeast-2"
"ap-south-1"
"ap-southeast-2"
"ca-central-1"
"eu-central-1"
"eu-west-1"
"eu-west-2"
"us-east-1"
"us-east-2"
"us-gov-west-1"
"ap-northeast-1"
"ap-southeast-1"
"us-west-2"

Here’s how to use the path to get the name of a service:

$ aws ssm get-parameters-by-path \
  --path /aws/service/global-infrastructure/services/athena --output json | \
  jq .Parameters[].Value
"Amazon Athena"

And here’s how you can find the regional endpoint for a given service, again using the path:

$ aws ssm get-parameter \
  --name /aws/service/global-infrastructure/regions/us-west-1/services/s3/endpoint \
  --output json | \
  jq .Parameter.Value
"s3.us-west-1.amazonaws.com"

Available Now
This data is available now and you can start using it today at no charge.

Jeff;

PS – Special thanks to my colleagues Blake Copenhaver and Phil Cali for their help with this post!

 

AWS re:Inforce 2019 – Security, Identity, and Compliance

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-reinforce-2019-security-identity-and-compliance/

AWS re:Inforce, our new conference dedicated to cloud security, opens in Boston on June 25th. We’re expecting about 8,000 attendees, making this bigger than the first re:Invent! Just like re:Invent, re:Inforce is a learning conference for builders.

With over 300 breakout sessions (intermediate, advanced, and expert) spanning four tracks and a virtual Capture The Flag event, attendees will walk away knowing how to use cloud-based infrastructure in a secure and compliant manner. The re:Inforce agenda also includes a healthy collection of bootcamps, chalk talks, workshops, full-day hands-on labs, builder sessions, leadership sessions, and the Security Jam.

Diving deeper into the session offerings, a wide range of services will be considered – including (to name a few) AWS WAF, AWS Firewall Manager, AWS KMS, AWS Secrets Manager, AWS Lambda, AWS Control Tower, Amazon SageMaker, Amazon GuardDuty, AWS CloudTrail, Amazon Macie, Amazon RDS, Amazon Aurora, AWS Identity and Access Management, Amazon EKS, and Amazon Inspector. You will have the opportunity to learn about a broad variety of important topics including building secure APIs, encryption, privileged access, auditing the cloud, open source, DevSecOps, building a security culture, hiring/staffing, and privacy by design as well as specific compliance regimes such as PCI, NIST, SOC, FedRAMP, and HIPAA.

To learn more about re:Inforce, read the FAQ, check out the Venue & Hotel info, and review the Code of Conduct.

Register Now & Save $100
If you register now and use code RFSAL19, you can save $100, while supplies last.

Jeff;

 

 

The Wide World of Microsoft Windows on AWS

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/the-wide-world-of-microsoft-windows-on-aws/

You have been able to run Microsoft Windows on AWS since 2008 (my ancient post, Big Day for Amazon EC2: Production, SLA, Windows, and 4 New Capabilities, shows you just how far AWS come in a little over a decade). According to IDC, AWS has nearly twice as many Windows Server instances in the cloud as the next largest cloud provider.

Today, we believe that AWS is the best place to run Windows and Windows applications in the cloud. You can run the full Windows stack on AWS, including Active Directory, SQL Server, and System Center, while taking advantage of 61 Availability Zones across 20 AWS Regions. You can run existing .NET applications and you can use Visual Studio or VS Code build new, cloud-native Windows applications using the AWS SDK for .NET.

Wide World of Windows
Starting from this amazing diagram drawn by my colleague Jerry Hargrove, I’d like to explore the Windows-on-AWS ecosystem in detail:

1 – SQL Server Upgrades
AWS provides first-class support for SQL Server, encompassing all four Editions (Express, Web, Standard, and Enterprise), with multiple version of each edition. This wide-ranging support has helped SQL Server to become one of the most popular Windows workloads on AWS.

The SQL Server Upgrade Tool (an AWS Systems Manager script) makes it easy for you to upgrade an EC2 instance that is running SQL Server 2008 R2 SP3 to SQL Server 2016. The tool creates an AMI from a running instance, upgrades the AMI to SQL Server 2016, and launches the new AMI. To learn more, read about the AWSEC2-CloneInstanceAndUpgradeSQLServer action.

Amazon RDS makes it easy for you to upgrade your DB Instances to new major or minor upgrades to SQL Server. The upgrade is performed in-place, and can be initiated with a couple of clicks. For example, if you are currently running SQL Server 2014, you have the following upgrades available:

You can also opt-in to automatic upgrades to new minor versions that take place within your preferred maintenance window:

Before you upgrade a production DB Instance, you can create a snapshot backup, use it to create a test DB Instance, upgrade that instance to the desired new version, and perform acceptance testing. To learn more, about upgrades, read Upgrading the Microsoft SQL Server DB Engine.

2 – SQL Server on Linux
If your organization prefers Linux, you can run SQL Server on Ubuntu, Amazon Linux 2, or Red Hat Enterprise Linux using our License Included (LI) Amazon Machine Images. Read the most recent launch announcement or search for the AMIs in AWS Marketplace using the EC2 Launch Instance Wizard:

This is a very cost-effective option since you do not need to pay for Windows licenses.

You can use the new re-platforming tool (another AWS Systems Manager script) to move your existing SQL Server databases (2008 and above, either in the cloud or on-premises) from Windows to Linux.

3 – Always-On Availability Groups (Amazon RDS for SQL Server)
If you are running enterprise-grade production workloads on Amazon RDS (our managed database service), you should definitely enable this feature! It enhances availability and durability by replicating your database between two AWS Availability Zones, with a primary instance in one and a hot standby in another, with fast, automatic failover in the event of planned maintenance or a service disruption. You can enable this option for an existing DB Instance, and you can also specify it when you create a new one:

To learn more, read Multi-AZ Deployments Using Microsoft SQL Mirroring or Always On.

4 – Lambda Support
Let’s talk about some features for developers!

Launched in 2014, and the subject of continuous innovation ever since, AWS Lambda lets you run code in the cloud without having to own, manage, or even think about servers. You can choose from several .NET Core runtimes for your Lambda functions, and then write your code in either C# or PowerShell:

To learn more, read Working with C# and Working with PowerShell in the AWS Lambda Developer Guide. Your code has access to the full set of AWS services, and can make use of the AWS SDK for .NET; read the Developing .NET Core AWS Lambda Functions post for more info.

5 – CDK for .NET
The AWS CDK (Cloud Development Kit) for .NET lets you define your cloud infrastructure as code and then deploy it using AWS CloudFormation. For example, this code (stolen from this post) will generate a template that creates an Amazon Simple Queue Service (SQS) queue and an Amazon Simple Notification Service (SNS) topic:

var queue = new Queue(this, "MyFirstQueue", new QueueProps
{
    VisibilityTimeoutSec = 300
}
var topic = new Topic(this, "MyFirstTopic", new TopicProps
{
    DisplayName = "My First Topic Yeah"
});

6 – EC2 AMIs for .NET Core
If you are building Linux applications that make use of .NET Core, you can use use our Amazon Linux 2 and Ubuntu AMIs. With .NET Core, PowerShell Core, and the AWS Command Line Interface (CLI) preinstalled, you’ll be up and running— and ready to deploy applications—in minutes. You can find the AMIs by searching for core when you launch an EC2 instance:

7 – .NET Dev Center
The AWS .Net Dev Center contains materials that will help you to learn how design, build, and run .NET Applications on AWS. You’ll find articles, sample code, 10-minute tutorials, projects, and lots more:

8 – AWS License Manager
We want to help you to manage and optimize your Windows and SQL Server applications in new ways. For example,  AWS License Manager helps you to manage the licenses for the software that you run in the cloud or on-premises (read my post, New AWS License Manager – Manage Software Licenses and Enforce Licensing Rules, to learn more). You can create custom rules that emulate those in your licensing agreements, and enforce them when an EC2 instance is launched:

The License Manager also provides you with information on license utilization so that you can fine-tune your license portfolio, possibly saving some money in the process!

9 – Import, Export, and Migration
You have lots of options and choices when it comes to moving your code and data into and out of AWS. Here’s a very brief summary:

TSO Logic – This new member of the AWS family (we acquired the company earlier this year) offers an analytics solution that helps you to plan, optimize, and save money as you make your journey to the cloud.

VM Import/Export – This service allows you to import existing virtual machine images to EC2 instances, and export them back to your on-premises environment. Read Importing a VM as an Image Using VM Import/Export to learn more.

AWS Snowball – This service lets you move petabyte scale data sets into and out of AWS. If you are at exabyte scale, check out the AWS Snowmobile.

AWS Migration Acceleration Program – This program encompasses AWS Professional Services and teams from our partners. It is based on a three step migration model that includes a readiness assessment, a planning phase, and the actual migration.

10 – 21st Century Applications
AWS gives you a full-featured, rock-solid foundation and a rich set of services so that you can build tomorrow’s applications today! You can go serverless with the .NET Core support in Lambda, make use of our Deep Learning AMIs for Windows, host containerized apps on Amazon ECS or eks], and write code that makes use of the latest AI-powered services. Your applications can make use of recommendations, forecasting, image analysis, video analysis, text analytics, document analysis, text to speech, translation, transcription, and more.

11 – AWS Integration
Your existing Windows Applications, both cloud-based and on-premises, can make use of Windows file system and directory services within AWS:

Amazon FSx for Windows Server – This fully managed native Windows file system is compatible with the SMB protocol and NTFS. It provides shared file storage for Windows applications, backed by SSD storage for fast & reliable performance. To learn more, read my blog post.

AWS Directory Service – Your directory-aware workloads and AWS Enterprise IT applications can use this managed Active Directory that runs in the AWS Cloud.

Join our Team
If you would like to build, manage, or market new AWS offerings for the Windows market, be sure to check out our current openings. Here’s a sampling:

Senior Digital Campaign Marketing Manager – Own the digital tactics for product awareness and run adoption campaigns.

Senior Product Marketing Manager – Drive communications and marketing, create compelling content, and build awareness.

Developer Advocate – Drive adoption and community engagement for SQL Server on EC2.

Learn More
Our freshly updated Windows on AWS and SQL Server on AWS pages contain case studies, quick starts, and lots of other useful information.

Jeff;

In the Works – AWS Region in Indonesia

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/in-the-works-aws-region-in-indonesia/

Last year we launched two new AWS Regions—a second GovCloud Region in the United States, and our first Nordic Region in Sweden—and we announced that we are working on regions in Cape Town, South Africa and Milan, Italy.

Jakarta in the Future
Today, I am happy to announce that we are working on the AWS Asia Pacific (Jakarta) Region in Indonesia. The new region will be based in Greater Jakarta, will be comprised of three Availability Zones, and will give AWS customers and partners the ability to run their workloads and store their data in Indonesia. The AWS Asia Pacific (Jakarta) Region will be our ninth region in Asia Pacific, joining existing regions in Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, Tokyo, and an upcoming region in Hong Kong SAR.

AWS customers are already making use of 61 Availability Zones across 20 infrastructure regions worldwide. Today’s announcement brings the total number of global regions (operational and in the works) up to 25.

We are looking forward to serving new and existing customers in Indonesia and working with partners across Asia Pacific. The addition of the AWS Asia Pacific (Jakarta) Region will enable more Indonesian organizations to leverage advanced technologies such as Analytics, Artificial Intelligence, Database, Internet of Things (IoT), Machine Learning, Mobile services, and more to drive innovation. Of course, the new region will also be open to existing AWS customers who would like to process and store data in Indonesia. We are already working to help prepare developers in Indonesia for the digital future, with programs like AWS Educate, and AWS Activate. Dozens of universities and business schools across Indonesia are already participating in our educational programs, as are a plethora of startups and accelerators.

Stay Tuned
I’ll be sure to share additional news about this and other upcoming AWS regions as soon as I have it, so stay tuned!

Jeff;

New – Advanced Request Routing for AWS Application Load Balancers

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-advanced-request-routing-for-aws-application-load-balancers/

AWS Application Load Balancers have been around since the summer of 2016! They support content-based routing, work well for serverless & container-based applications, and are highly scalable. Many AWS customers are using the existing host and path-based routing to power their HTTP and HTTPS applications, while also taking advantage of other ALB features such as port forwarding (great for container-based applications), health checks, service discovery, redirects, fixed responses, and built-in authentication.

Advanced Request Routing
The host-based routing feature allows you to write rules that use the Host header to route traffic to the desired target group. Today we are extending and generalizing this feature, giving you the ability to write rules (and route traffic) based on standard and custom HTTP headers and methods, the query string, and the source IP address. We are also making the rules and conditions more powerful; rules can have multiple conditions (AND’ed together), and each condition can specify a match on multiple values (OR’ed).

You can use this new feature to simplify your application architecture, eliminate the need for a proxy fleet for routing, and to block unwanted traffic at the load balancer. Here are some use cases:

  • Separate bot/crawler traffic from human traffic.
  • Assign customers or groups of customers to cells (distinct target groups) and route traffic accordingly.
  • Implement A/B testing.
  • Perform canary or blue/green deployments.
  • Route traffic to microservice handlers based on method (PUTs to one target group and GETs to another, for example).
  • Implement access restrictions based on IP address or CDN.
  • Selectively route traffic to on-premises or in-cloud target groups.
  • Deliver different pages or user experiences to various types and categories of devices.

Using Advanced Request Routing
You can use this feature with your existing Application Load Balancers by simply editing your existing rules. I will start with a simple rule that returns a fixed, plain-text response (the examples in this post are for testing and illustrative purposes; I am sure that yours will be more practical and more interesting):

I can use curl to test it:

$ curl http://TestALB-156468799.elb.amazonaws.com
Default rule reached!

I click Insert Rule to set up some advanced request routing:

Then I click Add condition and examine the options that are available to me:

I select Http header, and create a condition that looks for a cookie named user with value jeff. Then I create an action that returns a fixed response:

I click Save, wait a few seconds for the change to take effect, and then issue a pair of requests:

$ curl http://TestALB-156468799.elb.amazonaws.com
Default rule reached!

$ curl --cookie "user=jeff" http://TestALB-156468799.elb.amazonaws.com
Hello Jeff

I can also create a rule that matches one or more CIDR blocks of IP addresses:

$ curl http://TestALB-156468799.elb.amazonaws.com
Hello EC2 Instance

I can match on the query string (this is very useful for A/B testing):

$ curl http://TestALB-156468799.elb.amazonaws.com?ABTest=A
A/B test, option A selected 

I can also use a wildcard if all I care about is the presence of a particular field name:

I can match a standard or custom HTTP method. Here, I will invent one called READ:

$ curl --request READ http://TestALB-156468799.elb.amazonaws.com
Custom READ method invoked

I have a lot of flexibility (not new, but definitely worth reviewing) when it comes to the actions:

Forward to routes the request to a target group (a set of EC2 instances, a Lambda function, or a list of IP addresses).

Redirect to generates a 301 (permanent) or 302 (found) response, and can also be used to switch between HTTP and HTTPS.

Return fixed response generates a static response with any desired response code, as I showed you earlier.

Authenticate uses Amazon Cognito or an OIDC provider to authenticate the request (applicable to HTTPS listeners only).

Things to Know
Here are a couple of other things that you should know about this cool and powerful new feature:

Metrics – You can look at the Rule Evaluations and HTTP fixed response count CloudWatch metrics to learn more about activity related to your rules (learn more):

Programmatic Access – You can also create, modify, examine, and delete rules using the ALB API and CLI (CloudFormation support will be ready soon).

Rule Matching – The rules are powered by string matching, so test well and double-check that your rules are functioning as intended. The matched_rule_priority and actions_executed fields in the ALB access logs can be helpful when debugging and testing (learn more).

Limits – Each ALB can have up to 100 rules, not including the defaults. Each rule can reference up to 5 values and can use up to 5 wildcards. The number of conditions is limited only by the number of unique values that are referenced.

Available Now
Advanced request routing is available now in all AWS regions at no extra charge (you pay the usual prices for the Application Load Balancer).

Jeff;

 

AWS App Mesh – Application-Level Networking for Cloud Applications

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-app-mesh-application-level-networking-for-cloud-applications/

AWS App Mesh helps you to run and monitor HTTP and TCP services at scale. You get a consistent way to route and monitor traffic, giving you insight into problems and the ability to re-route traffic after failures or code changes. App Mesh uses the open source Envoy proxy, giving you access to a wide range of tools from AWS partners and the open source community.

Services can run on AWS Fargate, Amazon EC2, Amazon ECS, Amazon Elastic Container Service for Kubernetes, or Kubernetes. All traffic in and out of the each service goes through the Envoy proxy so that it can be routed, shaped, measured, and logged. This extra level of indirection lets you build your services in any desired languages without having to use a common set of communication libraries.

App Mesh Concepts
Before we dive in, let’s review a couple of important App Mesh concepts and components:

Service Mesh – A a logical boundary for network traffic between the services that reside within it. A mesh can contain virtual services, virtual nodes, virtual routers, and routes.

Virtual Service – An abstraction (logical name) for a service that is provided directly (by a virtual node) or indirectly (through a virtual router). Services within a mesh use the logical names to reference and make use of other services.

Virtual Node – A pointer to a task group (an ECS service or a Kubernetes deployment) or a service running on one or more EC2 instances. Each virtual node can accept inbound traffic via listeners, and can connect to other virtual nodes via backends. Also, each node has a service discovery configuration (currently a DNS name) that allows other nodes to discover the IP addresses of the tasks, pods, or instances.

Virtual Router – A handler for one or more virtual services within a mesh. Each virtual router listens for HTTP traffic on a specific port.

Route – Routes use prefix-based matching on URLs to route traffic to virtual nodes, with optional per-node weights. The weights can be used to test new service versions in production while gradually increasing the amount of traffic that they handle.

Putting it all together, each service mesh contains a set of services that can be accessed by URL paths specified by routes. Within the mesh, services refer to each other by name.

I can access App Mesh from the App Mesh Console, the App Mesh CLI, or the App Mesh API. I’ll show you how to use the Console and take a brief look at the CLI.

Using the App Mesh Console
The console lets me create my service mesh and the components within it. I open the App Mesh Console and click Get started:

I enter the name of my mesh and my first virtual service (I can add more later), and click Next:

I define the first virtual node:

I can click Additional configuration to specify service backends (other services that this one can call) and logging:

I define my node’s listener via protocol (HTTP or TCP) and port, set up an optional health check, and click Next:

Next, I define my first virtual router and a route for it:

I can apportion traffic across several virtual nodes (targets) on a percentage basis, and I can use prefix-based routing for incoming traffic:

I review my choices and click Create mesh service:

The components are created in a few seconds and I am just about ready to go:

The final step, as described in the App Mesh Getting Started Guide, is to update my task definitions (Amazon ECS or AWS Fargate) or pod specifications (Amazon EKS or Kubernetes) to reference the Envoy container image and the proxy container image. If my service is running on an EC2 instance, I will need to deploy Envoy there.

Using the AWS App Mesh Command Line
App Mesh lets you specify each type of component in a simple JSON form and provides you with command-line tools to create each one (create-mesh, create-virtual-service, create-virtual-node, and create-virtual-router). For example, I can define a virtual router in a file:

{
  "meshName": "mymesh",
  "spec": {
        "listeners": [
            {
                "portMapping": {
                    "port": 80,
                    "protocol": "http"
                }
            }
        ]
    },
  "virtualRouterName": "serviceA"
}

And create it with one command:

$ aws appmesh create-virtual-router --cli-input-json file://serviceA-router.json

Now Available
AWS App Mesh is available now and you can start using it today in the US East (N. Virginia), US East (Ohio), US West (Oregon), US West (N. California), Canada (Central), Europe (Ireland), Europe (Frankfurt), Europe (London), Asia Pacific (Mumbai), Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Singapore), and Asia Pacific (Seoul) Regions today.

Jeff;

New – AWS Deep Learning Containers

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-aws-deep-learning-containers/

We want to make it as easy as possible for you to learn about deep learning and to put it to use in your applications. If you know how to ingest large datasets, train existing models, build new models, and to perform inferences, you’ll be well-equipped for the future!

New Deep Learning Containers
Today I would like to tell you about the new AWS Deep Learning Containers. These Docker images are ready to use for deep learning training or inferencing using TensorFlow or Apache MXNet, with other frameworks to follow. We built these containers after our customers told us that they are using Amazon EKS and ECS to deploy their TensorFlow workloads to the cloud, and asked us to make that task as simple and straightforward as possible. While we were at it, we optimized the images for use on AWS with the goal of reducing training time and increasing inferencing performance.

The images are pre-configured and validated so that you can focus on deep learning, setting up custom environments and workflows on Amazon ECS, Amazon Elastic Container Service for Kubernetes, and Amazon Elastic Compute Cloud (EC2) in minutes! You can find them in AWS Marketplace and Elastic Container Registry, and use them at no charge. The images can be used as-is, or can be customized with additional libraries or packages.

Multiple Deep Learning Containers are available, with names based on the following factors (not all combinations are available):

  • Framework – TensorFlow or MXNet.
  • Mode – Training or Inference. You can train on a single node or on a multi-node cluster.
  • Environment – CPU or GPU.
  • Python Version – 2.7 or 3.6.
  • Distributed Training – Availability of the Horovod framework.
  • Operating System – Ubuntu 16.04.

Using Deep Learning Containers
In order to put an AWS Deep Learning Container to use, I create an Amazon ECS cluster with a p2.8xlarge instance:

$ aws ec2 run-instances --image-id  ami-0ebf2c738e66321e6 \
  --count 1 --instance-type p2.8xlarge \
  --key-name keys-jbarr-us-east ... 

I verify that the cluster is running, and check that the ECS Container Agent is active:

Then I create a task definition in a text file (gpu_task_def.txt):

{
  "requiresCompatibilities": [
    "EC2"
  ],
  "containerDefinitions": [
    {
      "command": [
        "tensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=saved_model_half_plus_two_gpu  --model_base_path=/models/saved_model_half_plus_two_gpu"
      ],
      "entryPoint": [
        "sh",
        "-c"
      ],
      "name": "EC2TFInference",
      "image": "841569659894.dkr.ecr.us-east-1.amazonaws.com/sample_tf_inference_images:gpu_with_half_plus_two_model",
      "memory": 8111,
      "cpu": 256,
      "resourceRequirements": [
        {
          "type": "GPU",
          "value": "1"
        }
      ],
      "essential": true,
      "portMappings": [
        {
          "hostPort": 8500,
          "protocol": "tcp",
          "containerPort": 8500
        },
        {
          "hostPort": 8501,
          "protocol": "tcp",
          "containerPort": 8501
        },
        {
          "containerPort": 80,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/TFInference",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "volumes": [],
  "networkMode": "bridge",
  "placementConstraints": [],
  "family": "Ec2TFInference"
}

I register the task definition and capture the revision number (3):

Next, I create a service using the task definition and revision number:

I use the console to navigate to the task:

Then I find the external binding for port 8501:

Then I run three inferences (this particular model was trained on the function y = ax + b, with a = 0.5 and b = 2):

$ curl -d '{"instances": [1.0, 2.0, 5.0]}' \
  -X POST http://xx.xxx.xx.xx:8501/v1/models/saved_model_half_plus_two_gpu:predict
{
    "predictions": [2.5, 3.0, 4.5
    ]
}

As you can see, the inference predicted the values 2.5, 3.0, and 4.5 when given inputs of 1.0, 2.0, and 5.0. This is a very, very simple example but it shows how you can use a pre-trained model to perform inferencing in ECS via the new Deep Learning Containers. You can also launch a model for training purposes, perform the training, and then run some inferences.

Jeff;

New – Concurrency Scaling for Amazon Redshift – Peak Performance at All Times

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-concurrency-scaling-for-amazon-redshift-peak-performance-at-all-times/

Amazon Redshift is a data warehouse that can expand to exabyte-scale. Today, tens of thousands of AWS customers (including NTT DOCOMO, Finra, and Johnson & Johnson) use Redshift to run mission-critical BI dashboards, analyze real-time streaming data, and run predictive analytics jobs.

A challenge arises when the number of concurrent queries grows at peak times. When a multitude of business analysts all turn to their BI dashboards or long-running data science workloads compete with other workloads for resources, Redshift will queue queries until enough compute resources become available in the cluster. This ensures that all of the work gets done, but it can mean that performance is impacted at peak times. Two options present themselves:

  • Overprovision the cluster to meet peak needs. This option addresses the immediate issue, but wastes resources and costs more than necessary.
  • Optimize the cluster for typical workloads. This option forces you to wait longer for results at peak times, possibly delaying important business decisions.

New Concurrency Scaling
Today I would like to offer a third option. You can now configure Redshift to add more query processing power on an as-needed basis. This happens transparently and in a manner of seconds, and provides you with fast, consistent performance even as the workload grows to hundreds of concurrent queries. Additional processing power is ready in seconds and does not need to be pre-warmed or pre-provisioned. You pay only for what you use, with per-second billing and also accumulate one hour of concurrency scaling cluster credits every 24 hours while your main cluster is running. The extra processing power is removed when it is no longer needed, making this a perfect way to address the bursty use cases that I described above.

You can allocate the burst power to specific users or queues, and you can continue to use your existing BI and ETL applications. Concurrency Scaling Clusters are used to handle many forms of read-only queries, with additional flexibility in the works; read about Concurrency Scaling to learn more.

Using Concurrency Scaling
This feature can be enabled for an existing cluster in minutes! We recommend starting with a fresh Redshift Parameter Group for testing purposes, so I start by creating one:

Then I edit my cluster’s Workload Management Configuration, select the new parameter group, set the Concurrency Scaling Mode to auto, and click Save:

I will use the Cloud Data Warehouse Benchmark Derived From TPC-DS as a source of test data and test queries. I download the DDL, customize it with my AWS credentials, and use psql to connect to my cluster and create the test data:

sample=# create database sample;
CREATE DATABASE
sample=# \connect sample;
psql (9.2.24, server 8.0.2)
WARNING: psql version 9.2, server version 8.0.
         Some psql features might not work.
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
You are now connected to database "sample" as user "awsuser".
sample=# \i ddl.sql

The DDL creates the tables and loads populates them using data stored in an S3 bucket:

sample=# \dt
                 List of relations
 schema |          name          | type  |  owner
--------+------------------------+-------+---------
 public | call_center            | table | awsuser
 public | catalog_page           | table | awsuser
 public | catalog_returns        | table | awsuser
 public | catalog_sales          | table | awsuser
 public | customer               | table | awsuser
 public | customer_address       | table | awsuser
 public | customer_demographics  | table | awsuser
 public | date_dim               | table | awsuser
 public | dbgen_version          | table | awsuser
 public | household_demographics | table | awsuser
 public | income_band            | table | awsuser
 public | inventory              | table | awsuser
 public | item                   | table | awsuser
 public | promotion              | table | awsuser
 public | reason                 | table | awsuser
 public | ship_mode              | table | awsuser
 public | store                  | table | awsuser
 public | store_returns          | table | awsuser
 public | store_sales            | table | awsuser
 public | time_dim               | table | awsuser
 public | warehouse              | table | awsuser
 public | web_page               | table | awsuser
 public | web_returns            | table | awsuser
 public | web_sales              | table | awsuser
 public | web_site               | table | awsuser
(25 rows)

Then I download the queries and open up a bunch of PuTTY windows so that I can generate a meaningful load for my Redshift cluster:

I run an initial set of parallel queries, and then ramp up over time, I can see them in the Cluster Performance tab for my cluster:

I can see the additional processing power come online as needed, and then go away when no longer needed, in the Database Performance tab:

As you can see, my cluster scales as needed in order to handle all of the queries as expeditiously as possible. The Concurrency Scaling Usage shows me how many seconds of additional processing power I have consumed (as I noted earlier, each cluster accumulates a full hour of concurrency credits every 24 hours).

I can use the parameter max_concurrency_scaling_clusters to control the number of Concurrency Scaling Clusters that can be used (the default limit is 10, but you can request an increase if you need more).

Available Today
You can start making use of Concurrency Scaling Clusters today in the US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo) Regions today, with more to come later this year.

Jeff;

 

New AMD EPYC-Powered Amazon EC2 M5ad and R5ad Instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amd-epyc-powered-amazon-ec2-m5ad-and-r5ad-instances/

Last year I told you about our New Lower-Cost, AMD-Powered M5a and R5a EC2 Instances. Built on the AWS Nitro System, these instances are powered by custom AMD EPYC processors running at 2.5 GHz. They are priced 10% lower than comparable EC2 M5 and R5 instances, and give you a new opportunity to balance your instance mix based on cost and performance.

Today we are adding M5ad and R5ad instances, both powered by custom AMD EPYC 7000 series processors and built on the AWS Nitro System.

M5ad and R5ad Instances
These instances add high-speed, low latency local (physically connected) block storage to the existing M5a and R5a instances that we launched late last year.

M5ad instances are designed for general purpose workloads such as web servers, app servers, dev/test environments, gaming, logging, and media processing. They are available in 6 sizes:

Instance NamevCPUsRAMLocal StorageEBS-Optimized BandwidthNetwork Bandwidth
m5ad.large
28 GiB1 x 75 GB NVMe SSDUp to 2.120 GbpsUp to 10 Gbps
m5ad.xlarge
416 GiB1 x 150 GB NVMe SSDUp to 2.120 GbpsUp to 10 Gbps
m5ad.2xlarge
832 GiB1 x 300 GB NVMe SSDUp to 2.120 GbpsUp to 10 Gbps
m5ad.4xlarge
1664 GiB2 x 300 GB NVMe SSD2.120 GbpsUp to 10 Gbps
m5ad.12xlarge
48192 GiB2 x 900 GB NVMe SSD5 Gbps10 Gbps
m5ad.24xlarge
96384 GiB4 x 900 GB NVMe SSD10 Gbps20 Gbps

R5ad instances are designed for memory-intensive workloads: data mining, in-memory analytics, caching, simulations, and so forth. The R5ad instances are available in 6 sizes:

Instance NamevCPUsRAMLocal StorageEBS-Optimized BandwidthNetwork Bandwidth
r5ad.large
216 GiB1 x 75 GB NVMe SSDUp to 2.120 GbpsUp to 10 Gbps
r5ad.xlarge
432 GiB1 x 150 GB NVMe SSDUp to 2.120 GbpsUp to 10 Gbps
r5ad.2xlarge
864 GiB1 x 300 GB NVMe SSDUp to 2.120 GbpsUp to 10 Gbps
r5ad.4xlarge
16128 GiB2 x 300 GB NVMe SSD2.120 GbpsUp to 10 Gbps
r5ad.12xlarge
48384 GiB2 x 900 GB NVMe SSD5 Gbps10 Gbps
r5ad.24xlarge
96768 GiB4 x 900 GB NVMe SSD10 Gbps20 Gbps

Again, these instances are available in the same sizes as the M5d and R5d instances, and the AMIs work on either, so go ahead and try both!

Here are some things to keep in mind about the local NMVe storage on the M5ad and R5ad instances:

Naming – You don’t have to specify a block device mapping in your AMI or during the instance launch; the local storage will show up as one or more devices (/dev/nvme*1 on Linux) after the guest operating system has booted.

Encryption – Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key. Each key is destroyed when the instance is stopped or terminated.

Lifetime – Local NVMe devices have the same lifetime as the instance they are attached to, and do not stick around after the instance has been stopped or terminated.

M5ad and R5ad instances are available in the US East (N. Virginia), US West (Oregon), US East (Ohio), and Asia Pacific (Singapore) Regions in On-Demand, Spot, and Reserved Instance form.

Jeff;

 

New Amazon S3 Storage Class – Glacier Deep Archive

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-s3-storage-class-glacier-deep-archive/

Many AWS customers collect and store large volumes (often a petabyte or more) of important data but seldom access it. In some cases raw data is collected and immediately processed, then stored for years or decades just in case there’s a need for further processing or analysis. In other cases, the data is retained for compliance or auditing purposes. Here are some of the industries and use cases that fit this description:

Financial – Transaction archives, activity & audit logs, and communication logs.

Health Care / Life Sciences – Electronic medical records, images (X-Ray, MRI, or CT), genome sequences, records of pharmaceutical development.

Media & Entertainment – Media archives and raw production footage.

Physical Security – Raw camera footage.

Online Advertising – Clickstreams and ad delivery logs.

Transportation – Vehicle telemetry, video, RADAR, and LIDAR data.

Science / Research / Education – Research input and results, including data relevant to seismic tests for oil & gas exploration.

Today we are introducing a new and even more cost-effective way to store important, infrequently accessed data in Amazon S3.

Amazon S3 Glacier Deep Archive Storage Class
The new Glacier Deep Archive storage class is designed to provide durable and secure long-term storage for large amounts of data at a price that is competitive with off-premises tape archival services. Data is stored across 3 or more AWS Availability Zones and can be retrieved in 12 hours or less. You no longer need to deal with expensive and finicky tape drives, arrange for off-premises storage, or worry about migrating data to newer generations of media.

Your existing S3-compatible applications, tools, code, scripts, and lifecycle rules can all take advantage of Glacier Deep Archive storage. You can specify the new storage class when you upload objects, alter the storage class of existing objects manually or programmatically, or use lifecycle rules to arrange for migration based on object age. You can also make use of other S3 features such as Storage Class Analysis, Object Tagging, Object Lock, and Cross-Region Replication.

The existing S3 Glacier storage class allows you to access your data in minutes (using expedited retrieval) and is a good fit for data that requires faster access. To learn more about the entire range of options, read Storage Classes in the S3 Developer Guide. If you are already making use of the Glacier storage class and rarely access your data, you can switch to Deep Archive and begin to see cost savings right away.

Using Glacier Deep Archive Storage – Console
I can switch the storage class of an existing S3 object to Glacier Deep Archive using the S3 Console. I locate the file and click Properties:

Then I click Storage class:

Next, I select Glacier Deep Archive and click Save:

I cannot download the object or edit any of its properties or permissions after I make this change:

In the unlikely event that I need to access this 2013-era video, I select it and choose Restore from the Actions menu:

Then I specify the number of days to keep the restored copy available, and choose either bulk or standard retrieval:

Using Glacier Deep Archive Storage – Lifecycle Rules
I can also use S3 lifecycle rules. I select the bucket and click Management, then select Lifecycle:

Then I click Add lifecycle rule and create my rule. I enter a name (ArchiveOldMovies), and can optionally use a path or tag filter to limit the scope of the rule:

Next, I indicate that I want the rule to apply to the Current version of my objects, and specify that I want my objects to transition to Glacier Deep Archive 30 days after they are created:

Using Glacier Deep Archive – CLI / Programmatic Access
I can use the CLI to upload a new object and set the storage class:

$ aws s3 cp new.mov s3://awsroadtrip-videos-raw/ --storage-class DEEP_ARCHIVE

I can also change the storage class of an existing object by copying it over itself:

$ aws s3 cp s3://awsroadtrip-videos-raw/new.mov s3://awsroadtrip-videos-raw/new.mov --storage-class DEEP_ARCHIVE

If I am building a system that manages archiving and restoration, I can opt to receive notifications on an SNS topic, an SQS queue, or a Lambda function when a restore is initiated and/or completed:

Other Access Methods
You can also use Tape Gateway configuration of AWS Storage Gateway to create a Virtual Tape Library (VTL) and configure it to use Glacier Deep Archive for storage of archived virtual tapes. This will allow you to move your existing tape-based backups to the AWS Cloud without making any changes to your existing backup workflows. You can retrieve virtual tapes archived in Glacier Deep Archive to S3 within twelve hours. With Tape Gateway and S3 Glacier Deep Archive, you no longer need on-premises physical tape libraries, and you don’t need to manage hardware refreshes and rewrite data to new physical tapes as technologies evolve. For more information, visit the Test Your Gateway Setup with Backup Software page of Storage Gateway User Guide.

Now Available
The S3 Glacier Deep Archive storage class is available today in all commercial regions and in both AWS GovCloud regions. Pricing varies by region, and the storage cost is up to 75% less than for the existing S3 Glacier storage class; visit the S3 Pricing page for more information.

Jeff;